50 Levels of AI Engineering
The path from a single prompt to a system that can observe its own weaknesses, redesign its harness, prove the improvement, and deploy it.
How to read it
This is a capability ladder, not a mandatory architecture. Each level solves a class of problems introduced by the levels before it.
The early levels make a model useful. The middle levels make agents dependable. The final levels turn the surrounding harness into the thing that learns.
Basic LLM application
The model becomes useful when intent is made explicit and reliable code contains its uncertainty.
- 01
Single prompt
One input, one output.
- 02
Structured prompting
Explicit goals, constraints, and schemas.
- 03
Few-shot prompting
Examples shape behavior.
- 04
Prompt pipelines
Multiple model calls chained together.
- 05
Deterministic preprocessing and postprocessing
Code surrounds the model.
Tool-using agents
The system moves from generating an answer to taking actions and judging their results.
- 06
Basic tool calling
APIs, search, databases, and filesystems.
- 07
Agent loop
Decide, act, observe, repeat.
- 08
Error recovery
Retries and alternative actions.
- 09
Explicit planning
Creates a task plan.
- 10
Plan execution tracking
Knows what is complete and what remains.
- 11
Self-verification
Checks whether the objective was actually achieved.
Stateful agents
Useful work outlives a context window and becomes a durable, searchable environment.
- 12
Workspace-based state
Files and artifacts become part of the working environment.
- 13
Retrieval
Searches instead of stuffing everything into context.
- 14
Persistent memory
Facts and decisions survive between sessions.
- 15
Context compaction
Old history becomes structured summaries.
- 16
Dynamic context assembly
Chooses what information to load for each step.
- 17
Checkpoint and resume
Long tasks can stop and continue reliably.
Engineering-grade agents
Agent behavior becomes inspectable, testable, recoverable, and bounded like any production system.
- 18
Sandbox execution
Safely runs code and commands.
- 19
Tests as feedback
Learns from executable failures.
- 20
Environment introspection
Understands repo structure, dependencies, logs, and runtime.
- 21
Artifact-first workflows
Plans, specifications, and results live in durable artifacts rather than chat.
- 22
Observability
Traces actions, tool calls, cost, latency, and failures.
- 23
Policy and permission boundaries
Tools and actions have controlled scopes.
Multi-agent systems
A single loop becomes a coordinated team with explicit ownership and evidence-bearing handoffs.
- 24
Delegation
A parent agent gives bounded work to another agent.
- 25
Parallel agents
Independent investigations run simultaneously.
- 26
Specialized roles
Explorer, coder, reviewer, researcher, and tester.
- 27
Structured handoffs
Agents exchange artifacts and evidence rather than conversational summaries.
- 28
Independent review
One agent validates another's work.
- 29
Manager and orchestrator
Dynamically decomposes work, assigns agents, and tracks dependencies.
Adaptive agent systems
The system changes its own execution strategy as the problem, evidence, and constraints evolve.
- 30
Dynamic orchestration
The topology itself changes based on the problem.
- 31
Capability routing
Chooses agents and models based on measured strengths.
- 32
Confidence-based escalation
Difficult cases move to stronger agents.
- 33
Budget-aware planning
Optimizes quality against compute, cost, and latency.
- 34
Speculative solving
Several competing solutions are attempted.
- 35
Automated selection
Evaluators choose the strongest candidate.
- 36
Dynamic DAG execution
Dependencies determine runtime scheduling.
- 37
Event-driven continuation
Workflows resume on external events.
Self-improving harnesses
The harness closes the loop between production weakness, controlled change, and measured improvement.
- 38
Continuous evaluation
Production behavior feeds benchmark suites.
- 39
Failure mining
Recurrent failures are automatically clustered.
- 40
Automatic prompt improvement
Instructions are changed based on failures.
- 41
Automatic memory-policy improvement
The system learns what to store and retrieve.
- 42
Automatic routing improvement
Learns which agents and models should handle what.
- 43
Automatic workflow improvement
Modifies decomposition and control flow.
- 44
Tool synthesis
Creates new scripts and tools for recurring needs.
- 45
Harness modification
Modifies the agent runtime itself.
- 46
Regression evaluation
Proposed improvements must beat held-out tests.
- 47
Automated experiment generation
The system proposes variants to test.
- 48
Evolutionary search
Maintains and mutates multiple harness designs.
- 49
Meta-agent engineering
Agents explicitly engineer other agents.
- 50
Autonomous agent R&D
The full observe, hypothesize, modify, evaluate, and deploy loop is automated.
The real threshold is not autonomy. It is evidence.
A mature AI system does not merely act without supervision. It preserves the trail from objective to action to result, and it can show that a change improved the system without quietly breaking something else.
Back to top