AI EngineeringAdvanced

Agent Design Patterns: Start One Rung Lower

Prompt chaining, routing, orchestrator-workers, ReAct, plan-and-execute, reflection and tree search: what each costs, what the research shows, and when to climb to the next one.

By Amit UpadhyayPublished 07 Nov 20268 minArticle 1 of 12 in AI Engineering

A ladder of agent patterns: a single call with retrieval, then prompt chaining and routing, then orchestrator-workers and evaluator loops, then a ReAct agent loop with a step limit, with reflection only when tests or errors give real feedback.

On this page

The idea at a glance

The problem

When

A task needs several steps, and what to do next depends on what the last step found

The obvious move

Asking the model to work out the whole answer in one reply

Fails because

It has to guess the results of lookups it never made, so everything after the first step is invented

The fix

Agent patterns works by

A loop where the model chooses one action, sees the real result and then decides the next, using ReAct, plan-and-execute or reflection to suit the task

At the cost of

More model calls, more latency, and a loop that needs limits, tracing and a way to stop

The scenario every mode follows

“My new laptop hasn’t arrived and I can’t log in to the VPN.”

An IT helpdesk agent with tools for orders, accounts and tickets
~3,000 requests a day
claude-sonnet-5-5

Where the obvious approach breaks

56% of failures came from invented facts

If you reason through the whole request in one reply, without looking anything up.

In the ReAct paper (Yao et al., 2022), hallucination made up 56% of chain-of-thought’s failures on HotpotQA, where the model reasons without retrieving anything. Interleaving actions with reasoning removed most of those failures.

Helps to know firstFunction calling

Most Agent Projects Start One Pattern Too High

A team needs to turn support emails into tickets: read the email, pull out the product and the order number, pick a queue, write a summary. Someone builds it as an autonomous agent with six tools and a loop. In testing it works. In production it sometimes calls the order API four times for the same email, sometimes skips the queue lookup, and the cost per email swings by a factor of ten. Nobody can say why a given run took the path it did.

That task has a fixed shape. Every email goes through the same four steps, so it wants a chain of calls in your own code, with a check between steps. Anthropic's guide to building agents makes the same split: workflows run LLMs and tools through predefined code paths, while agents let the model direct its own process and tool use. Their advice is to find the simplest solution that works and add complexity only when it pays for itself, which for many applications means a single well-prompted call with retrieval and a few examples.

Patterns help because they give you names for those levels. Below are the ones I reach for, in the order I'd try them, with the evidence behind each and where it stops being worth the cost.

You need this when

You are about to give a model a loop and a set of tools, and you aren't sure it needs either
An agent works in demos but its cost and step count vary wildly in production
Someone suggested adding "reflection" and you want to know when it actually helps
You need a shared vocabulary for design reviews of LLM features

The Ladder

Think of the patterns as rungs. Each rung up gives the model more control over what happens next, and costs you predictability, latency and money. Climb only when the rung below has failed on real inputs.


flowchart TD
    A["Single call<br/>+ retrieval + examples"] -->|"task has fixed steps"| B["Prompt chaining<br/>with gates"]
    A -->|"inputs fall into clear categories"| C["Routing"]
    B -->|"steps are independent"| D["Parallelization<br/>sectioning or voting"]
    B -->|"subtasks unknown up front"| E["Orchestrator-workers"]
    B -->|"output can be checked and improved"| F["Evaluator-optimizer"]
    E -->|"path depends on what tools return"| G["Agent loop<br/>ReAct, plan-and-execute"]

Workflow Patterns: Your Code Holds the Control Flow

Pattern	What it does	Use it when
Prompt chaining	Each call works on the previous call's output; code checks run between steps	The task splits cleanly into fixed steps, such as outline, check the outline, write
Routing	Classify the input, then send it to a specialised prompt, tool set or model	Distinct categories that are better handled apart, such as refunds versus technical questions, or easy versus hard
Parallelization	Sectioning runs independent subtasks at once; voting runs the same task several times	Subtasks don't depend on each other, or you want several independent opinions, such as a security review
Orchestrator-workers	A central call breaks the task down at run time and hands pieces to workers	You can't know the subtasks in advance, such as how many files a code change touches
Evaluator-optimizer	One call produces, another evaluates and gives feedback, in a loop	You have clear criteria and a second pass measurably improves the result

The support-email task above is prompt chaining. Extraction is one call with a JSON schema, a code check rejects the result if the order number doesn't match your format, queue selection is a second call, and the summary is a third. Every run takes the same path, and when it fails you know which step failed.

Routing is the cheapest win I know. A small model or even a plain classifier decides which bucket an input belongs in, and only the hard bucket goes to the large model. The gate that checks the route matters as much as the route: log the chosen bucket on every request, because a router that quietly sends everything to one path is a common failure.

Agent Patterns: The Model Holds the Control Flow

An agent is, at its core, a model calling tools in a loop and reading what comes back. What changes between patterns is how it plans and whether it looks back at its own work.

ReAct: reason, act, observe

ReAct (Yao et al., 2022) interleaves a reasoning step with an action and feeds the observation back into the next step. On HotpotQA and Fever, letting the model query a simple Wikipedia API reduced the hallucination and error propagation seen in chain-of-thought reasoning on its own. On the ALFWorld and WebShop decision benchmarks, ReAct beat imitation and reinforcement learning baselines by 34 and 10 points of absolute success rate. Most tool-calling loops you write today are ReAct, whether or not they use the name. The journey for this topic walks one run step by step.

MAX_STEPS = 8

def run_agent(task: str) -> str:
    messages = [system_prompt(), user(task)]
    for step in range(MAX_STEPS):
        reply = call_model(messages, tools=TOOLS)       # your provider's SDK
        if not reply.tool_calls:
            return reply.text                            # model decided it is done
        for call in reply.tool_calls:
            if call.name in WRITE_TOOLS and not approved(call):
                result = "Not run: needs user confirmation."
            else:
                result = run_tool(call.name, call.arguments)
            messages += [reply.as_message(), tool_result(call.id, result)]
    return escalate(task, messages)                      # out of steps: hand to a person

Three lines in that loop matter more than the prompt. MAX_STEPSis the stopping condition Anthropic's guide recommends for keeping control. The write-tool check means anything that changes state waits for a person. And running out of steps escalates instead of returning whatever the model said last.

Plan-and-execute: decide the steps once

ReAct pays for a full model call after every observation, and each call re-reads everything before it. Plan-and-execute writes the plan first and then runs the steps, calling the model again only to put the answer together or to re-plan after a failure. ReWOO (Xu et al., 2023) is the clearest version: a planner writes the tool calls with placeholders for the results, workers run them, and a solver combines the evidence. The authors report 5 times fewer tokens and a 4-point accuracy gain on HotpotQA. Plan-and-Solve prompting (Wang et al., 2023) applies a similar "plan first" idea inside a single prompt.

The trade-off is adaptability. A plan written before any tool has run can't react to a surprising result, so I use plan-and-execute when the tools are reliable and the steps are mostly predictable, and keep ReAct for exploratory work like debugging or research.

Reflection: only with something real to reflect on

Reflexion (Shinn et al., 2023) has the agent write a short critique after a failed attempt, keep it in memory, and try again. It reached 91% pass@1 on HumanEval, against 80% for GPT-4 at the time. Self-Refine (Madaan et al., 2023) uses one model as generator, critic and refiner, with no training.

Read the fine print before you add a reflection step everywhere. Huang et al. (ICLR 2024) found that, on reasoning tasks, models struggle to correct themselves without external feedback, and sometimes get worse after trying. The coding results above work because the feedback is real: tests failed. So my rule is that reflection needs a signal from outside the model, such as a test run, a compiler error, a schema validation failure or a tool error. Asking the model "are you sure?" with nothing new to look at mostly buys you a second bill.

def write_function(spec: str, max_rounds: int = 3) -> str:
    code = call_model(prompt_for(spec)).text
    for _ in range(max_rounds):
        result = run_tests(code)                 # external feedback, not opinion
        if result.passed:
            return code
        critique = call_model(reflect_prompt(spec, code, result.failures)).text
        code = call_model(fix_prompt(spec, code, critique)).text
    raise NeedsHuman(spec, code, result.failures)

Tree search: when you can afford many attempts

LATS (Zhou et al., 2023) combines Monte Carlo tree search with model-generated value estimates and self-reflection, exploring several paths instead of one. It reached 92.7% pass@1 on HumanEval with GPT-4. The cost is the point: you run many branches to get one answer. It fits offline or batch work where a correct answer is worth many calls and you can score candidates automatically. I have not seen it pay off for interactive requests.

Choosing: The Questions I Ask

Can I write the steps down today?If yes, it's a workflow. Chaining or routing first.
Does the next step depend on what a tool returns? If yes, you need a loop. Start with ReAct and a hard step limit.
Is there an automatic check of quality?Tests, a schema, a policy rule. If yes, an evaluator-optimizer loop or reflection can earn its cost. If no, don't add one.
What does a wrong action cost? Anything that sends, deletes, pays or changes access goes behind a confirmation, whatever the pattern.
What is the latency budget? Each extra model call adds a round trip. A pattern that needs eight calls rarely fits a chat reply.

What Goes Wrong

⚠️ Common Mistakes

1. An agent where a chain would do

If every input follows the same path, the loop only adds variance. Move the control flow into code and keep the model for each step's judgement.

2. No step limit, no budget

A confused agent repeats itself. Cap steps, cap tokens per run, and alert on the same tool called with the same arguments three times. The agent observability topic covers how to see that in traces.

3. Reflection without feedback

A critique step that sees nothing new is a second guess at the same answer. Give it test results or errors, or remove it.

4. Choosing a framework before a pattern

Frameworks make the first version quick and can hide the prompts and responses you need to debug. Write the pattern by hand once, so you know what the framework is doing for you.

5. Copying benchmark numbers as expectations

The results above come from specific models and benchmarks, mostly from 2022 and 2023. They show the patterns work, not what you will get. Build a small evaluation set from your own traffic and measure each rung before you climb.

Where to Go Next

If one agent turns into several, the multi-agent systems topic covers handoffs and supervisors. For the tool side of every pattern here, start with function calling, because a pattern is only as reliable as the tool definitions it calls.

Decisions and trade-offs

The choices this rests on

Each choice has a reason and a point where it stops being right. The Test questions are built from these.

1
Pick the loop shape from the task: ReAct for exploring, plan-and-execute for known sequences, reflection for quality
ReAct is flexible but wanders. Plan-and-execute writes the plan once and follows it, which is cheaper and easier to audit when the steps are predictable. Reflection adds a review pass when the output itself needs to be good, as with code or a report.
Stops being right when
The task is fully predictable. Then you do not need an agent at all; write a workflow in code.
2
Cap every run at 8 steps and 20 s per tool call
Agent loops fail by not stopping. A cap turns an endless loop into a bounded failure you can see in the logs and explain to the user.
Stops being right when
Some tasks legitimately need many steps, like a deep research job. Budget those by tokens and time, and checkpoint progress.
3
Tools return errors that say what went wrong and what to try
“Account service timed out; try again in a minute or create a ticket” gives the model a next move. An empty string or a stack trace gives it nothing, and it retries forever.
Stops being right when
Never. This is the cheapest reliability fix in agent work.
4
Read tools run freely; anything that changes state waits for confirmation
An agent that unlocks accounts or orders hardware based on its own reading of a message will one day act on a misunderstanding, or on injected text. A confirmation step makes that a question instead of an incident.
Stops being right when
Fully automated back-office flows. Use policy checks and idempotency keys there instead of a human prompt.
5
Record every thought, tool call and result as one trace per run
When an agent gets something wrong, the answer to “why?” is in the sequence of steps, not the final reply. Without the trace you are guessing.
Stops being right when
Steps contain data you cannot store. Redact the arguments and results, but keep the sequence.

In production

Running it once real traffic arrives

What to watch, a rough cost at the scenario’s volume, and what to check first when something goes wrong.

What to watch

The targets are starting points for this scenario, not standards. Set yours from your own baseline.

Task success rate
Measured on whole tasks, not single steps. An agent can make every call correctly and still not solve the problem.
≥ 85% on the eval set
Steps per task
Runs at the cap are loops. Read those traces first.
Median 3–5; few at the cap
Repeated identical tool calls
The clearest sign of a tool returning something the model cannot act on.
≈ 0
p95 task latency
Each step is a model call plus a tool call. Latency grows with steps, not with answer length.
< 15 s

What it might cost

Model calls~6 calls × ~2,500 input tokens × $2 per 1M≈ $0.03/task

Output~6 calls × ~150 tokens × $10 per 1M≈ $0.01/task

At scenario volume3,000 tasks a day × 30 days × $0.04≈ $3,600/month

EstimateAbout $3,600 a month

Illustrative estimate for this scenario, not a budget. It counts only the lines shown, so anything not listed is extra.

Claude Sonnet 5.5 list prices. Prompt caching on the system prompt and tool definitions usually cuts the input line substantially.

When it breaks

Runs keep hitting the step cap

CheckLook for the same tool call repeated in the trace, and read what that tool returned
FixMake the tool return an actionable error, and stop after two identical calls

The agent does the right steps in a strange order

CheckCompare the system prompt’s guidance with the tool descriptions
FixState the order in the prompt, or switch that task to plan-and-execute

Users report actions they did not ask for

CheckFind the write tool call in the trace and the message that triggered it
FixRequire confirmation for that tool, and check whether tool output carried injected text

In the real world

A helpdesk agent that learned to stop

A composite of common internal-helpdesk deployments. The numbers are typical ranges, not one company’s results.

Before

The first agent had twelve tools and no step limit. Most requests went fine, but a few each day looped on a flaky account service for minutes, and once it reset a password nobody had asked to reset.

What they built

A step cap and a per-tool timeout on every run
Tool errors rewritten to say what happened and what to try
Confirmation before any tool that changes an account, order or ticket
Plan-and-execute for the three most common request types; ReAct for everything else

What changed

Runaway runs dropped to a handful a week, each with a clear trace
No more unrequested actions
Common requests got faster and cheaper on plan-and-execute

The lessonMost agent reliability work is plumbing: limits, good errors and confirmation. The model was rarely the problem.

When not to use it

Reach for something else when

If you needUse instead

A task whose steps are always the sameA workflow in code that calls the model where it needs language

A single lookup and a replyOne function call

Decisions that must be fully auditable in advancePlan-and-execute with an approved plan, or no agent

Check yourself

Two questions before you move on

Answer in your head first, then open the answer.

Why does the agent check the order before acting on the VPN problem, instead of answering both at once?

It does not know either answer yet. Each tool result changes what the next step should be. The laptop went to the wrong address, and the account was locked rather than expired, which a one-shot reply would have had to guess.

When would you choose plan-and-execute over ReAct?

When the steps are predictable enough to plan up front. A plan written once is cheaper, easier to audit and less likely to wander than deciding every step as you go.

Where this leads

Multi-agent systems

One agent with a dozen tools starts to lose focus. Splitting the work between specialist agents is the next step, and it has its own costs.

Next topic →

Next: See →

Sources

agent-patterns ReAct reflection workflows agents

← PreviousModel Serving With vLLM: The KV Cache Decides Your Capacity

Continue reading

All AI Engineering articles →

AI Engineering8 min

Model Serving With vLLM: The KV Cache Decides Your Capacity

KV cache arithmetic, PagedAttention, continuous batching, prefix caching, FP8 and speculative decoding, the vLLM flags that matter, and the five metrics that show which limit you hit.

AI Engineering5 min

Voice Agents: A Real-Time Problem With a Model Inside

Speech-to-speech or a pipeline, a latency budget for every stage, interruptions that really work, prompts written for the ear, and what a call actually costs.

AI Engineering13 min

Guardrails: Input/Output Safety in Production

Input, output, schema, tool, and cost guardrails — the layered defence system that catches LLM failures before they reach a user or downstream system.

Discussion