A prototype agent works in demos, but its cost and latency are too high for production. What do you examine first?
Why C is the answer
Which steps really need model judgment and which could be plain code
Many agent steps are deterministic. Moving them into code cuts cost and latency, and it removes a source of variation.
In detail
- Many agent steps are deterministic.
- Moving them into code cuts cost and latency.
- It also removes a source of variation.
- Trace a real run and mark each step as judgment or logic.
Use case
An illustrative scenario, not a real incident.
A prototype agent uses the model to format dates and pick file names. Moving those steps into code cuts its calls in half, and results stop varying.
See it in different ways
flowchart LR A["Single call with retrieval"] --> B["Workflow"] B --> C["Single agent"] C --> D["Multi-agent system"]
stateDiagram-v2 [*] --> SingleCall SingleCall --> Workflow: steps are known Workflow --> Agent: path depends on findings Agent --> MultiAgent: evals show overflow or specialist failures MultiAgent --> [*]
flowchart TD
Q{"Does the step need judgment?"}
Q -->|"No"| A0["Plain code"]
Q -->|"Yes"| A1["Model"]Key point: If a step needs no judgment, make it code.
Why the other options are wrong
- A. Whether a larger model could finish in fewer turns than the current one does
- A larger model costs more per turn, and fewer turns are not assured.
- B. Whether users would accept a longer wait in exchange for slightly better output
- This accepts the problem rather than fixing it.
- D. Whether adding a second agent would let each of the steps run at the same time
- A second agent adds cost, and parallel steps may not even be possible.