A new prompt raises success but doubles cost. What is the right next step?
Why C is the answer
Compare it against the agreed cost and quality targets
Tuning trades off metrics against targets.
In detail
- Compare against the agreed targets.
- Agree targets in advance.
Use case
An illustrative scenario, not a real incident.
A prompt raises success but doubles cost, so the team compares it with agreed targets.
See it in different ways
flowchart LR A["Scenario set"] --> B["Run agent"] B --> C["Score task success"] B --> D["Score tool choices"] B --> E["Measure cost and latency"] C --> F["Compare with baseline"] D --> F E --> F
mindmap
root((Agent evaluation))
Outcome
Task success
Path
Tool accuracy
Loops
Cost
Tokens
Latency
Safety
Regression setflowchart TD
Q{"Does it meet both targets?"}
Q -->|"Yes"| A0["Adopt it"]
Q -->|"No"| A1["Keep tuning"]Key point: Tune against targets.
Why the other options are wrong
- A. Reject it, since costs may never change at all
- Costs can change.
- B. Remove cost tracking from the evaluation report
- That hides the trade-off.
- D. Ship it, since success rate is the only metric
- Success is not the only metric.