Start With One Good Model
At ten requests a day, pick one good model and move on. Nobody will notice the bill. The techniques below are for the day a feature reaches real volume, when the same choice is multiplied by ten thousand.
"Which model should we use" almost always gets answered as if it were purely a quality question. It isn't. Every model choice is a point on a cost-performance curve, and the frontier model that wins your internal eval by a few points can be several times more expensive per call than the option that would have shipped the feature just fine. Nobody notices at ten requests a day. Everybody notices at ten thousand.
- Your OpenAI or Anthropic bill tripled after a feature went from beta to general availability, and finance is asking why
- You're running the frontier model on a classification task that a much cheaper model would nail just as often
- Someone suggested "just use the cheap model for everything" and you're not sure where that quietly breaks
- You need to forecast next quarter's LLM spend for a feature that doesn't exist in production yet
Frontier vs Mini/Flash: What You're Actually Trading
Every major lab now ships two tiers. A frontier tier — GPT-6 Astra or GPT-6.1 Sol-class, Claude Fable 5.2 or Opus 5.5-class, Gemini 4 Argon — built for the hardest reasoning, the longest context, the most ambiguous instructions. And a smaller, faster tier — GPT-6 Luna, Gemini Flash-Lite, Claude Haiku 5.5-class — priced one to two orders of magnitude cheaper per token, with latency to match. The mini tier isn't a worse version of the frontier model in every way that matters. It's tuned for a different job: high-volume, lower-ambiguity tasks where the frontier model's extra reasoning capacity mostly goes unused. Classification, extraction, short rewrites, simple routing decisions — a mini model handles these at a quality gap that's often smaller than the price gap suggests. Multi-step reasoning, long-document synthesis, anything where getting it subtly wrong is expensive to catch later — that's where the frontier tier earns its price.
| Frontier tier | Mini / Flash tier | |
|---|---|---|
| Relative cost per token | Highest in the lineup | Roughly an order of magnitude lower |
| Latency | Slower, especially with reasoning enabled | Often sub-second time-to-first-token |
| Best for | Ambiguous, multi-step, high-stakes tasks | High-volume, well-defined, low-ambiguity tasks |
| Context window | Usually largest in the family | Often smaller, sometimes matched |
Check it: List your call types and mark each one as ambiguous or well-defined. Run the largest well-defined one on the mini tier against labelled examples, and compare the pass rate.
Model Routing and Cascading
Once you accept that no single model is right for every call in your system, routing follows naturally. Cascading is the specific pattern worth knowing: try the cheap model first, and only escalate to the expensive one when the cheap model's output looks unreliable. The trick is defining "looks unreliable" in a way that's cheap to check — a confidence score if the model exposes one, a schema validation failure, a length or format check, or a cheap secondary classifier trained specifically to flag "this needs the bigger model." Get that check wrong and you either escalate everything (no savings) or escalate nothing (silent quality loss), so it's worth treating as a real component with its own tests, not an afterthought bolted onto the routing function.
flowchart TD
Req["Incoming request"] --> Cheap["Cheap/fast model
(mini or flash tier)"]
Cheap --> Conf{"Confident?
(schema valid, score high,
task in scope)"}
Conf -->|Yes| Return["Return result"]
Conf -->|No| Expensive["Escalate to frontier model"]
Expensive --> Return
style Cheap fill:#059669,color:#fff,stroke:#047857
style Expensive fill:#dc2626,color:#fff,stroke:#b91c1c
style Return fill:#4f46e5,color:#fff,stroke:#4338ca
async function escalate(ticket: string): Promise<Classification> {
const frontier = await callModel('gpt-6.1-sol', buildPrompt(ticket));
// The expensive answer can fail validation too, so check it before returning
if (!isValidSchema(frontier.output)) throw new Error('Escalated classification failed validation');
return frontier.output;
}
async function classifyTicket(ticket: string): Promise<Classification> {
// Long tickets go straight to the bigger model, so check this before the cheap call
if (ticket.length > 4000) return escalate(ticket);
const cheap = await callModel('gpt-6-luna', buildPrompt(ticket));
// Escalate on structural failure or low self-reported confidence —
// not on vibes. Define this threshold from labeled data, not intuition.
const needsEscalation = !isValidSchema(cheap.output) || cheap.confidence < 0.7;
return needsEscalation ? escalate(ticket) : cheap.output;
}In practice, a well-tuned cascade routes the large majority of traffic to the cheap model and reserves the expensive one for the tail that actually needs it — which is exactly the shape of most production workloads: mostly routine, occasionally hard.
Check it: Log the share of requests that escalate. Close to all of them means no saving. Close to none, on a mixed workload, means a quality loss you haven't found yet.
Batch API Pricing
Most major providers offer an asynchronous batch endpoint: submit a large set of requests, get results back within a window (often hours rather than seconds), at a meaningful discount off synchronous pricing. This fits offline evaluation runs, bulk classification of a historical dataset, nightly enrichment jobs, anything where nobody is staring at a spinner waiting for the answer. It does not fit user-facing chat, or anything where a human is on the other end expecting a response inside a few seconds. The mistake is assuming batch pricing applies broadly when it actually applies to a narrower slice of your workload than the discount makes tempting — check which parts of your pipeline can genuinely tolerate the latency before restructuring anything around it.
Check it: List the jobs where no person is waiting for the answer. Those are the only candidates for batch.
Prompt Caching as a Cost Lever
If your requests share a large, stable prefix — a long system prompt, a set of few-shot examples, a document you're asking repeated questions about — prompt caching (covered in more depth in the context-windows article) discounts the repeated portion on every call after the first. It stacks cleanly with routing and cascading: cache the shared prefix once, then route the variable part of the request through whichever tier makes sense.
Worked Example: Support-Ticket Triage
Say you're triaging 10,000 support tickets a day — category, priority, routing — averaging roughly 500 input tokens and 150 output tokens per ticket. These numbers are illustrative, not current pricing; plug in your provider's actual rate when you do this for real.
| Approach | Illustrative $/M tokens (in / out) | Daily tokens | Illustrative daily cost |
|---|---|---|---|
| Frontier model, every ticket | $5 / $15 (illustrative) | 5M in / 1.5M out | ~$47.50 |
| Mini model, every ticket | $0.30 / $1.20 (illustrative) | 5M in / 1.5M out | ~$3.30 |
| Cascade: mini runs on every ticket, ~12% then also go to frontier | blended | 5M in / 1.5M out, plus 0.6M in / 0.18M out on frontier | ~$9.00 |
The mini-only run is roughly fourteen times cheaper than frontier-only in this illustration. The cascade costs $3.30 plus 12% of $47.50, or about $9.00 a day. That's well above mini-only and still about a fifth of frontier-only, and it keeps the escalation path for the tickets that are actually ambiguous — angry customers, multi-issue tickets, anything where a misrouted ticket costs more in support time than the model call ever would. If you can route those 12% directly without trying the mini model first, it drops to about $8.60. Multiply the daily numbers by 30 for a monthly estimate, and notice how much the answer depends on that escalation rate. It's worth measuring on real traffic before committing to a number in a planning doc.
1. Optimizing cost per token instead of cost per good result
A model that's 70% cheaper with a retry on 1 in 5 calls still comes in at about 36% of the original token cost, so retries alone don't erase the saving. What can erase it is the downstream cost of the failures nobody retried. Measure cost against outcomes that passed your quality bar, not raw token spend.
2. Routing on vibes instead of a measured threshold
"The cheap model seems fine for this" is not a routing policy. Build the escalation trigger from labeled examples where you know the right answer, and revisit the threshold when either model gets updated — providers change model behavior under a fixed name more often than the release notes suggest.
3. Putting batch pricing on the critical path
Batch discounts are real, but the multi-hour turnaround makes it a poor fit for anything a user is waiting on. Keep batch scoped to genuinely asynchronous work, or you'll end up building a synchronous-feeling UI on top of an asynchronous backend and fighting that mismatch forever.
4. Forecasting cost off the sticker price, not the traffic shape
Per-token pricing tells you almost nothing until you multiply it by your actual token distribution, including the long tail of unusually large requests that skew the average. Model the P95 request, not just the median, before you commit a number to a budget.
Try It on Your Own System This Week
Three checks on a feature with a real bill, about an hour in total:
- Work out the cost per result that passed your quality bar. Token spend alone hides the failures nobody retried.
- Find the P95 request size as well as the median. The long tail of large requests moves the average.
- Ask how the escalation threshold was chosen. If nobody measured it, measure it on labelled examples.
Key Takeaways
- The mini/flash tier is not a worse frontier model — it's tuned for a different job. Match the tier to task ambiguity, not to habit.
- Cascading routes the boring majority of traffic cheaply and reserves the expensive model for the tail that actually needs it — but only if the escalation trigger is measured, not guessed.
- Batch pricing is a discount on patience, not a universal lever. It fits offline and bulk work; it doesn't fit anything with a human waiting.
- Cost estimates are only as good as the traffic distribution behind them. Model your real request mix, including the outliers, before you put a number in front of finance.