Every LLM call has a measurable cost: input tokens × input price + output tokens × output price. Understanding token accounting, prompt caching, model tiering, and batch APIs is the difference between sustainable margins and a runaway AWS bill.
Per-request cost path: quota → cache → route → meter → bill.
Move stable system prompts and few-shot examples to the prefix. Anthropic gives 90% off cached, OpenAI 50%.
gpt-4o-mini handles ~80% of tasks at 1/15th the cost. Only escalate to gpt-4o when you have a quality signal.
OpenAI / Anthropic batch APIs give 50% discount with 24h SLA. Use for offline embeddings, evals, summaries.
Long system prompts cost on every call. Refactor to shorter directives + few-shot. Summarize chat history > 10 turns.
Meter and attribute. A 99th-percentile user can be 100x average — without slicing you'll never spot it.
Per-user, per-org daily token limits. Prevents abuse + bugs from torching your monthly budget overnight.
Sign in to share your feedback and join the discussion.