Token tracking connects LLM usage to cost and performance. Every LLM call has input tokens (prompt) + output tokens (completion) with different per-token pricing. Track at feature/team/model granularity. Prompt caching (OpenAI, Anthropic) reduces cost for repeated context. Context window management (truncation, summarisation) balances cost and quality. Cost attribution enables informed decisions about model selection and optimization.
Designing prompts to maximise cache hit rate.
RAG pipelines re-send the same system prompt and retrieved documents every call. Structure prompts with stable content first (system prompt, instructions, documents) and variable content last. The stable prefix is cached. 50-90% cost reduction on cached tokens.
Without per-feature tagging, you see one big LLM bill with no attribution. With tags (feature, team, model), you can identify the most expensive features, right-size model choices, and build team-level budgets.
gpt-4o-mini costs ~20× less than gpt-4o and handles classification, simple Q&A, and extraction well. Reserve gpt-4o for complex reasoning, nuanced generation, and high-stakes decisions. Right-sizing model can reduce LLM costs by 60%.
Sign in to share your feedback and join the discussion.