Token Tracking and Cost Management

Problem Context

Tokens are the unit of cost, latency, and capacity for every LLM you call. Without token-level telemetry you cannot answer the three questions that come up every week in production: which tenant is most expensive, which feature is regressing on cost, and which prompt is the biggest line item?

What to Count

• input_tokens — what you sent.
• output_tokens — what came back.
• cached_input_tokens — billed at a discount (Anthropic 90% off, OpenAI ~50% off).
• reasoning_tokens — hidden chain-of-thought on o-series / thinking models.
• tool_call_tokens — tool schemas in input + tool args in output. Often the silent majority.

Token counts come back in every provider response — read them from the response, do not estimate from tiktoken after the fact unless you have to.

flowchart LR
    R[Response] --> U[usage{ input, output,<br/>cached, reasoning }]
    U --> C[cost = Σ tokens × price by class]
    C --> M[Emit metric:<br/>llm.tokens / llm.cost.usd]
    M --> D[Dashboards by:<br/>tenant, endpoint, model, prompt_id]
      

The Cost Formula

function computeCost(usage: Usage, model: ModelPrice): number {
  return (
    (usage.input_tokens - (usage.cached_input_tokens ?? 0)) * model.input
    + (usage.cached_input_tokens ?? 0) * model.cached_input
    + usage.output_tokens * model.output
    + (usage.reasoning_tokens ?? 0) * model.output  // reasoning bills as output
  ) / 1_000_000;
}

Keep a single source-of-truth price table in code, versioned with the date the price went into effect. Vendors change prices quietly; you do not want this on a wiki.

Dimensions That Pay Off

Always emit token + cost metrics with at least these dimensions:

• model (with version) — to spot silent fallback to a pricier model.
• endpoint / feature — e.g. support_chat, summary_v2.
• tenant_id (or hashed) — for per-customer attribution.
• prompt_template_id + version — to A/B prompt cost changes.
• cache_hit — boolean; tracks whether your cache strategy is actually working.

Budgets and Hard Limits

Track per-tenant rolling spend (15-minute window + daily total) in Redis. Three thresholds:

• Soft warn at 80% of daily quota — email the tenant's admin and your ops channel.
• Throttle at 100% — switch to a cheaper model or queue requests.
• Hard stop at 150% — return a typed error explaining the cap; this catches runaway loops fast.

Caching: Half the Cost, If You Set It Up Right

• Put the stable system prompt + few-shot examples + retrieved context at the top. Both Anthropic and OpenAI cache by prefix.
• Anthropic requires explicit cache_control: { type: "ephemeral" } blocks; OpenAI auto-caches when the prefix > 1024 tokens.
• Track cache_hit_rate = cached_tokens / input_tokens per endpoint — below 50% on a stable prefix means your prompt is unstable upstream.

Failure Modes

• Reasoning tokens ignored — o-series / thinking models can burn 5–20× tokens you never see in the message. Always add reasoning_tokens to your cost.
• Streamed responses not totaled — some SDKs only emit usage on the final chunk; make sure your wrapper waits for it.
• Estimating with length / 4 — wrong by 30–60% for non-English text and code.
• Mixing currencies — Azure prices in regional pricing; convert once at ingest, store in USD.

Production Checklist

• Read usage from the provider response; do not estimate.
• Versioned price table in code; updated within 24h of vendor changes.
• Emit llm.tokens and llm.cost.usd with dimensions {model, endpoint, tenant, prompt_id, cache_hit}.
• Three-tier budget: warn, throttle, hard-stop.
• Track cache hit rate per endpoint as a first-class KPI.

Key Takeaways

• Token tracking is the foundation of cost control — you cannot optimise what you cannot decompose.
• Dimensions matter more than the number; per-tenant + per-prompt is the minimum useful split.
• Caching, budgets, and reasoning-token awareness are the three biggest wins on the bill.