Token Tracking and Cost Management

The Invoice Is One Number. You Need Many.

Tokens are the unit of cost, latency, and capacity for every LLM you call. Without token-level telemetry you cannot answer the three questions that come up every week in production: which tenant is most expensive, which feature is regressing on cost, and which prompt is the biggest line item?

What to Count

• input_tokens — what you sent.
• output_tokens — what came back.
• cached_input_tokens — billed at a discount that varies by provider and model (often 50 to 90% off the input price).
• reasoning_tokens — hidden reasoning on reasoning and thinking models. Reported separately, but already counted inside the output tokens by OpenAI and Anthropic.
• tool_call_tokens — tool schemas in input + tool args in output. Often the silent majority.

Token counts come back in every provider response — read them from the response, do not estimate from tiktoken after the fact unless you have to.

flowchart LR
    R[Response] --> U["usage{ input, output,<br/>cached, reasoning }"]
    U --> C[cost = Σ tokens × price by class]
    C --> M[Emit metric:<br/>llm.tokens / llm.cost.usd]
    M --> D[Dashboards by:<br/>tenant, endpoint, model, prompt_id]
      

The Cost Formula

// Works on a normalised usage record (see "Normalise Usage Per Provider" below)
function computeCost(u: NormalisedUsage, price: ModelPrice): number {
  return (
    u.uncachedInput * price.input
    + u.cacheRead * price.cachedInput
    + u.cacheWrite * (price.cacheWrite ?? price.input)
    + u.output * price.output        // already includes reasoning tokens: do not add them again
  ) / 1_000_000;
}

Keep a single source-of-truth price table in code, versioned with the date the price went into effect. Vendors change prices quietly; you do not want this on a wiki.

Normalise Usage Per Provider

The providers report usage in shapes that look alike and add up differently, and that is where most cost dashboards go wrong. The two differences that matter:

• Cached input. OpenAI reports cached tokens as a subset of the input count (input_tokens_details.cached_tokens on the Responses API). Anthropic reportsinput_tokens as the uncached part only, with cache_read_input_tokens andcache_creation_input_tokens alongside it. Subtract on one, add on the other.
• Cache writes. Anthropic charges a premium for writing to the cache (more than the normal input price), so a prompt that is cached but rarely reused can cost more than no caching at all.

type NormalisedUsage = { uncachedInput: number; cacheRead: number; cacheWrite: number; output: number; reasoning: number };

function fromOpenAI(u: OpenAIUsage): NormalisedUsage {
  const cached = u.input_tokens_details?.cached_tokens ?? 0;
  return {
    uncachedInput: u.input_tokens - cached,   // cached is part of input_tokens
    cacheRead: cached,
    cacheWrite: 0,
    output: u.output_tokens,                  // includes reasoning
    reasoning: u.output_tokens_details?.reasoning_tokens ?? 0,
  };
}

function fromAnthropic(u: AnthropicUsage): NormalisedUsage {
  return {
    uncachedInput: u.input_tokens,            // already excludes cache reads and writes
    cacheRead: u.cache_read_input_tokens ?? 0,
    cacheWrite: u.cache_creation_input_tokens ?? 0,
    output: u.output_tokens,                  // includes thinking
    reasoning: 0,                             // not broken out separately
  };
}

Normalise at the point where you receive the response, store the normalised record, and compute cost only from that. Then reconcile once a month: sum your computed cost per provider and compare it with the invoice. A gap of a few percent is rounding and pricing changes; a gap of twenty means a mapping is wrong, and it is much better to learn that from a reconciliation than from finance.

Dimensions That Pay Off

Always emit token + cost metrics with at least these dimensions:

• model (with version) — to spot silent fallback to a pricier model.
• endpoint / feature — e.g. support_chat, summary_v2.
• tenant_id (or hashed) — for per-customer attribution.
• prompt_template_id + version — to A/B prompt cost changes.
• cache_hit — boolean; tracks whether your cache strategy is actually working.

Budgets and Hard Limits

Track per-tenant rolling spend (15-minute window + daily total) in Redis. Three thresholds:

• Soft warn at 80% of daily quota — email the tenant's admin and your ops channel.
• Throttle at 100% — switch to a cheaper model or queue requests.
• Hard stop at 150% — return a typed error explaining the cap; this catches runaway loops fast.

Caching: Half the Cost, If You Set It Up Right

• Put the stable system prompt + few-shot examples + retrieved context at the top. Both Anthropic and OpenAI cache by prefix.
• Anthropic requires explicit cache_control: { type: "ephemeral" } blocks; OpenAI auto-caches when the prefix > 1024 tokens.
• Track cache_hit_rate = cached_tokens / input_tokens per endpoint — below 50% on a stable prefix means your prompt is unstable upstream.

Failure Modes

• Reasoning tokens ignored — reasoning models can produce many more tokens than you see in the message. They are billed as output, so they are in your cost as long as you use the reported output count; track reasoning_tokens separately to see where the spend goes.
• Streamed responses not totaled — some SDKs only emit usage on the final chunk; make sure your wrapper waits for it.
• Estimating with length / 4 — wrong by 30–60% for non-English text and code.
• Mixing currencies — Azure prices in regional pricing; convert once at ingest, store in USD.

Production Checklist

• Read usage from the provider response; do not estimate.
• Versioned price table in code; updated within 24h of vendor changes.
• Emit llm.tokens and llm.cost.usd with dimensions {model, endpoint, tenant, prompt_id, cache_hit}.
• Three-tier budget: warn, throttle, hard-stop.
• Track cache hit rate per endpoint as a first-class KPI.

Key Takeaways

• Token tracking is the foundation of cost control — you cannot optimise what you cannot decompose.
• Dimensions matter more than the number; per-tenant + per-prompt is the minimum useful split.
• Caching, budgets, and reasoning-token awareness are the three biggest wins on the bill.