LLM Monitoring in Production

Problem Context

LLM applications fail differently from traditional services. A 200 response can still be wrong, hallucinated, biased, or off-policy. CPU and memory tell you nothing. You need observability designed for non-deterministic, expensive, latency-sensitive calls where the "bug" is often a single tokens-different drift in the output.

The Four Pillars of LLM Observability

• Traces — one span per LLM call, tool call, retrieval, parsing step. Linked into a session/trace tree.
• Metrics — latency percentiles, token counts, cost, error rate, cache hit rate. Per model, per endpoint, per tenant.
• Quality signals — online evals (faithfulness, relevance), thumbs-up/down, user-reported issues, guardrail block rate.
• Cost telemetry — tokens × price × user/tenant attribution. The metric that makes you fix things first.

flowchart LR
    Q[Request] --> SP[Trace start]
    SP --> RT[Retrieval span]
    SP --> LLM[LLM span<br/>tokens, cost, latency]
    SP --> EV[Eval span<br/>faithfulness, relevance]
    SP --> SE[Trace end]
    SE --> EXP[OTLP exporter]
    EXP --> P[Phoenix / LangSmith /<br/>Datadog / App Insights]
      

Span Attributes That Earn Their Keep

Every LLM span should carry, at minimum:

gen_ai.system           "openai" | "anthropic" | "azure_openai"
gen_ai.request.model    "gpt-4o-2024-08-06"
gen_ai.request.temperature 0.2
gen_ai.usage.input_tokens 1248
gen_ai.usage.output_tokens 312
gen_ai.usage.cached_tokens 1024
gen_ai.response.finish_reason "stop" | "length" | "tool_calls"
gen_ai.cost.usd         0.0021
user.id / tenant.id     (hashed)
prompt.template.id      "support_v3"
prompt.template.version "3.2.1"

The OpenTelemetry gen_ai.* semantic conventions are the emerging standard — adopt them now so your vendor choice is reversible.

Sampling: 100% Traces Are a Trap

Tracing every call at scale is expensive in storage and slow at query time. Sane defaults:

• 100% sampling for errors, guardrail blocks, and negative user feedback.
• 100% for tenants on enterprise tier (you're accountable to them).
• Tail-based sampling for the rest: 10% baseline, boost to 100% if latency > p99 or cost > threshold.
• Always store the full prompt+response for sampled traces — without it the trace is useless.

Cost Telemetry Is the First Dashboard

Build the cost dashboard before anything else. Group by (model, endpoint, tenant), day-over-day. The moment a tenant 2× their spend overnight or a new endpoint quietly burns $500/day, you need to see it. Cost is also the proxy that catches most quality regressions — retries spike before users notice.

Online Evals: The Quality Pulse

Offline evals (a fixed eval set in CI) catch regressions before shipping. Online evals catch the regressions that only happen on real traffic. Run a lightweight eval (faithfulness, relevance, schema validity) on 1–5% of production traffic and emit it as a metric. When it drops, you have a window of hours rather than days to react.

Vendor Landscape (2026)

• LangSmith — deepest integration with LangChain/LangGraph; strong eval workflows.
• Phoenix (Arize) — open-source, OTLP-native; great for self-hosted, model-agnostic.
• Langfuse — open-source + cloud; pragmatic prompt versioning + evals.
• Datadog LLM Observability / App Insights — if you already pay them for everything else, the cheapest path to one pane of glass.
• Helicone / Traceloop / Honeyhive — viable niche options; pick on integration surface, not feature list.

Failure Modes

• Storing raw prompts with PII — observability becomes a compliance liability. Hash or redact at the SDK level.
• No trace_id propagation — you can't tie a user complaint to a trace. Always thread trace_id through your API responses.
• Metrics without dimensions — "average latency" with no model/endpoint split is useless.
• Eval drift — your online eval prompt itself changes and skews the metric. Version the eval prompt.

Production Checklist

• OpenTelemetry with gen_ai.* attributes, one span per LLM call.
• Cost-by-tenant dashboard before anything else.
• Tail-based sampling with full prompt/response capture on the sampled set.
• Online evals on 1–5% of traffic, emitted as metrics.
• trace_id returned in every API response header for support workflows.

Key Takeaways

• LLM observability is traces + cost + quality, not just APM extended.
• Adopt gen_ai.* conventions so your tooling choice stays portable.
• Online evals close the loop that offline evals can't — ship both.