In this article
Context
We need an observability platform to trace LLM calls, evaluate output quality, debug RAG retrieval, and monitor production AI quality over time. Without observability, we are flying blind on LLM quality, cost, and reliability.
Options Evaluated
LangSmith (LangChain)
Pros
- +Seamless tracing for LangChain/LangGraph-based workflows
- +Prompt Hub for versioned prompt management
- +Eval datasets and automated evaluation workflows
- +Production monitoring with online evaluations
Cons
- −Best value when using LangChain ecosystem — we use direct SDK calls
- −Paid tiers required for teams (free tier limited to personal use)
- −Data leaves our infrastructure (SaaS only)
Arize Phoenix (open-source)
Pros
- +Open-source — self-host on our infrastructure (no data egress)
- +Native OpenTelemetry ingestion — not locked to any framework
- +Excellent RAG retrieval quality analysis and embedding drift visualisation
- +Free for self-hosted deployments
Cons
- −Requires hosting and maintaining the Phoenix instance
- −Fewer pre-built integrations than LangSmith
- −UI less polished than commercial alternatives
Custom OpenTelemetry + Grafana
Pros
- +Maximum flexibility and data ownership
- +Already using Grafana for infra monitoring — unified dashboard
- +No per-seat pricing
Cons
- −Significant build effort — must instrument all AI calls manually
- −No AI-specific analysis features out of the box (need custom dashboards)
- −Maintenance burden ongoing
Decision
We chose Arize Phoenix (self-hosted) as our primary AI observability platform, combined with OpenLLMetry for automatic instrumentation. Phoenix's open-source model keeps our trace data in our infrastructure (important for content privacy), its RAG analysis features match our primary use case, and OTel compatibility means we're not locked in. LangSmith free tier supplements for prompt versioning.
Consequences
- •Self-host Phoenix on a small Fly.io or Railway instance
- •Add @arizeai/openinference-instrumentation-openai for automatic trace capture
- •Configure Phoenix as the OTLP trace receiver endpoint in all services
- •Build weekly quality review process using Phoenix trace analysis
- •Track: faithfulness scores, context retrieval quality, latency p50/p95, cost per query