Phoenix: Open-Source LLM Observability

Problem Context

Phoenix is Arize's open-source LLM observability and evaluation stack. It is OTLP-native, runs locally or in your cluster, and gives you traces, eval workflows, and prompt experiments without sending data to a vendor. For teams that need self-hosted or want a free entry into LLM ops, Phoenix is the leading choice in 2026.

What Phoenix Gives You

• Trace viewer — nested runs with token/cost/latency, identical structure to LangSmith/Langfuse.
• OpenInference instrumentations — auto-trace OpenAI, Anthropic, LangChain, LlamaIndex, DSPy, MCP.
• Datasets & experiments — curate eval sets from traces, run experiments comparing prompts/models.
• Built-in evaluators — hallucination, QA correctness, relevance, toxicity. All open and inspectable.
• Prompt playground — iterate on a prompt against a dataset without leaving the UI.

flowchart LR
    APP[App] -->|OTLP| PX[Phoenix server]
    PX --> TR[Traces UI]
    PX --> DS[Datasets]
    DS --> EX[Experiment:<br/>prompt × model]
    EX --> EV[Evaluators]
    EV --> LB[Leaderboard]
      

Starting Phoenix Locally (Five Minutes)

# Run the server (Docker or pip)
docker run -p 6006:6006 -p 4317:4317 arizephoenix/phoenix:latest

# Or
pip install arize-phoenix
phoenix serve

from phoenix.otel import register
from openinference.instrumentation.openai import OpenAIInstrumentor

tracer_provider = register(
    project_name="support-agent",
    endpoint="http://localhost:4317",
)
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

# All openai calls are now traced; visit http://localhost:6006

OpenInference: The Convention

Phoenix uses the OpenInference semantic conventions (a superset of OTel gen_ai.*). The instrumentations are framework-aware: a LangChain run shows up as a single nested tree, a LlamaIndex query becomes a retrieval span, an MCP call is tagged as a tool. The traces are portable — any OTLP backend can ingest them.

Evals: Run Them as Code

import phoenix as px
from phoenix.evals import (
    HallucinationEvaluator,
    QAEvaluator,
    run_evals,
    OpenAIModel,
)

eval_model = OpenAIModel(model="gpt-4o-mini")
results = run_evals(
    dataframe=traces_df,           # exported from Phoenix
    evaluators=[
        HallucinationEvaluator(eval_model),
        QAEvaluator(eval_model),
    ],
    provide_explanation=True,
)
px.Client().log_evaluations(SpanEvaluations(eval_name="hallucination",
                                            dataframe=results["hallucination"]))

Eval scores attach to the spans in the UI — you can filter traces by "hallucination = high" and curate them straight into a dataset for the next round.

Experiments: The Phoenix Workflow

A Phoenix "experiment" runs a task function over a dataset, applies evaluators, and stores results next to the baseline. Typical use:

1. Curate dataset from production failure traces.
2. Define task: (input) => chain.invoke(input).
3. Run experiment with prompt v17 and v18 in parallel.
4. Compare scores in the UI; ship the winner.

Self-Hosting Reality Check

• Phoenix server is a single container — Postgres-backed for production (SQLite for local).
• Retention is your problem — set up archival and pruning jobs.
• RBAC and SSO are in the enterprise edition; OSS is open access on the VPC boundary.
• Scaling: high-throughput projects benefit from a dedicated ingest worker pool.

Phoenix vs LangSmith vs Langfuse

• Phoenix — best for self-hosted, model-agnostic, OTel-native.
• LangSmith — deepest LangChain integration; smoothest cloud experience; first-class prompt hub.
• Langfuse — open-source + cloud, strong prompt versioning, pragmatic feature set; good middle ground.

These tools are 80% the same. Pick on (a) data residency, (b) framework you already use, (c) team appetite for self-hosting.

Failure Modes

• SQLite in production — fine for dev, choking at scale. Move to Postgres before the first incident.
• No retention policy — disk fills, ingest stalls. Schedule prune jobs from day one.
• Evaluators using the same model as the app — bias toward agreement; use a different family.
• Instrumenting only the framework — your own business spans don't show up. Add explicit tracer.start_as_current_span at top-level handlers.

Production Checklist

• Phoenix container + Postgres + persistent volume.
• OpenInference instrumentation for every LLM SDK and framework you use.
• Curated datasets from production failure traces; experiments before shipping prompt changes.
• Retention + prune jobs; backup the Postgres volume.
• Mirror OTLP to your APM if you need one pane of glass with the rest of your services.

Key Takeaways

• Phoenix is the leading open-source choice for LLM tracing and evals — OTel-native, model-agnostic, self-hostable.
• The experiment workflow (dataset → prompt × model → score → ship) is the loop worth adopting regardless of vendor.
• Self-hosted does not mean zero-ops — plan Postgres, retention, and scaling early.