Problem Context

LangSmith is the observability and evaluation product from the LangChain team. Even if you don't use LangChain, it is one of the easiest ways to get traces, prompt versioning, and structured eval pipelines wired together — because the SDK is just OTLP under the hood and the UI is purpose-built for LLM debugging (nested chains, tool calls, retrievals).

🤔 Sound familiar?
  • Your traces in a general-purpose APM don't show nested LLM/tool/retrieval calls cleanly
  • You want prompt experiments without building an internal eval platform
  • You need to ship online + offline evals and your team is one engineer

What LangSmith Actually Gives You

  • Traces — nested runs (chain → LLM → tool → retrieval) with token, cost, latency per node.
  • Datasets — eval sets you can build by importing traces from production.
  • Evaluators — built-in (exact match, embedding similarity) and custom LLM-as-judge.
  • Prompt hub — versioned prompts pulled at runtime; A/B and rollback without redeploys.
  • Experiments — run a dataset through different prompt/model combos, compare metrics.

flowchart LR
    P[Production traces] --> D[Curate dataset]
    D --> E[Run experiment:<br/>prompts × models]
    E --> S[Score: judge + heuristics]
    S --> C[Compare vs baseline]
    C --> A[Approve → ship prompt]
    A --> P

Minimal Setup (Any Python/TS Code)

import { traceable, wrapOpenAI } from 'langsmith/wrappers';
import OpenAI from 'openai';

const openai = wrapOpenAI(new OpenAI()); // auto-traces every call

const answer = traceable(
  async (q: string) => {
    const docs = await retrieve(q);
    const r = await openai.chat.completions.create({
      model: 'gpt-4o-mini',
      messages: [
        { role: 'system', content: SYS },
        { role: 'user', content: `Context:\n${docs}\n\nQ: ${q}` },
      ],
    });
    return r.choices[0].message.content;
  },
  { name: 'support_answer', project_name: 'support-agent-prod' },
);

LANGSMITH_API_KEY + LANGSMITH_PROJECT environment variables and you have traces with nested spans, token usage, and cost out of the box. The traceable decorator is the unit of nesting.

Datasets from Production

The killer workflow: in the LangSmith UI, filter traces (e.g. low thumbs-up, high latency, specific tenant), select 50–500, and add them to a dataset. That dataset is now your offline eval set — built from real failures, not imagined ones.

Evaluators You Will Actually Use

  • String distance / exact match — only useful for closed-form answers.
  • Embedding similarity — for "close enough" semantic comparison.
  • LLM-as-judge (custom) — the workhorse. Rubrics for faithfulness, relevance, safety.
  • JSON schema validity — for structured outputs.
  • Trajectory evaluators — for agents: did the tool sequence make sense?
from langsmith.evaluation import evaluate
from langsmith.evaluation.evaluator import RunEvaluator

def faithfulness(run, example):
    score = judge_llm(run.outputs["answer"], run.inputs["context"])
    return {"key": "faithfulness", "score": score}

evaluate(
    lambda x: my_chain.invoke(x),
    data="support-eval-v3",
    evaluators=[faithfulness],
    experiment_prefix="prompt-v17",
)

Prompt Hub: Decouple Prompts from Code

Prompts live in the hub with semver. Pull at startup or per-request with caching:

import { pull } from 'langchain/hub';
const promptTemplate = await pull('amit/support-system-prompt:v17');

Result: product/PM can edit prompts and roll back without a deploy; experiments compare v17 vs v18 on the same dataset; rollbacks are a metadata flip.

Self-Host vs Cloud

  • Cloud — fast to start; data leaves your boundary; pricing scales with traces stored.
  • Self-hosted (LangSmith Enterprise) — your VPC, your retention, more ops work. Required for many regulated industries.
  • OTLP export — LangSmith can also act as an OTel target, and traces can be mirrored to your existing APM.

Failure Modes

  • Sampling 100% in prod — bills and ingestion lag balloon. Sample 10–20% steady-state, 100% on errors/feedback.
  • Storing PII in traces — turn off message-content capture in regulated contexts or redact via a wrapper.
  • Eval datasets that never refresh — your test set ages out of reality. Schedule a monthly curate-from-prod job.
  • Judge prompt drift — version your evaluator prompts in the hub too.

Production Checklist

  • traceable on every top-level entry point; wrapOpenAI/wrapAnthropic at the SDK boundary.
  • Sample 10–20% steady, 100% on errors and negative feedback.
  • Datasets curated monthly from real traces, not synthetic.
  • Prompts in the hub with semver; rollback path tested.
  • Self-host if your data residency or compliance requires it; otherwise cloud + redaction.

Key Takeaways

  • LangSmith is the fastest path from "no observability" to "traces + datasets + evals + prompt versioning".
  • Curating datasets from production traces is the workflow that makes everything else useful.
  • Prompt hub + experiments turn prompt engineering into a measurable, reversible process.