AI Wisdom
Learning Track

AI Observability

Tracing, evals, drift detection, cost telemetry, hallucination monitoring — keeping production AI systems honest.

14+
Topics
14
Published
100%
Free
Browse articles ↓

14 articlesin AI Observability

AI Observability
CustomerSupport agentToolsMCP serversRun traceOnline evals

Agent Observability: When the Trace Says Everything Worked

Intermediate

Agents fail with healthy infrastructure. One trace per run, OpenTelemetry GenAI spans on a schema that is still moving, cost per task, and evals that catch what dashboards miss.

Updated 5 min
Read →
AI Observability
Your teamEval frameworkRAG systemDataset in the repoYour labelsRetrieval scoreGeneration score

LLM Evaluation Frameworks Compared

Intermediate

Promptfoo, DeepEval, RAGAS, LangSmith and Phoenix compared by what each is good at, with working examples and a way to choose.

Updated 11 min
Read →
AI Observability
DeveloperPull requestEval suiteYour systemGradersMerge gate

Offline Evals: The Regression Suite for LLM Features

Intermediate

Build an evaluation dataset from real use, pick the cheapest grader that works, measure the noise floor, and turn the suite into a release gate.

Updated 11 min
Read →
AI Observability
CustomersYour productShared model clientModelUsage recordsProvider invoicePrompt cacheTenant budgets

Token Tracking and Cost Management

Intermediate

Monitor token usage and costs — per-request tracking, budgets, cost allocation across teams, and the optimization strategies that shrink the bill.

Updated 12 min
Read →
AI Observability
DeveloperYour pipelinePhoenixDatasetExperimentEvals as codePostgres and backups

Phoenix: Open-Source LLM Observability

Intermediate

Arize's open-source, OTel-native LLM observability stack — traces, eval workflows, and prompt experiments, self-hosted with no data leaving your VPC.

Updated 14 min
Read →
AI Observability
CustomerAssistantModelOutput checksCustomers and systemsRetrieved textSafe fallback

Validating LLM Outputs

Intermediate

Validate LLM outputs — schema conformance, factuality checks against source documents, toxicity detection, and structured output parsing before serving users.

Updated 14 min
Read →
AI Observability
UserWeb appAPIVector storeModelCollectorTrace backend

OpenTelemetry for LLM Applications

Intermediate

Instrument LLM apps with OpenTelemetry — the gen-ai semantic conventions, distributed tracing across RAG pipelines, and vendor-neutral export to any backend.

Updated 14 min
Read →
AI Observability
UsersAssistantModelCode checksJudge modelHuman labelsOffline dataset

Online Evals: Production Quality Monitoring

Intermediate

Production-time evaluation — implicit user feedback, A/B testing, and real-time LLM-as-judge scoring for continuous quality monitoring after ship.

Updated 12 min
Read →
AI Observability
UsersAssistantModelTracesCost by tenantOnline evalsAlerts

LLM Monitoring in Production

Intermediate

Production dashboards for LLM apps — latency, error rates, token usage, and quality drift detection, wired into the alerting you already trust.

Updated 12 min
Read →
AI Observability
UserAssistantRetrievalGuardrailsModelPrompt cacheSmall model

LLM Latency: Measurement and Optimization

Intermediate

Measure and optimize LLM response times — Time to First Token, tokens per second, P99 tail latency, and the streaming/caching levers that actually move the needle.

Updated 11 min
Read →
AI Observability
CustomerAssistantRetrievalModelTracesDatasetExperiments

LangSmith: LLM Observability Platform

Intermediate

LangChain's observability platform — distributed tracing, dataset curation, evaluation suites, and the prompt playground, for LangChain and framework-agnostic apps alike.

Updated 13 min
Read →
AI Observability
UserSize and rateRedactionInjection checkTopic gateModelRetrieved text

LLM Input Validation Strategies

Intermediate

Validate and sanitize LLM inputs — prompt injection detection, PII filtering, and content moderation that run before the model call to prevent misuse and cut cost.

Updated 12 min
Read →
AI Observability
Public usersAssistantInput classifierModelOutput classifierSupport resourcesAudit log

Content Safety for AI Applications

Intermediate

Detect and filter harmful, biased, or inappropriate AI content — moderation APIs, classifiers, and the multi-layer approach from input filtering to human review.

Updated 11 min
Read →
AI Observability
YouTest casesYour featureCode checksJudge modelRelease gate

LLM Evaluation Beyond Vibes

Systematic approaches to evaluating LLM outputs — automated metrics, human evaluation frameworks, regression testing, and building evaluation pipelines.

Updated 9 min
Read →