AI ObservabilityIntermediate

Framework Feature Matrix

Promptfoo, DeepEval, RAGAS, LangSmith and Phoenix compared by what each is good at, with working examples and a way to choose.

By Amit UpadhyayPublished 13 Oct 202611 min

Evaluation frameworks provide structured metrics and tooling for measuring LLM pipeline quality. RAGAS specialises in RAG evaluation: faithfulness (are claims grounded in context?), answer relevance (does the answer address the question?), context recall (did retrieval find relevant docs?). DeepEval provides a pytest-like interface for unit testing LLM outputs. Azure AI Evaluation adds Microsoft-specific safety metrics. Choose based on your pipeline type and hosting environment.

The pipeline

Each stage in order — click any step to read what it does.

Interactive Pipeline Flow

UnderstandImplementOptimise

🔍 RAGAS — RAG metrics

RAGAS (RAG Assessment): evaluate RAG pipelines with faithfulness, answer_relevance, context_recall, context_precision. Results 0-1. Uses an LLM internally to compute metrics.

Architecture diagrams

RAGAS evaluation metrics for RAG pipelines.

Loading diagram...

Key design decisions

The trade-offs worth knowing before you build this.

🔍RAGAS works best with GPT-6.1 Sol as the judge model

RAGAS uses an LLM internally to decompose answers and evaluate statements. GPT-6.1 Sol produces more accurate evaluations than weaker models. Using a small, cheap model as a judge can give misleading scores — the evaluator itself needs to be strong.

🧪DeepEval makes LLM testing feel like pytest

DeepEval imports into pytest. Each test case is a parametrized test over your dataset. Familiar CI integration — tests appear in pytest reports, CI passes/fails on threshold violations.

🔧Build domain-specific metrics for specialized applications

Generic metrics (faithfulness, relevance) miss domain-specific quality signals. Medical chatbot: is the response safe and risk-appropriate? Code assistant: does the code run? Financial: are all numbers accurate? Define custom judge prompts for your domain.

Next: Animate →

evaluation Promptfoo DeepEval RAGAS LangSmith

← PreviousOffline Evals: The Regression Suite for LLM Features

Continue reading

All AI Observability articles →

AI Observability11 min

Offline Evals: The Regression Suite for LLM Features

Build an evaluation dataset from real use, pick the cheapest grader that works, measure the noise floor, and turn the suite into a release gate.

AI Observability10 min

Token Tracking and Cost Management

Monitor token usage and costs — per-request tracking, budgets, cost allocation across teams, and the optimization strategies that shrink the bill.

AI Observability14 min

Phoenix: Open-Source LLM Observability

Arize's open-source, OTel-native LLM observability stack — traces, eval workflows, and prompt experiments, self-hosted with no data leaving your VPC.

Discussion