Promptfoo Workflow
Promptfoo, DeepEval, RAGAS, LangSmith and Phoenix compared by what each is good at, with working examples and a way to choose.
Create evaluation dataset with question/answer/contexts/ground_truth
Like preparing an exam answer sheet — question, student answer, reference material, and correct answer.
RAGAS needs 4 columns: question (input), answer (LLM output), contexts (retrieved docs), ground_truth (expected answer).
> Dataset created: 1 sample > columns: question, answer, contexts, ground_truth
Create evaluation dataset with question/answer/contexts/ground_truth
Like preparing an exam answer sheet — question, student answer, reference material, and correct answer.
RAGAS needs 4 columns: question (input), answer (LLM output), contexts (retrieved docs), ground_truth (expected answer).
> Dataset created: 1 sample > columns: question, answer, contexts, ground_truth
Continue reading
All AI Observability articles →Offline Evals: The Regression Suite for LLM Features
Build an evaluation dataset from real use, pick the cheapest grader that works, measure the noise floor, and turn the suite into a release gate.
Token Tracking and Cost Management
Monitor token usage and costs — per-request tracking, budgets, cost allocation across teams, and the optimization strategies that shrink the bill.
Phoenix: Open-Source LLM Observability
Arize's open-source, OTel-native LLM observability stack — traces, eval workflows, and prompt experiments, self-hosted with no data leaving your VPC.
Discussion
Sign in to share your feedback and join the discussion.