LLM Evaluation Beyond Vibes

"Looks Good to Me" Is Not a Test

Here is how most prompt changes ship. Someone edits the prompt, runs it on the five tickets they keep in a scratch file, reads the outputs, decides they are better, and merges. Two weeks later a support lead mentions that agents are rewriting most of the drafts, and nobody can say which change caused it.

This is not carelessness. LLM output rarely has one right answer, so ordinary unit tests do not fit, and reading outputs feels like the only option. It is not the only option. A fixed set of real cases, scored by code where possible and by a calibrated model where not, turns "I think it is better" into a number you can put in a pull request. The rest of this page builds that, from choosing the cases to running it in CI.

What You Are Actually Measuring

Good LLM evaluation operates at three levels: offline evaluation (before deployment), online metrics (during production), and human feedback loops (continuous calibration). Each level catches different problems.

      flowchart TD
          A["Define Quality\nDimensions"] --> B["Build Test Suite\n(golden dataset)"]
          B --> C["Automated Evaluation\n(metrics + LLM-as-judge)"]
          C --> D["Human Evaluation\n(calibration)"]
          D --> E["CI Integration\n(regression gate)"]
          E --> F["Production Monitoring\n(online metrics)"]
          F -->|"Feedback loop"| B
      
          style A fill:#4f46e5,color:#fff,stroke:#4338ca
          style C fill:#059669,color:#fff,stroke:#047857
          style E fill:#7c3aed,color:#fff,stroke:#6d28d9
          style F fill:#dc2626,color:#fff,stroke:#b91c1c
      

Quality Dimensions

Before measuring anything, define what you're measuring. Common dimensions for engineering tasks:

• Correctness: Is the output factually and technically accurate?
• Relevance: Does the output address what was asked?
• Completeness: Does it cover all required aspects?
• Format compliance: Does it follow the specified structure?
• Conciseness: Is it appropriately brief without losing substance?
• Safety: Does it avoid harmful, biased, or inappropriate content?

LLM-as-Judge

Use a more capable model (e.g., GPT-5.6 Sol) to evaluate outputs from your production model. Provide rubrics, reference answers, and scoring criteria. It's not perfect — judges have biases too — but it scales far better than human review and correlates well with human judgment on structured rubrics.

Building the Evaluation, Step by Step

Step 1: Build a Golden Test Suite

public class EvalTestCase
      {
          public required string Id { get; init; }
          public required string Category { get; init; }
          public required string Input { get; init; }
          public string? ReferenceOutput { get; init; }
          public required Dictionary<string, double> MinScores { get; init; }
      }
      
      // Example test cases
      var testSuite = new List<EvalTestCase>
      {
          new()
          {
              Id = "review-null-check",
              Category = "code-review",
              Input = "Review: if (user != null && user.Name != null) { ... }",
              ReferenceOutput = "Use pattern matching: `if (user is { Name: not null })`...",
              MinScores = new()
              {
                  ["correctness"] = 0.8,
                  ["actionability"] = 0.7,
                  ["conciseness"] = 0.6
              }
          }
      };
      

Step 2: Automated Scoring with LLM-as-Judge

public class LlmJudge
      {
          private readonly ChatClient _judgeClient; // GPT-5.6 Sol
      
          public async Task<Dictionary<string, double>> ScoreAsync(
              string input, string output, string? reference)
          {
              var prompt = $"""
                  Score the following AI output on these dimensions (0.0-1.0):
                  - correctness: Technical accuracy of the response
                  - relevance: How well it addresses the input
                  - completeness: Coverage of important aspects
                  - conciseness: Brevity without losing substance
      
                  INPUT: {input}
                  AI OUTPUT: {output}
                  {(reference != null ? $"REFERENCE: {reference}" : "")}
      
                  Return JSON: {{"correctness": 0.0, "relevance": 0.0,
                  "completeness": 0.0, "conciseness": 0.0}}
                  """;
      
              var response = await _judgeClient.CompleteChatAsync(
                  [new UserChatMessage(prompt)],
                  new ChatCompletionOptions
                  {
                      ResponseFormat = ChatResponseFormat.CreateJsonObjectFormat(),
                      Temperature = 0.1f
                  });
      
              return JsonSerializer.Deserialize<Dictionary<string, double>>(
                  response.Value.Content[0].Text)!;
          }
      }
      

Step 3: Regression Testing Pipeline

public class EvalPipeline
      {
          public async Task<EvalReport> RunAsync(
              IChatClient modelUnderTest,
              List<EvalTestCase> testSuite)
          {
              var results = new List<EvalResult>();
      
              foreach (var testCase in testSuite)
              {
                  var output = await modelUnderTest.CompleteAsync(testCase.Input);
                  var scores = await _judge.ScoreAsync(
                      testCase.Input, output, testCase.ReferenceOutput);
      
                  var passed = testCase.MinScores.All(
                      req => scores.GetValueOrDefault(req.Key, 0) >= req.Value);
      
                  results.Add(new EvalResult
                  {
                      TestCaseId = testCase.Id,
                      Scores = scores,
                      Passed = passed,
                      Output = output
                  });
              }
      
              return new EvalReport
              {
                  TotalTests = results.Count,
                  Passed = results.Count(r => r.Passed),
                  Failed = results.Count(r => !r.Passed),
                  AverageScores = AggregateScores(results),
                  FailedTests = results.Where(r => !r.Passed).ToList()
              };
          }
      }
      

Step 4: Production Quality Monitoring

public class ProductionMonitor
      {
          public async Task RecordInteraction(
              string input, string output, string modelId)
          {
              // Sample 5% of interactions for async evaluation
              if (Random.Shared.NextDouble() > 0.05) return;
      
              var scores = await _judge.ScoreAsync(input, output, reference: null);
      
              _metrics.RecordGauge("llm.quality.correctness", scores["correctness"],
                  new("model", modelId));
              _metrics.RecordGauge("llm.quality.relevance", scores["relevance"],
                  new("model", modelId));
      
              // Alert on quality drops
              if (scores.Values.Average() < 0.6)
                  _alerts.Trigger("llm-quality-below-threshold", new { modelId, scores });
          }
      }
      

How Evaluations Mislead You

Where I Would Start