Start by Reading the Answers

While a system is a demo and nobody relies on its answers, reading a sample yourself is enough. Measurement starts to matter before a release, when someone will ask how often it states things that aren't true, and you need a number.

A hallucination is an output that is stated as fact and is not supported by anything: not by the documents you supplied, and not by reality. It is the failure users forgive least, because nothing in the answer signals it. The sentence is as fluent as a correct one. Newer models hallucinate less often than older ones, but less often is not a number you can put in a release decision. Testing for it gives you that number.

Signs you are not measuring it yet
  • Your assistant quoted a policy clause that does not exist
  • The answers cite sources, and some citations do not say what the answer claims
  • The model answers questions your documents do not cover instead of saying it does not know
  • Someone asked what your hallucination rate is and you had no figure

Three Different Failures

TypeWhat happensExample
Unfaithful to the sourceThe answer contradicts or goes beyond the supplied contextThe document says 30 days and the answer says 60
FabricationThe model invents a fact, name, number, quotation or referenceA citation to a paper that was never written
False premise acceptedThe question assumes something untrue and the model plays along"How do I enable the export feature?" when no such feature exists

They need different tests. Unfaithfulness is checked against the context. Fabrication is checked against a trusted reference. False-premise acceptance is checked by asking questions that should be refused or corrected.

A Test Set With Questions That Have No Answer

Most evaluation sets contain only questions the system can answer, so they never measure whether it declines properly. A hallucination test set needs the questions that tempt the model to invent something.

  • Answerable: the context contains the answer. This measures faithfulness.
  • Unanswerable: the topic is covered but the specific fact is absent. The correct response is to say so.
  • Out of scope: nothing in the knowledge base is relevant. The correct response is to decline or redirect.
  • False premise: the question assumes something untrue. The correct response is to correct the premise.
  • Conflicting sources: two retrieved passages disagree. The correct response is to report the conflict.

A reasonable starting mix is half answerable and half spread across the other four. Record the expected behaviour for each case, not only an expected answer.

✅

Check it: Count the cases in your test set where the right behaviour is to decline, or to correct the question. If there are none, you are measuring the easy half.

Scoring Faithfulness Claim by Claim

Asking a judge model whether a whole answer is accurate gives unreliable scores, because one unsupported sentence is easy to miss among five correct ones. Split the answer into individual claims and check each against the context.

CLAIMS_PROMPT = """List every factual claim in the answer below, one per line.
Split compound sentences into separate claims. Do not add anything.

Answer:
{answer}"""

VERDICT_PROMPT = """Context:
{context}

Claim: {claim}

Is the claim directly supported by the context?
Reply with one word: SUPPORTED, CONTRADICTED, or NOT_FOUND."""

def faithfulness(answer, context, judge):
    claims = [c for c in judge(CLAIMS_PROMPT.format(answer=answer)).splitlines() if c.strip()]
    if not claims:
        raise ValueError("no claims extracted")  # an evaluator failure, not a score of zero
    verdicts = [judge(VERDICT_PROMPT.format(context=context, claim=c)).strip() for c in claims]
    # Also count any reply outside the three labels, and calibrate against human labels
    supported = sum(v == "SUPPORTED" for v in verdicts)
    return {
        "faithfulness": supported / len(claims),
        "contradicted": [c for c, v in zip(claims, verdicts) if v == "CONTRADICTED"],
        "unsupported": [c for c, v in zip(claims, verdicts) if v == "NOT_FOUND"],
    }

Keep the two failure lists separate. A contradicted claim usually points to a reading error. An unsupported claim usually means the model filled a gap from its own memory, which is the behaviour to discourage in a grounded system.

✅

Check it: Take an answer that you know has one invented detail, and score it both ways: as a whole, and claim by claim. See which one catches it.

The Numbers to Track

MetricMeasured onMeaning
FaithfulnessAnswerable casesShare of claims supported by the context
Correct abstention rateUnanswerable and out-of-scope casesHow often the system declines when it should
False refusal rateAnswerable casesHow often it declines when the answer was available
False premise acceptanceFalse-premise casesHow often it answers without correcting the assumption
Citation accuracyCited answersShare of citations that support the sentence they are attached to

Watch abstention and false refusal together. Instructions that make a system more cautious reduce hallucination and increase unhelpful refusals, and you need both figures to see what a change cost.

✅

Check it: Make the instructions more cautious and run the set again. Faithfulness and false refusals should both be in the report, so you can see what the change cost.

What Reduces the Rate

  • Ground the answer. Supply the relevant source text and instruct the model to answer only from it. Retrieval quality is the largest single factor, since a model cannot stay faithful to context it was not given.
  • Give permission to decline. State what to do when the context does not contain the answer. Without that, answering is the default.
  • Require citations. Asking for a supporting quotation for each claim reduces invention and makes the answer checkable by code.
  • Constrain the output. For extraction, a schema with a nullable field gives the model a valid way to say a value is absent.
  • Verify the claims that matter. Add a second pass that checks the draft against the sources for high-stakes answers. Use it selectively, because it doubles the cost.
⚠️ Common Mistakes

1. Testing only questions that have answers

That measures the easy half. The unanswerable and false-premise cases are where hallucination shows.

2. Assuming a newer model solved it

Rates vary by model, task and prompt, and they do not reach zero. Measure on your own data after every model change.

3. Treating citations as proof

A model can attach a real source to a claim that the source does not make. Check that each cited passage supports its sentence.

4. Measuring only before release

Production questions drift away from the test set. Sample live answers regularly, run the same faithfulness check, and alert when the trend rises.

Testing Agents for Hallucinated Actions

Agents add a new kind of hallucination. A model can invent a tool result, claim to have run a test it never ran, or report a task as finished when it wasn't. These are more dangerous than a wrong fact, since other code may act on them. Compare the final message against the trace: if it says "I've updated the record", is there a successful tool call to back it up? This is a simple automated check, and it catches a surprising amount.

✅

Check it: Take ten finished agent runs and compare each final message with its trace. Every action it claims should have a successful tool call behind it.

Try It on Your Own System This Week

Three checks on a system that answers from documents, about an hour in total:

  1. Ask it five questions your documents don't cover. Count how many it answers anyway.
  2. Ask it how to use a feature that doesn't exist. It should correct the premise.
  3. Open five cited answers and read the cited passages. Each one should support the sentence it is attached to.

Key Takeaways

✅ Key Lessons
  • Hallucination is three failures: unfaithful to the source, fabricated, and false premise accepted. Test each one.
  • Include questions with no answer. Correct abstention is half of what you are measuring.
  • Score faithfulness per claim, and separate contradicted from unsupported.
  • Track abstention with false refusal, so caution is not mistaken for accuracy.
  • Grounding, permission to decline, and citations lower the rate more than any wording trick.