Hallucination Testing: Measuring What the Model Makes Up

Fluent, Confident, and Wrong

A hallucination is an output that is stated as fact and is not supported by anything: not by the documents you supplied, and not by reality. It is the failure users forgive least, because nothing in the answer signals it. The sentence is as fluent as a correct one. Newer models hallucinate less often than older ones, but less often is not a number you can put in a release decision. Testing for it gives you that number.

Three Different Failures

They need different tests. Unfaithfulness is checked against the context. Fabrication is checked against a trusted reference. False-premise acceptance is checked by asking questions that should be refused or corrected.

A Test Set With Questions That Have No Answer

Most evaluation sets contain only questions the system can answer, so they never measure whether it declines properly. A hallucination test set needs the questions that tempt the model to invent something.

• Answerable: the context contains the answer. This measures faithfulness.
• Unanswerable: the topic is covered but the specific fact is absent. The correct response is to say so.
• Out of scope: nothing in the knowledge base is relevant. The correct response is to decline or redirect.
• False premise: the question assumes something untrue. The correct response is to correct the premise.
• Conflicting sources: two retrieved passages disagree. The correct response is to report the conflict.

A reasonable starting mix is half answerable and half spread across the other four. Record the expected behaviour for each case, not only an expected answer.

Scoring Faithfulness Claim by Claim

Asking a judge model whether a whole answer is accurate gives unreliable scores, because one unsupported sentence is easy to miss among five correct ones. Split the answer into individual claims and check each against the context.

CLAIMS_PROMPT = """List every factual claim in the answer below, one per line.
Split compound sentences into separate claims. Do not add anything.

Answer:
{answer}"""

VERDICT_PROMPT = """Context:
{context}

Claim: {claim}

Is the claim directly supported by the context?
Reply with one word: SUPPORTED, CONTRADICTED, or NOT_FOUND."""

def faithfulness(answer, context, judge):
    claims = [c for c in judge(CLAIMS_PROMPT.format(answer=answer)).splitlines() if c.strip()]
    verdicts = [judge(VERDICT_PROMPT.format(context=context, claim=c)).strip() for c in claims]
    supported = sum(v == "SUPPORTED" for v in verdicts)
    return {
        "faithfulness": supported / max(len(claims), 1),
        "contradicted": [c for c, v in zip(claims, verdicts) if v == "CONTRADICTED"],
        "unsupported": [c for c, v in zip(claims, verdicts) if v == "NOT_FOUND"],
    }

Keep the two failure lists separate. A contradicted claim usually points to a reading error. An unsupported claim usually means the model filled a gap from its own memory, which is the behaviour to discourage in a grounded system.

The Numbers to Track

Watch abstention and false refusal together. Instructions that make a system more cautious reduce hallucination and increase unhelpful refusals, and you need both figures to see what a change cost.

What Reduces the Rate

• Ground the answer. Supply the relevant source text and instruct the model to answer only from it. Retrieval quality is the largest single factor, since a model cannot stay faithful to context it was not given.
• Give permission to decline. State what to do when the context does not contain the answer. Without that, answering is the default.
• Require citations. Asking for a supporting quotation for each claim reduces invention and makes the answer checkable by code.
• Constrain the output. For extraction, a schema with a nullable field gives the model a valid way to say a value is absent.
• Verify the claims that matter. Add a second pass that checks the draft against the sources for high-stakes answers. Use it selectively, because it doubles the cost.

Key Takeaways