Validating LLM Outputs

Problem Context

Output validation is the last line of defence before the LLM's response reaches a user, a downstream API, or a database. Where input validation protects the model, output validation protects everything downstream of the model. Skipping it means hallucinations become user-facing facts, malformed JSON crashes the next service in the chain, and policy violations become support tickets.

Five Things to Check Before the Response Leaves Your System

• Schema — does the JSON / structured output match the contract?
• Faithfulness — is the answer grounded in the retrieved context (for RAG)?
• Policy / toxicity — does it violate content rules?
• PII leak — did the model echo personal data into the response?
• Action safety — for tool calls, are arguments sane (no DROP TABLE, no email to *)?

flowchart LR
    LLM[LLM response] --> SC[Schema check<br/>Zod/Pydantic]
    SC -- repair --> LLM
    SC --> F[Faithfulness check<br/>vs context]
    F --> P[Policy / toxicity]
    P --> PI[PII scrub]
    PI --> A[Action safety<br/>tool args]
    A -- pass --> U[User / downstream]
    F -- fail --> R[Rewrite or refuse]
    P -- fail --> R
    A -- fail --> R
      

Schema: Use Native, Fall Back to Repair

• Native structured output first — OpenAI response_format: json_schema, Anthropic tool-use, Gemini JSON mode. Failure rate < 1% on modern models.
• Validate-and-repair as a backstop — parse with Zod/Pydantic, on failure send the validation error back to the model withmaxRetries: 2. Cap retries; an infinite loop here is a real outage.
• Reject after retries — return a typed error to the caller, log the offending response for the eval set.

const Schema = z.object({
  intent: z.enum(['cancel', 'refund', 'other']),
  reason: z.string().max(280),
  confidence: z.number().min(0).max(1),
});

const { object, finishReason } = await generateObject({
  model: openai('gpt-4o'),
  schema: Schema,
  prompt,
  maxRetries: 2,
});
if (finishReason !== 'stop') throw new SchemaRepairFailed();

Faithfulness: The RAG-Specific Check

For RAG, the highest-value output check is faithfulness. Run a small LLM-as-judge call (or use a dedicated NLI model) that takes the response and the retrieved context and returns a 1–5 score plus the list of claims not supported by the context. Unsupported claims at score ≤ 2: rewrite with a stricter prompt or refuse and ask a clarifying question.

Policy / Toxicity

Use a managed content-safety classifier (Azure AI Content Safety, OpenAI Moderation, Perspective API). These catch the obvious harm categories cheaply. For domain-specific policy (no investment advice, no medical diagnoses, no competitor mentions), pair a regex/keyword check with an LLM-as-judge.

PII in the Response

Models will occasionally regurgitate PII from training data or from retrieved context. Run the same Presidio/Comprehend pass you use for input validation, but with a tuned policy:

• Block credit cards, SSNs, API keys — these should never appear in a response.
• Allow names/emails if they came from the request (echoing the user's own data is fine); redact if they didn't.

Tool-Call Argument Safety

Tool calls are the most dangerous output — they touch the real world. Validate every argument before executing:

• SQL tools: reject statements with DROP/TRUNCATE/ALTER; allow-list table names.
• Email tools: deny * or unbounded recipient lists; require an explicit user-id allow-list.
• HTTP tools: domain allow-list, no localhost / metadata endpoint (169.254.169.254), no SSRF.
• Shell / code-exec tools: sandbox (Docker, gVisor, microVM); time + memory caps.

Streaming Outputs Are Harder

You cannot fully validate a streamed response until it ends. Pragmatic patterns:

• Stream to the client but also buffer; run policy checks on the buffer; if a violation is detected, send a typed correction event and stop the stream.
• For schema, consider non-streaming for structured outputs — the UX cost is small relative to a malformed-JSON outage.
• For tool calls, never execute until the call is fully received and validated.

Failure Modes

• Repair loops without a cap — the model gets stuck producing the same invalid output; cap at 2.
• Validation only on the happy path — error responses and refusals also need to match the contract.
• Faithfulness check using the same model — biased toward agreement. Use a different family or a smaller dedicated model.
• Silent rewrites — user thinks they got an answer; you swapped it. Always log + return a flag.

Production Checklist

• Native structured output where supported; Zod/Pydantic validate-and-repair as fallback with a 2-retry cap.
• Faithfulness check for every RAG response; rewrite or refuse on low scores.
• Managed content-safety classifier on every response.
• Tool-call argument validation with explicit allow-lists and sandboxing.
• Streamed responses go through a buffered policy check before they reach the user.

Key Takeaways

• Output validation protects everything downstream of the model — schema, content, action, and reputation.
• Schema wins go to native structured output; faithfulness wins go to a separate judge model; action safety wins go to allow-lists and sandboxes.
• Streaming is the trickiest surface — buffer + validate + intercept.