Problem Context

Content safety is the runtime layer that prevents your LLM from generating, or being used to generate, harmful content — hate, self-harm, sexual content, violence, illegal acts, jailbreak attempts. It is distinct from policy (your product's rules) and from quality (is the answer good?). Compliance and reputation both live here.

🤔 Sound familiar?
  • Your only safety story is "we put it in the system prompt"
  • Marketing asked about EU AI Act compliance and you don't have a concrete answer
  • You can't produce numbers for "blocked harmful requests this month"

The Standard Harm Taxonomy

Most managed services converge on four categories, each with severity 0–6:

  • Hate & fairness — slurs, dehumanising language, discriminatory generalisations.
  • Sexual — sexual content; strict on minors (CSAM is its own pipeline).
  • Violence — depictions, glorification, planning, weapons.
  • Self-harm — suicide, eating disorders; usually treated with referral text rather than block.

Plus orthogonal categories: jailbreak, protected material (copyrighted text), code vulnerabilities.


flowchart LR
    R[Request] --> CS1[Content Safety — input]
    CS1 -- pass --> LLM
    LLM[LLM] --> CS2[Content Safety — output]
    CS2 -- pass --> U[User]
    CS1 -- block --> N1[Refuse + log]
    CS2 -- block --> N2[Rewrite, redirect, or refuse]

Pick a Managed Classifier

Do not build this yourself. Use one of:

  • Azure AI Content Safety — text + image, 4 harm categories with severity, jailbreak & prompt-shield, protected material.
  • OpenAI Moderation — text, free, lower coverage on injection/jailbreak.
  • AWS Bedrock Guardrails — tight with Bedrock models.
  • Google Cloud Model Armor — Gemini-aligned.
  • Lakera Guard / Protect AI — vendor-neutral, fast.

Severity Thresholds: Action by Category

Don't use a single threshold. Map severity to action per category:

Category          Block ≥    Soft-warn ≥    Notes
hate              4          2              Often false-positive on academic / quoted text
sexual            2          —              Strictest; 6+ → permaban path
violence          4          2              Allow safety/education contexts
self-harm         —          2              Always show helpline; never just block silently
jailbreak         (any flag) —              Block + log + monitor for repeat offenders

Refuse, Rewrite, or Redirect

  • Refuse — the response/request is fundamentally outside policy. Short, neutral, no detail that helps a retry.
  • Rewrite — the underlying intent is legitimate (e.g. medical question, security research). Re-prompt the model with stricter guidance.
  • Redirect — self-harm signals → mental-health resources for the user's region; weapons → terms of service link.

Image and Multimodal

If your app accepts or generates images, run image content safety on both. Azure AI Content Safety and OpenAI's moderation both support images. Watch for: CSAM (mandatory reporting pipelines apply — talk to legal before storing), violent imagery, copyright. Most image generators also have built-in filters; do not rely on them alone.

Compliance Hooks (EU AI Act, US state laws)

  • Audit log every block: timestamp, user (hashed), category, severity, action, model.
  • Retention: typically 12 months, in-region, encrypted at rest.
  • User-facing transparency: a disclosure when a response is filtered ("This response was modified for safety reasons").
  • Right-to-explanation: be able to surface the category and severity to a regulator, if asked.

Failure Modes

  • Over-blocking — academic, legal, medical, security queries flagged as harmful. Tune category-specific thresholds against a real eval set; allow context-aware bypass for verified domains.
  • Under-blocking via paraphrase — "how do I make a [redacted word in a different language]". Use multilingual classifiers.
  • Stateful jailbreaks — slow build-up across turns. Score the whole conversation periodically, not just the last message.
  • Silent failures — the classifier API times out; you fail open. Always have a circuit breaker that fails closed on critical paths.

Production Checklist

  • Managed classifier on both input and output paths.
  • Per-category severity thresholds with per-category actions (refuse/rewrite/redirect).
  • Audit log with retention matching your compliance regime.
  • Conversation-level periodic re-scoring to catch slow jailbreaks.
  • Circuit breaker that fails closed on safety-classifier outage.

Key Takeaways

  • Content safety is a compliance and reputation control — not just a UX nicety.
  • Buy a managed classifier; tune thresholds per category against your own corpus.
  • Audit, redirect, and conversation-aware re-scoring are what differentiate a good safety story from a bad one.