Anthropic Constitutional AI
Principle-based self-improvement for harmlessness and helpfulness
Verdict
Pioneering approach where the model critiques and revises its own outputs based on a set of principles. Built into Claude models. Influence on the field is massive even if not a standalone product.
Other Guardrails & Safety
- Azure AI Content SafetyProduction
Microsoft's content moderation API for text and images
- Guardrails AIStable
Add input/output validation and safety rails to LLM calls
- Lakera GuardStable
Real-time prompt injection and jailbreak protection API
- Llama Guard 3Experimental
Meta's fine-tuned safety classifier for prompt and response screening
- Microsoft PresidioProduction
Data protection and anonymization for PII in LLM pipelines
- NeMo GuardrailsExperimental
NVIDIA toolkit for programmable guardrails via Colang language