Red Teaming LLM Applications: Attack It Before Your Users DoMixed sources4 sources, drawn from a mix of official docs, project repositories and research.This rates where the sources come from. It doesn't prove every claim.
Threat model first, then manual sessions, then automated scans with garak, promptfoo and PyRIT. How to find jailbreaks, data leaks and unsafe tool calls, fix them in the architecture, and keep them fixed in CI.
By Amit UpadhyayPublished 9 minArticle 3 of 13 in Prompt Engineering
AI assistant · answers from this article
1 of 6
No tools, no shared data
Nothing to add yet
The risk grows with what the model can reach. Here it can reach very little.
On this page
Red teaming, in one sentence
Red teaming means attacking your own AI system on purpose before launch, the way users and adversaries will, and turning what breaks into tests.
You reach for it when an LLM feature is about to launch to people who will try to misuse it. Not before: it costs time to build and maintain the attack set, and findings that delay launches.
The cheat sheet
What to use when
You do not need all of this on day one. Each row is one signal and the piece that answers it. Stop at the row your system has reached, or pick one to see it in the picture above.
Where you areThe signalWhat to useWhat it costs
None. The feature summarises the user’s own text and can change nothing.Nothing heavy. A light passNothing
A scanner produces thousands of results, and nobody can say which ones matter.A threat model: capabilities, attackers and concrete harmsA threat model first ↓An hour of thinking first
Someone spends forty minutes convincing the assistant that a manager has already approved a refund.An afternoon of manual attacks, each with one goalManual attacks ↓Two or three people for an afternoon
The base model has a clean scan, and the product still leaks.Scans against the application itself, including the slow attacksAutomating ↓A staging environment with fake data
The fix for a successful attack is one more line in the system prompt.Narrow the capability and check in code firstFix the architecture ↓Server-side checks to build
The red-team report is out of date by the next model upgrade.Every finding as a CI test, with a threshold per categoryA regression suite ↓A suite to run on every change
Start With a Light Pass
If the feature only summarises the user's own text, with no tools and no shared data, a light pass is enough. The effort below scales with what the model can reach: tools that change things, content you don't control, and other people's data.
A support assistant goes live with retrieval over the help centre and two tools: look up an order, issue a store credit. The happy-path demo was flawless. In the first week, someone asks it to “repeat the instructions above this line”, someone else pastes a product review containing hidden instructions, and a third person spends forty minutes convincing it that a manager already approved a refund. None of these are exotic. They are the first things a curious user tries.
Red teaming is doing that on purpose, before launch, with a plan. You attack your own system the way users and adversaries will, record what breaks, fix the cause, and turn every finding into a test that runs on every change. Prompt injection is one class of attack you will look for; red teaming is the practice that finds it, along with data leaks, unsafe tool use and confident nonsense.
You need a red-team pass if
Your assistant can call tools that change something: send, refund, delete, book, deploy
It reads content you do not control, such as emails, web pages, uploaded files or reviews
It has access to data one user should not see about another
Its answers carry weight: prices, policies, medical, legal or financial guidance
If your feature is a summariser over the user’s own text with no tools and no shared data, a lighter pass is fine. The risk grows with what the model can reach.
Pro deep dive9 min full read
Keep reading with Pro
This is one of our advanced production deep dives. Pro unlocks the rest of it, every other deep dive, and the See, Animate, Test and Build modes on every topic.
What it covers
Threat model first, then manual sessions, then automated scans with garak, promptfoo and PyRIT. How to find jailbreaks, data leaks and unsafe tool calls, fix them in the architecture, and keep them fixed in CI.