AI red-teaming is the step that gets cut first and regretted most. The build finishes, the demo lands well, a launch date appears in a slide, and the only testing anyone has done is a product manager typing friendly questions into a chat box. Then a customer pastes in a document that quietly instructs the assistant to ignore its rules, and a system that passed every functional test starts doing something nobody signed off.
The gap is not competence. It is that ordinary software testing asks whether the thing works, and an AI system needs a second question: what happens when someone actively tries to make it misbehave. That second question is what AI red-teaming answers, and it needs a deliberate slot in the plan alongside your usual IT security work rather than an afternoon before go-live.
This guide sets out what to test before launch, and how to run AI red-teaming that earns its place in a delivery plan. It covers scoping, the five attack classes that matter for business systems, who should run the tests, how to score what comes back, what a real fix looks like, what the exercise costs in days and money, and the evidence you keep afterwards. Every recommendation assumes a normal organisation with a small team and a deadline, not a frontier lab with a dedicated safety department.
The structure below is deliberately ordered the way an engagement runs. Read it once end to end before you scope anything, because the most expensive mistake in AI red-teaming is testing the wrong system boundary very thoroughly.
Table of contents
- What AI red-teaming actually tests, and what it does not
- Scoping AI red-teaming before you write a single test prompt
- Attack class one: prompt injection and instruction hijacking
- Attack class two: data leakage through context, memory and retrieval
- Attack class three: tool abuse and unsafe agent actions
- Attack class four: harmful output, bias and the quality of refusals
- Attack class five: the model supply chain and everything around it
- Who runs AI red-teaming: internal, external or automated
- Scoring AI red-teaming findings and deciding what blocks a launch
- Fixing what AI red-teaming finds
- What AI red-teaming costs and how long it takes
- AI red-teaming evidence, reporting and regulatory mapping
- Mistakes that make AI red-teaming worthless
- Frequently asked questions
- References
What AI red-teaming actually tests, and what it does not
AI red-teaming is structured adversarial testing of an AI system by people trying to make it fail in ways that matter. It is not a benchmark score, not a model card, and not a security scan of the hosting environment. The subject is the whole product — model, prompts, retrieval, tools, guardrails and the humans around it — under pressure from someone who wants a bad outcome.
AI red-teaming, evaluation and penetration testing are three different jobs
Evaluation measures average quality on tasks you expect. Penetration testing attacks the infrastructure the system runs on. AI red-teaming attacks the system’s behaviour through its own legitimate interfaces. All three are worth doing, and none substitutes for the others, because they fail in different directions and produce different evidence.
Why a green evaluation dashboard is not a safe launch
Evaluation suites are built from the cases the team imagined. An adversary’s whole job is to operate outside that set. A system can score ninety-something on accuracy and still hand over another customer’s data on the first deliberate attempt, because accuracy and abuse resistance are unrelated properties measured by unrelated tests. AI red-teaming exists to cover exactly that gap.
The four questions a red team is trying to answer
Can someone make it say something that damages us. Can someone make it reveal something it should not. Can someone make it do something it should not. And can someone make it unavailable or ruinously expensive. Everything in a well-run AI red-teaming exercise ladders back to one of those four.
What sits outside the scope of AI red-teaming
Model weights, provider infrastructure and the training pipeline of a commercial model are not yours to test, and attempting it usually breaches the supplier’s terms. What you test is your configuration, your data boundaries, your tool permissions and your fallback behaviour. That is where nearly every real-world failure in a business deployment actually lives.
| Activity | Question it answers | Who typically runs it | Blocks launch? |
|---|---|---|---|
| Functional QA | Does it do the job on expected input | Product and QA team | Yes |
| Model evaluation | How good is the average answer | Engineering, data science | Usually |
| AI red-teaming | What happens when someone attacks the behaviour | Mixed internal team or specialist | Yes, for critical findings |
| Penetration testing | Can the hosting and network be breached | External security firm | Yes, for critical findings |
| Bias and fairness review | Does it treat groups differently | Risk, legal, subject experts | Depends on use case |
| Live monitoring | Is it still behaving after launch | Operations | No, but must exist |
Scoping AI red-teaming before you write a single test prompt
Scoping is where AI red-teaming engagements are won or lost. A team that skips it spends a fortnight discovering that the chatbot refuses to write malware, which nobody was worried about, while never testing the retrieval layer that quietly indexes the HR folder.
Define the system boundary, not just the model
Draw the actual diagram: interfaces, prompts, retrieval sources, tools and integrations, guardrail services, logging, and the human review steps. Everything inside that boundary is in scope. If your AI system inventory is current, this takes an hour rather than a week, which is one of several reasons the inventory earns its keep.
Write down what harm means for this specific product
Generic harm categories are close to useless in a business context. For an invoice assistant, harm is approving a fraudulent payment. For a triage bot, harm is telling someone their symptoms are fine. Write five to ten concrete harm statements in your own domain language, and let those drive AI red-teaming test design rather than a downloaded taxonomy.
Pick the threat actors you actually face
Most business systems face three: the curious insider poking at limits, the ordinary customer who stumbles into a failure by accident, and the motivated outsider with a financial reason to try. A public-facing consumer product adds a fourth — the person doing it for entertainment, at volume, and posting the results.
Agree the rules of engagement in writing
Which environment gets tested, which accounts may be used, what happens if a tester finds live customer data, who is on call, and what stops the exercise. AI red-teaming against production without written authorisation is the fastest way to turn a security exercise into a personnel matter, so get sign-off from the system owner named in your AI governance framework.
Attack class one: prompt injection and instruction hijacking
Prompt injection is the defining vulnerability class of language-model systems, and no current technique eliminates it. Your goal in AI red-teaming is not to prove it impossible but to find out what an injection can actually reach once it succeeds, and to make that reach small.
Direct injection from the person using the system
The user tells the assistant to disregard its instructions, adopt a different persona, or reveal its configuration. Test polite versions, technical versions, versions framed as debugging, versions split across several turns, and versions in another language. Note which ones work and, more importantly, what they unlock.
Indirect injection from content the system reads
This is the class that catches teams out. Instructions hidden in a document, a web page, a calendar invite, an email footer or a support ticket get treated as input by a system that cannot reliably tell data from commands. If your assistant summarises anything a third party can author, this is your highest-priority test.
The specific AI red-teaming tests to run
Plant benign marker instructions in every content channel the system ingests and see whether they execute. Try white-on-white text, comment fields, metadata, and text inside images if the system reads them. Success is measured by whether the marker changes behaviour, not by whether the model apologises afterwards.
What a pass actually looks like
A pass is not zero injections. A pass is that a successful injection cannot read data the user could not already read, cannot invoke a tool that changes anything, and shows up in logs clearly enough that your AI incident response plan can be triggered by a human within a defined window.
Attack class two: data leakage through context, memory and retrieval
Leakage is the finding that most often turns into a notifiable incident, and it is usually caused by configuration rather than by the model doing anything exotic. It is also the class where AI red-teaming pays for itself fastest, because a single confirmed boundary failure justifies the whole exercise.
System prompt and configuration disclosure
Assume the system prompt will leak eventually and test what that costs you. If it contains an internal URL, a business rule you would not publish, a discount threshold or anything resembling a credential, that is a finding regardless of how hard it was to extract.
Cross-user and cross-tenant boundaries
Run two accounts side by side and try to make one see the other’s material through conversation history, cached retrieval, shared memory or a summarisation feature. In multi-tenant products this is the single highest-severity AI red-teaming test on the list, and it deserves several distinct attempts rather than one.
Retrieval scope and permission inheritance
Retrieval systems commonly index more than the user is entitled to and rely on the model to be discreet, which is not a control. Test whether careful questioning surfaces content from restricted folders, and whether permissions are enforced at query time rather than at indexing time. Pair this with your data protection obligations, because the answer determines your lawful basis.
Memory, logging and persistence
Where does conversation content go, how long does it live, who can read it, and does it reach the model provider. Test whether a user can retrieve their own earlier sensitive input after they believe it was deleted, and whether support staff can read material they should not.
Output leakage into other systems
If the assistant writes to a ticket, a CRM record or an email, the leak path may be downstream rather than on screen. Trace one full journey and check what gets persisted where, because that is the version an auditor will eventually read.
Attack class three: tool abuse and unsafe agent actions
The moment an AI system can act rather than only answer, the severity ceiling rises sharply. This class deserves the largest share of your AI red-teaming time for any agentic deployment, because it is the only class where a single successful attempt can change the world outside the chat window.
Enumerate every tool and its blast radius
List each tool the system can call, what it can change, whether the change is reversible, and what the worst single call could do. Teams are routinely surprised here, because tools accumulate quietly and nobody has read the list end to end since the prototype. That list is the AI red-teaming plan for this class, in priority order.
Test the confused deputy problem directly
The system has permissions the user does not. The attack is to persuade it to use them on the user’s behalf. Try asking indirectly, framing the request as an internal task, and combining it with an injected instruction from content, then check whether authorisation is evaluated per action or assumed from the session.
Chained actions and irreversible steps
Single calls are rarely the problem. Sequences are. Test whether the system will chain a read, a transformation and a write without a human checkpoint, and confirm that anything irreversible — payments, deletions, external messages — sits behind a human-in-the-loop step that a person actually performs.
Cost, rate and availability abuse
An agent that can loop is an agent that can spend. Test whether a crafted request produces runaway tool calls or unbounded token use, and confirm there is a hard ceiling rather than a dashboard someone checks on Mondays. This one is cheap to test and expensive to discover in a monthly invoice.
| Tool capability | Reversible? | Test priority | Required control before launch |
|---|---|---|---|
| Read internal documents | n/a | High | Query-time permission check |
| Write to a record or ticket | Usually | High | Audit trail plus rollback path |
| Send external email or message | No | Critical | Human approval on every send |
| Move money or issue credit | No | Critical | Approval plus value ceiling |
| Delete or overwrite data | No | Critical | Soft delete only, no hard delete |
| Execute code or queries | Varies | Critical | Sandbox with no production credentials |
| Call a paid third-party API | n/a | Medium | Hard spend cap per session |
Attack class four: harmful output, bias and the quality of refusals
This class is where generic AI red-teaming wastes the most time, because the interesting failures are domain-specific and the generic ones are already handled by the provider.
Test the harms in your domain, not the ones in the taxonomy
A commercial provider has already spent enormous effort on universally recognised harmful content. Your marginal value is elsewhere: confident invented product specifications, unauthorised discounts, informal legal or medical advice, promises about delivery dates, and statements a regulator would treat as a financial recommendation.
Over-refusal is a failure too
A system that refuses legitimate requests fails commercially and trains users to route around it. Build a set of benign-but-awkward requests — a customer describing a distressing situation, a clinical term, a security question from your own staff — and measure the false refusal rate as seriously as the harmful output rate.
Bias testing that would survive scrutiny
Hold the substance constant and vary only the attribute: names associated with different backgrounds, gendered pronouns, locations, ages, or non-native phrasing. Run enough repetitions to see a pattern rather than an anecdote, and record the method, because a documented method is what makes an AI red-teaming result defensible to a regulator.
Language, locale and register coverage
Guardrails are usually weakest outside English and outside standard register. If you serve customers in other languages, test in those languages rather than testing translations, and include the informal spellings and abbreviations real customers use.
Grounding failures under pressure
Ask questions the source material does not answer and see whether the system says so. A model that invents an answer under mild pressure will invent one under a customer complaint, which is why this pairs directly with hallucination monitoring once you are live.
Attack class five: the model supply chain and everything around it
The final class covers what changes underneath you and what surrounds the system, and it is the one most often left out of an AI red-teaming scope entirely.
Model and version changes
Your provider will update the model, and behaviour will shift without your code changing. Establish a small regression set of AI red-teaming cases now, so that a version change can be re-tested in an hour instead of debated in a meeting. Version pinning where available is worth the extra cost.
Plugins, connectors and third-party components
Every connector is another party with access to your context. Test what each one receives, whether it can inject content back, and what happens if it returns something malformed or hostile. Your vendor management process should already record who they are, and AI vendor due diligence covers what to ask them.
Output handling in downstream systems
Model output is untrusted input to whatever consumes it. If output reaches a browser, a shell, a spreadsheet formula, a database query or a document renderer, test what happens when the model produces markup, a formula or a command. This is a classic injection problem wearing new clothes, and ordinary application security controls fix it.
Availability and denial of wallet
Test what a burst of expensive requests does to latency, cost and other users. Confirm that a degraded provider produces a sensible fallback rather than a blank screen or a silent failure, and that the fallback path is itself tested rather than theoretical.
Who runs AI red-teaming: internal, external or automated
There is no single right answer, but there is a wrong one, which is letting the team that built the system be the only team that tests it.
The internal team you can assemble today
Three to five people from different functions beats one specialist. Include somebody from support who knows how customers actually phrase things, somebody from the business who knows what a costly mistake looks like, and somebody technical who understands the architecture. Domain knowledge finds more real findings than security credentials do.
When to buy an external AI red-teaming engagement
Buy external when the system is customer-facing at scale, when it takes consequential actions, when a regulator or enterprise buyer will ask for independent testing, or when the internal team has already tested and you need the blind spots found. Ask for the methodology and a sample report before you sign anything.
Automated harnesses and where they genuinely help
Automated AI red-teaming tooling is excellent at breadth, regression and repetition: thousands of variations, run again on every release. It is poor at judgement, novelty and anything requiring business context. Use it to cover the known space cheaply so your humans spend their time on the unknown space.
The hybrid pattern most organisations land on
Automated suites in the release pipeline, a focused internal AI red-teaming exercise before each significant launch, and an external engagement annually or when the system materially changes. That cadence is affordable, defensible, and consistent with how the rest of your change management already works.
| Approach | Typical cost | Best at | Blind spot |
|---|---|---|---|
| Internal cross-functional team | 6-12 person-days | Domain-specific harm, business context | Novel technical attacks |
| External specialist engagement | £8k-£40k | Independence, depth, credible report | Your business nuance, day two onwards |
| Automated harness | Tooling plus 2-4 days setup | Breadth, regression, repeatability | Judgement and genuinely new attacks |
| Bug bounty or customer reports | Variable, ongoing | Long-tail discovery after launch | Nothing before launch |
| Provider safety testing | Included | Universal harmful content | Your data, tools and configuration |
Scoring AI red-teaming findings and deciding what blocks a launch
An exercise that produces a list nobody can prioritise has failed, no matter how clever the attacks were. Agree the scoring scheme before AI red-teaming starts so that nobody negotiates severity after seeing the results.
A severity scale that survives an argument
Score on impact if exploited, how reliably it reproduces, and how much skill or access it takes. Critical means data crossing a customer boundary, an irreversible unauthorised action, or output creating legal exposure. Anything that only works one time in fifty with insider knowledge is not critical, however alarming the transcript reads.
Reproducibility matters more than existence
Run each finding at least ten times and record the hit rate. A one-in-two failure is an operational certainty; a one-in-fifty needs a different response. Recording the rate is also what lets you prove the fix worked, because you can compare like with like after the change.
The launch-blocking bar, written down in advance
A defensible default: no criticals open, no highs without a compensating control, and mediums scheduled with an owner and a date. Write it into the launch checklist before AI red-teaming begins, because a bar agreed under deadline pressure is a bar that moves.
Accepted risks need a name and a date
Some findings will not be fixed before launch, and that is legitimate if it is a decision rather than a drift. Record what was accepted, why, who accepted it, what compensating control exists and when it will be revisited — the same discipline your AI risk assessment already applies elsewhere.
Fixing what AI red-teaming finds
Findings are cheap. Fixes are the product. The mistake to avoid is treating every AI red-teaming finding as a prompt-engineering problem, because the prompt layer is the weakest place to put a control.
Prompt-layer fixes and their honest limits
Adding instructions to the system prompt is fast and sometimes sufficient for tone, scope and refusal behaviour. It is not a security boundary. Anything an attacker can talk the model out of was never a control, so use this layer for quality problems and never as the only mitigation for a critical finding.
Architecture-layer fixes, which are the real ones
Reduce what the system can reach. Enforce permissions at query time. Remove a tool. Split a broad tool into narrow ones. Put a deterministic validator between the model and any action. Separate untrusted content from instructions structurally. These survive model changes, prompt drift and adversarial creativity, which is exactly why they cost more to build.
Policy and human-layer fixes
Some risks are best handled by requiring approval, restricting who has access, adding a disclosure to users, or narrowing the launch audience. These are legitimate controls when documented in your acceptable use policy and actually enforced rather than merely written.
Re-test rather than assume
Every fix gets re-tested against the original AI red-teaming finding, using the original method, at the original repetition count. A surprising share of fixes reduce reproducibility without removing the failure, and the only way to see that is to measure it the same way twice.
What AI red-teaming costs and how long it takes
Budgets fail when nobody separates the external invoice from the internal effort, and the internal effort is usually the larger number.
Effort by system complexity
A read-only assistant over public content is a two-to-three day exercise. A retrieval assistant over internal documents is roughly a week. An agent with write access to business systems is two to three weeks and should be repeated. Complexity is driven by tools and data reach, not by model size.
The internal cost people forget
Add engineering time to build a safe test environment, seed representative data, support the testers, triage findings and implement fixes. On a first exercise, fix effort commonly exceeds test effort. Budgeting for AI red-teaming but not for remediation is the most common way this work stalls at ninety percent.
External engagement pricing in practice
Independent engagements for a defined business system generally run from the low thousands for a focused review to the mid tens of thousands for a deep agentic assessment. Price tracks scope and depth, so a cheap quote usually means a narrow scope rather than a bargain — ask what is excluded.
AI red-teaming cadence after launch
Re-test on material change, on model version change, on new tool or data source, after any incident, and at least annually regardless. Between those points, the automated regression set carries the load, which is the cheapest part of the whole programme once it exists.
AI red-teaming evidence, reporting and regulatory mapping
An exercise nobody can evidence is an exercise you will repeat under worse conditions, usually while a customer waits for an answer.
What the report has to contain
Scope and system boundary, threat model, method, every finding with severity and reproduction steps, the fix or accepted-risk decision for each, re-test results, and residual risk with an owner. Two pages of summary for the board, the detail behind it, and no screenshots of live customer data.
Mapping to the frameworks you already answer to
AI red-teaming maps to the EU AI Act’s obligations on accuracy, robustness and cybersecurity for high-risk systems, to the NIST AI Risk Management Framework’s Measure function, and to the operational controls in ISO/IEC 42001. If you are working through an EU AI Act compliance checklist or ISO 42001 certification, this report is a primary piece of evidence.
What enterprise buyers ask to see
Increasingly, security questionnaires ask directly whether AI features have been adversarially tested, by whom, when, and what was found. A one-page AI red-teaming summary you can share under NDA removes a procurement blocker that otherwise costs weeks, which is a commercial return on the work rather than a compliance cost.
Retention and re-test triggers
Keep AI red-teaming reports for the life of the system plus your standard retention period, and record the triggers that force a re-test. Storing them alongside the rest of your IT governance evidence means the next questionnaire is a retrieval task rather than a project.
Mistakes that make AI red-teaming worthless
Every failed AI red-teaming exercise fails in one of a small number of recognisable ways, and all of them are cheap to avoid once named.
Testing the demo instead of the product
The demo has curated data, generous permissions and no real integrations. AI red-teaming findings from it do not transfer. Test the build that will launch, against representative data, with production-equivalent permissions, or accept that you have tested nothing that matters.
Letting the builders mark their own homework
The people who wrote the prompts cannot unsee them. They test what they defended and skip what never occurred to them. Independence in AI red-teaming does not require an external firm, but it does require someone who did not build the thing.
Stopping at the first refusal
A model declining once is not a control working. Testers who accept the first no produce reassuring reports and no safety. The finding lives two or three turns later, in a rephrasing, in another language, or in an instruction delivered through content instead of chat.
Treating AI red-teaming as a one-off gate
The model changes, the data changes, the tools change, and attackers improve. A single exercise before launch with nothing afterwards produces a certificate that decays quietly, which is why cadence belongs in the AI agent operating model rather than in a project plan that ends.
Findings without owners
A report that lists twenty issues and assigns none of them changes nothing. Every AI red-teaming finding needs a named owner, a date and a decision, tracked where your team already tracks work, not in an attachment nobody opens twice.
Confusing tooling with a programme
Buying an automated harness and running it monthly is useful and is not an AI red-teaming programme. Tools cover the space you already anticipated. Judgement covers the rest, and only people supply judgement — which is the whole argument for a deliberate AI strategy rather than a shopping list.
Frequently asked questions
How long before launch should AI red-teaming start?
Early enough that findings can change the architecture, which usually means when the system works end to end but before the launch date is public. Testing a week before go-live guarantees that anything structural gets accepted as a risk rather than fixed.
Do we need external testers for a first launch?
Not always. For an internal, read-only assistant, a competent internal AI red-teaming exercise is proportionate. For customer-facing systems, anything taking consequential actions, or anything a regulated buyer will scrutinise, independent testing is worth the cost and is increasingly expected.
Can prompt injection be fully prevented?
No. Treat it as a permanent property of systems that mix instructions and untrusted content. The AI red-teaming goal is containment: assume injection succeeds and ensure it cannot reach data or actions that matter. Any vendor claiming complete prevention is describing marketing, not architecture.
How do we test without exposing real customer data?
Use a representative synthetic dataset with the same structure, sensitivity labels and edge cases as production. Where real data is unavoidable, restrict it to a named test set, log all access, and agree in the rules of engagement what testers do if they encounter something they should not see.
What is the minimum viable AI red-teaming for a small team?
Two days, three people, five written harm statements, and a fixed checklist: direct injection, indirect injection, cross-user leakage, tool misuse, and one grounding test. Record everything. That minimum finds more than most organisations expect and gives you a baseline to improve against.
How does AI red-teaming relate to monitoring after launch?
They are the same programme at different points in time. AI red-teaming tells you what can go wrong; production monitoring tells you whether it is going wrong now. Findings should become monitored signals, and monitoring alerts should become new test cases.
Who should own AI red-teaming internally?
Whoever owns the system, supported by security and the business. Ownership sitting with a security team alone tends to produce technically valid findings with no commercial context, which get deprioritised. Ownership sitting only with the product team tends to produce optimism.
References
NIST AI Risk Management Framework
NIST AI 100-2 Adversarial Machine Learning Taxonomy and Terminology
OWASP Top 10 for Large Language Model Applications
NCSC Guidelines for Secure AI System Development
NCSC Principles for the Security of Machine Learning
ICO Guidance on Artificial Intelligence and Data Protection
EU AI Act Article 15 Accuracy, Robustness and Cybersecurity
European Commission Regulatory Framework for Artificial Intelligence
Artificial Intelligence Playbook for the UK Government
BSI ISO/IEC 42001 Artificial Intelligence Management Systems
More AI coverage: explore Progressive Robot's AI Models, Tools & Releases hub — hands-on reviews, setup guides and benchmarks in one place.
