Planning and running an AI red-team engagement

AI red teaming is easy to describe and easy to do badly. NIST’s Generative AI Profile defines it as a structured testing exercise used to probe an AI system to find flaws and vulnerabilities, “often in a controlled environment and in collaboration with system developers.” CISA goes further and says AI red teaming is a foundational part of safety and security evaluation, a subset of AI testing, evaluation, validation and verification (TEVV), which in turn should sit inside established software TEVV. In other words, treat it as a disciplined test with scope, method and a written result, not as a session of clever prompts.

1. Scope and rules of engagement

Write these down before anyone sends a prompt.

• The target: which application, model version, tools, data sources and environments (test or production) are in scope, and which are explicitly out. Where a model is provided by a third party, you can test your application and its configuration. Check the provider’s terms before testing anything of theirs.
• The threat model: whom are you simulating (an anonymous user, a customer, an insider, a poisoned document) and what outcome would count as serious. This comes from the Map function in NIST AI RMF.
• Data handling: use synthetic or planted data (a canary secret, a fake customer record) instead of real personal data. If people outside the team take part as testers, NIST points to human-subjects requirements such as informed consent.
• Stop conditions and contacts: what to do if you reach real data, cause real harm or find something urgent.
• Disclosure: who receives findings, how they are stored, and what may be shared afterwards.

2. Build the right team

NIST notes that the quality of red-team output relates to the background and expertise of the team, and that diverse, interdisciplinary teams can identify flaws in the different contexts where the system will be used. It describes general-public teams, expert teams, combinations of the two (for example experts refining prompts written by the public) and human-plus-AI teams. For a customer-facing assistant, that may mean a security tester plus someone who knows the real customers and the legal context.

3. Design tests from the frameworks

One design consequence comes from variability. CISA notes the concern that AI systems, built with probabilities and often with deliberate variance, may need multiple trials to reveal improper behaviour, while arguing that software in general has always had non-deterministic failures. So record how many attempts you made, and do not conclude that a system is safe from one failed attack.

4. Automated scanners and manual testing

Two open-source tools are worth knowing. garak, from NVIDIA, describes itself as an LLM vulnerability scanner: it checks whether an LLM can be made to fail in ways you do not want, probing for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks and other weaknesses. Its authors compare it to network scanners such as nmap, and it combines static, dynamic and adaptive probes. PyRIT, the Python Risk Identification Tool for generative AI, is an open-source framework, published under the MIT licence in Microsoft’s GitHub organisation, built to help security professionals and engineers proactively identify risks in generative AI systems.

$ python -m pip install -U garak
$ garak --list_probes
$ python3 -m garak --target_type huggingface --target_name gpt2 --spec probes.dan.Dan_11_0

By default garak tries every probe it knows against the chosen model, reports a pass or fail per detector, and by default makes ten generations per prompt, so a result is a failure rate over many samples. It logs each run to a file. That output is a lead, not a verdict: someone still has to read the failing transcripts and judge whether the behaviour matters in your context.

5. Report findings so they can be fixed

A useful finding is reproducible and mapped. For each one, record:

• a title, the affected component and model version, the date and the number of attempts;
• the steps or transcript, with sensitive data redacted, and what the impact would be;
• the mapping: ATLAS technique and tactic IDs with the ATLAS release, the OWASP LLM category, and the NIST AI RMF function it belongs to (usually Measure for the finding and Manage for the fix);
• a recommended mitigation, ideally one that works outside the model, such as permissions, human approval or output validation;
• a retest note: how to confirm the fix.

Also say what you did not test. NIST’s Measure 1.1 asks that risks which will not or cannot be measured be documented, and the Generative AI Profile says red-team results deserve additional analysis before they inform governance decisions and policy updates.

Responsible disclosure

Agree in advance where findings go and when. Findings in your own application go through your internal process, with restricted access because a report is also an attack guide. If you find a weakness in a third party’s model or product, use the channel that party publishes for reporting vulnerabilities rather than posting details publicly, and keep the evidence you collect to the minimum. The next lesson shows what a model developer says it publishes about its own testing.