What AI red teaming is, and what it is not

This course follows the arc of Microsoft’s public video series AI Red Teaming 101, presented by Dr Amanda Minnich, Gary Lopez and Nina Chikanov of Microsoft’s AI red team and its offensive research colleagues, and then goes further: the science behind the techniques, the open-source tools that automate them, and how to read and trust the results. The series is described by its presenters as drawn from their introductory training, taught at Black Hat and other events, and the labs behind it have been released as open source. Everything here is for authorised testing of systems you own or are permitted to test.

A different kind of red team

In traditional security, red teaming describes a double-blind exercise: the defenders do not know the testers are there, the testers go slowly and quietly, and they emulate a capable adversary such as a state-backed group with mature tools. In the series’ account, AI red teaming usually differs on three points. It tends to be single-blind, because the product team or model owner knows the testing is happening. It emulates both adversarial and benign users, since an ordinary person can trigger a harmful output by accident. And its tools and methods change quickly, so the work feels less like a fixed playbook and more like applied research.

Microsoft’s own planning guidance makes the same point about breadth. With language models, “both benign and adversarial usage can produce potentially harmful outputs”, which is why that guidance recommends recruiting testers with benign as well as adversarial mindsets ^[2].

Three built-in risks

The first episode groups the risks of generative models into three families, and the rest of the course keeps coming back to them.

• Fabrications: the model states something false with confidence, such as an invented citation.
• Alignment gaps: the model has learned behaviour that differs from what its builders intended.
• Prompt injection: the model cannot reliably tell instructions from data, because it receives everything as a single stream with no boundaries.

The presenters stress testing safety and security together, because they overlap: a prompt-injection flaw (security) is often used to produce harmful content (safety), and the reverse.

Who does this work

The series states that Microsoft’s AI red team has existed since 2018 and tests Microsoft’s production AI systems, including Copilots, models and autonomous agents, and that it evaluates OpenAI models before release, often as the last checkpoint before a system ships. Its members come from offensive security, adversarial machine learning, responsible AI, abuse prevention, privacy and other fields, because no single discipline covers all the risks. ^[1]

Eight lessons from testing 100 products

In January 2025 the team published Lessons From Red Teaming 100 Generative AI Products, which presents eight lessons together with case studies ^[3]. In short: understand what the system can do and where it is used; breaking AI systems does not require computing gradients; red teaming is not the same as running standard safety test suites; automation helps cover more of the risk landscape; human judgment stays essential; harms to responsible AI are widespread but hard to measure; language models amplify existing security risks and add new ones; and securing AI systems is never finished.

Two of these shape the structure of this course. Because automation extends coverage, a large part of it is about open-source tools and pipelines. Because human expertise remains essential, the tools are treated as helpers, with the reading of results left to people.

What AI red teaming is not

• Not measurement. Microsoft’s guidance says red teaming is “not a replacement for systematic measurement”, and warns that readers should not take specific examples as a metric for how common a harm is ^[2].
• Not a certificate of safety. Finding nothing shows only that your tests found nothing.
• Not permission to test anything. Work inside an agreed scope, on systems you own or are authorised to test, with planted canary data and not real personal data.