Capstone: run your own red-team pipeline and write the report

This capstone ties the course together. You will test a small AI application that you own, with planted canary data, using open-source tools, and finish with a report that another person could act on. It is a plan to follow at your own pace, over a weekend or a few evenings. Do it only on systems you own or have written permission to test.

Step 0: choose and scope the target

• Use Microsoft’s open-source Playground Labs, which run locally with Docker Compose against an Azure OpenAI or OpenAI endpoint, or build a minimal chat or summarisation app of your own ^[1].
• Plant canaries: a fake secret in the system prompt and a fake record in any retrieval store.
• Write a one-page scope: the target, the endpoints, what is out of bounds, the stop conditions, and where findings are stored. Module 13’s engagement lesson has a template.

Warm up: the lab map

The Playground Labs are a good first week before you test your own application. The repository lists twelve labs in four categories at three difficulty levels ^[1]:

A sensible order is easy before medium, and the lab pairs together: 1, 2 and 6 for the injection basics, then 5 for multi-turn, then 7, 8 and 11 to see the same attacks meet defences, and finally 9, 10 and 12. Labs 3 and 4 use dangerous-substance topics, so skip them or substitute a canary objective of your own, and remember that labs 3 to 5 switch their safety filters off on purpose. Keep a note, for each lab, of which technique family worked and which defence stopped the rest. That note is the start of your coverage matrix.

Step 1: a manual round

Microsoft’s planning guidance recommends starting with open-ended testing, in which testers explore and document any problematic behaviour, then building a list of harms with definitions and examples, and then running guided rounds against that list ^[2]. Record each example with the date, a unique identifier for the input and output pair, the input and a description of the output. Try, by hand, the families from lessons 3 to 6: direct injection, an indirect injection with a harmless marker in a document, a few single-turn framings and a short multi-turn escalation toward a canary.

Step 2: automate

1. Pick one scanner as your trend line, such as garak or promptfoo, and one scripted tool for multi-turn work, such as PyRIT. Pin their versions.
2. Select probes by family from the atlas: prompt injection, encoding variants through buffs, system prompt extraction, and the indirect-injection probes where available.
3. Define an objective for each that your scorer can check: appearance of a canary, a marker in a reply, or a refusal.
4. Run each configuration several times, because outputs vary, and keep the attempt counts.

Step 3: calibrate the judge

Take a random sample of replies, passes included, label them by hand against your rubric, and compute the judge’s false alarms and misses (lesson 13). If the error is large, fix the rubric or change the judge, and calibrate again. Record the judge’s model and prompt version.

Step 4: read the results

Follow the reading order from lesson 14: denominators, intervals, the failing replies, and a sample of passes. Rank real failures by what they would enable in your design, using the tool and permission inventory from the agents lesson.

Step 5: write the report

Microsoft’s guidance suggests a report that lists the top issues, links to the raw data, previews the plan for the next rounds and acknowledges the testers, and that makes clear red teaming exposes the risk surface and is not systematic measurement, with a content warning where examples are problematic ^[2].

Step 6: disclose responsibly if the weakness is not yours

If you find a flaw in someone else’s software, follow coordinated vulnerability disclosure. The CERT Guide defines it as the process of gathering information from vulnerability finders, coordinating the sharing of that information between relevant stakeholders, and disclosing the existence of vulnerabilities and their mitigations to stakeholders including the public, and it names four roles: finders, reporters, vendors and coordinators ^[3]. Report privately first, and do not publish working payloads.

Step 7: keep it running

• Turn each confirmed finding into a regression test that fails if it returns.
• Add a small scan stage to your pipeline (lesson 16) and schedule the full suite.
• Re-run after every change to the model, the prompt, a guardrail or a tool.
• Update the coverage matrix as new techniques are published.

A self-check

• Could someone else rerun your tests from the report and get comparable results?
• Do the numbers carry intervals and the judge’s error rates?
• Did you read both failures and passes?
• Would a fix be obvious to the owner from your findings?
• Are your payloads kept private?