Capstone: run your own red-team pipeline and write the report

Plan, run and report a small red-team engagement end to end on an application you own: scope, a manual round, automation, judge calibration, results, disclosure and a regression suite.

此页面暂时只有英文版。

This capstone ties the course together. You will test a small AI application that you own, with planted canary data, using open-source tools, and finish with a report that another person could act on. It is a plan to follow at your own pace, over a weekend or a few evenings. Do it only on systems you own or have written permission to test.

Step 0: choose and scope the target

Use Microsoft’s open-source Playground Labs, which run locally with Docker Compose against an Azure OpenAI or OpenAI endpoint, or build a minimal chat or summarisation app of your own ^[1].
Plant canaries: a fake secret in the system prompt and a fake record in any retrieval store.
Write a one-page scope: the target, the endpoints, what is out of bounds, the stop conditions, and where findings are stored. Module 13’s engagement lesson has a template.

Warm up: the lab map

The Playground Labs are a good first week before you test your own application. The repository lists twelve labs in four categories at three difficulty levels ^[1]:

The labs grouped by theme. The medium and hard versions add defences that more closely resemble real systems.
Labs	Category	Difficulty	Course lesson
1 and 7: credential exfiltration	Direct prompt injection	Easy, then medium	Lessons 3 and 5
2 and 8: extract a secret from the metaprompt	Metaprompt extraction	Easy, then medium	Lessons 3 and 5
3, 4 and 5: Crescendo	Multi-turn attacks	Easy (safety filters disabled)	Lesson 6
9 and 10: Crescendo with guardrails	Guardrails, multi-turn attacks	Medium, then hard	Lessons 6 and 10
6, 11 and 12: indirect injection	Indirect prompt injection	Easy, medium, hard	Lesson 4

A sensible order is easy before medium, and the lab pairs together: 1, 2 and 6 for the injection basics, then 5 for multi-turn, then 7, 8 and 11 to see the same attacks meet defences, and finally 9, 10 and 12. Labs 3 and 4 use dangerous-substance topics, so skip them or substitute a canary objective of your own, and remember that labs 3 to 5 switch their safety filters off on purpose. Keep a note, for each lab, of which technique family worked and which defence stopped the rest. That note is the start of your coverage matrix.

Step 1: a manual round

Microsoft’s planning guidance recommends starting with open-ended testing, in which testers explore and document any problematic behaviour, then building a list of harms with definitions and examples, and then running guided rounds against that list ^[2]. Record each example with the date, a unique identifier for the input and output pair, the input and a description of the output. Try, by hand, the families from lessons 3 to 6: direct injection, an indirect injection with a harmless marker in a document, a few single-turn framings and a short multi-turn escalation toward a canary.

Step 2: automate

Pick one scanner as your trend line, such as garak or promptfoo, and one scripted tool for multi-turn work, such as PyRIT. Pin their versions.
Select probes by family from the atlas: prompt injection, encoding variants through buffs, system prompt extraction, and the indirect-injection probes where available.
Define an objective for each that your scorer can check: appearance of a canary, a marker in a reply, or a refusal.
Run each configuration several times, because outputs vary, and keep the attempt counts.

Step 3: calibrate the judge

Take a random sample of replies, passes included, label them by hand against your rubric, and compute the judge’s false alarms and misses (lesson 13). If the error is large, fix the rubric or change the judge, and calibrate again. Record the judge’s model and prompt version.

Step 4: read the results

Follow the reading order from lesson 14: denominators, intervals, the failing replies, and a sample of passes. Rank real failures by what they would enable in your design, using the tool and permission inventory from the agents lesson.

Step 5: write the report

Microsoft’s guidance suggests a report that lists the top issues, links to the raw data, previews the plan for the next rounds and acknowledges the testers, and that makes clear red teaming exposes the risk surface and is not systematic measurement, with a content warning where examples are problematic ^[2].

A report skeleton. Keep payloads in restricted storage and describe them in the report.
Section	What to include
Scope and method	Target, versions, dates, tools and their versions, probes, buffs, judge, attempts per probe
Summary	Top findings in plain words, each with an owner and a severity
Findings	For each: the family, the exact conversation or document, what it enabled, how reliably it reproduces, and the suggested fix
Results table	Failures over attempts with intervals per family, and the judge’s measured error rates
Limits	What was not tested, what the judge may have missed, and what the numbers do not show
Next steps	Fixes to retest, new techniques to add, and the schedule

Step 6: disclose responsibly if the weakness is not yours

If you find a flaw in someone else’s software, follow coordinated vulnerability disclosure. The CERT Guide defines it as the process of gathering information from vulnerability finders, coordinating the sharing of that information between relevant stakeholders, and disclosing the existence of vulnerabilities and their mitigations to stakeholders including the public, and it names four roles: finders, reporters, vendors and coordinators ^[3]. Report privately first, and do not publish working payloads.

Step 7: keep it running

Turn each confirmed finding into a regression test that fails if it returns.
Add a small scan stage to your pipeline (lesson 16) and schedule the full suite.
Re-run after every change to the model, the prompt, a guardrail or a tool.
Update the coverage matrix as new techniques are published.

A self-check

Could someone else rerun your tests from the report and get comparable results?
Do the numbers carry intervals and the judge’s error rates?
Did you read both failures and passes?
Would a fix be obvious to the owner from your findings?
Are your payloads kept private?

要点

Scope the engagement, plant canaries and test only systems you own or may test.
Start with manual open-ended testing, then automate with pinned open-source tools and repeat runs.
Calibrate the judge and read results with denominators, intervals and the actual replies.
Report so that others can reproduce and act, keep payloads private, and disclose responsibly.
Turn findings into regression tests and a recurring pipeline stage.

自我检测

1. What should come before automation?
- A manual, open-ended round and a list of harms — 正确。
- Publishing results
- Production deployment
- Deleting logs
Manual exploration finds the risk surface that automation then measures.
2. Why plant canary data?
- It makes failures detectable without real data — 正确。
- It is required by licences
- It speeds scans
- It hides findings
A canary gives a safe, automatic signal of leakage.
3. Who are the four roles named in coordinated vulnerability disclosure?
- Finders, reporters, vendors and coordinators — 正确。
- Users, admins, auditors and judges
- Testers, bosses, clients and press
- Developers, designers, sellers and buyers
The CERT Guide names these four roles.
4. What turns a confirmed finding into lasting protection?
- A regression test in the pipeline — 正确。
- A longer report
- A bigger model
- Silence
A regression test fails when the problem returns.

用 FireAI 动手做

在你自己的 Mac 上练习这节课的内容。

来源

在你的 Mac 上实践

免费试用全部功能 17 天，无需绑定银行卡。

下载 Mac 版文档