Hands-on: comparing garak, Augustus and PyRIT on one target

The previous three lessons used each tool alone. This one asks a harder question: when you add a defence, does a scanner notice? We measured it on 12 October 2026, on one Mac, against a local model, and the whole experiment is published so you can repeat it. The full results page is here and every script is on a source page ^[1].

The design

• One target: a document summariser on a local model, served over HTTP. Everything a scanner sends is treated as the document to summarise, which is the indirect-injection setting from lesson 4.
• Five defence arms, changing only the application: no defence; an instruction not to obey the document; delimiters around the document; data marking with a fixed marker; and data marking with a random marker per request (lesson 10) ^[2].
• A ground truth we wrote ourselves: 16 short documents, each carrying six harmless planted instructions (plain, authority, postscript, a fake system tag, translated and base64) that ask the assistant to add a made-up code. A reply containing the code is a success for the attack. That makes 96 attacks per arm, scored by a deterministic check, plus 16 clean documents to confirm the summaries stay useful.
• Three scanners with their own attacks and detectors: garak 0.17.0 and Augustus (both with their promptinject and latentinjection families, lessons 13 and 14), and PyRIT 1.1.0 sending our own payloads through stacked converters with a substring scorer (lesson 15) ^{[3] [4] [5]}.

The results

We turn each cell into a rate with a Wilson interval and call a drop “clear” only when its interval does not overlap the no-defence interval, as lesson 18 describes ^[6].

What the comparison shows

1. The effective defence shows up. Random data marking is a clear drop in the ground truth, in garak and in Augustus. PyRIT’s interval overlaps the baseline at its sample size.
2. Instruction-only did nothing measurable anywhere: a rule in the prompt did not stop this small model from obeying text inside the document.
3. Delimiters are where the tools disagree. The ground truth saw no effect, Augustus saw a clear drop, and garak and PyRIT saw changes that overlap the baseline. A likely reason is that the scanners’ built-in attacks are blunt “ignore the above and print this” requests, while our payloads also use authority, a fake system tag, a postscript and another language. We did not test that explanation, so treat it as a hypothesis.
4. The numbers differ in kind. garak ran 160 attempts per arm, PyRIT 128 and Augustus 58, so Augustus has the widest intervals. Per attempt the time was about 0.8 s for Augustus, 1.8 s for garak and 1.9 s for PyRIT here.

Lessons for your own use

• A scanner’s verdict depends on its attack set. One suite is never the whole truth, so use more than one tool and keep a ground truth you wrote yourself.
• Compare arms by intervals, not by raw rates, and check how many attempts each cell has.
• Read setup effort as part of the cost. In our run, four Augustus probes needed a configured judge model and were skipped; garak’s latent-injection probes ignored our prompt cap; PyRIT needed us to write the script and supply prompts and a scorer.
• Pick one tool as your trend line and keep it fixed (lesson 21), and use the others to find gaps.

Reproduce it

Copy the scripts from the source page into a folder (it also holds the README), and follow it. It needs Ollama with a small model, a Python environment for garak, a separate one for PyRIT, and the Augustus binary built with Go. On our Mac the ground truth took about a quarter of an hour, garak about 24 minutes, PyRIT about 20 and Augustus about 4.

$ ollama pull gemma3:4b-it-q4_K_M
$ python3 bench/target.py &                      # the target, on port 8800
$ python3 bench/make_corpus.py && python3 bench/run_canary.py
$ python3 tools/run_all.py                       # three tools x five arms
$ python3 bench/make_report.py                   # builds results/report.html

Limits

• One small local model, one machine, one day, deterministic decoding. The results describe this target, not the tools in general.
• Single-turn attacks only. Multi-turn techniques such as Crescendo are out of scope.
• The scanners’ payloads and detectors differ from each other and from our ground truth, so absolute rates are not comparable between tools.
• PyRIT is a framework: its figures describe the prompts and scorer we supplied.

Try it

1. Run the scripts once as they are and confirm your numbers are close to the table.
2. Swap the model for a larger one and see whether instruction-only starts to help.
3. Add a fifth arm of your own, such as an input filter, and read the result with intervals.