Hands-on: scan a REST API with Augustus

Augustus describes itself as “a Go-based LLM vulnerability scanner for security professionals” ^[1]. It is built around the same ideas as garak (probes, detectors, generators and buffs, lesson 11), so what you learned there transfers. This lesson follows a real run on 12 October 2026 against a small summarising assistant.

Install it

$ GOBIN="$PWD/bin" go install github.com/praetorian-inc/augustus/cmd/augustus@latest
$ ./bin/augustus list | head -20

On our machine the build fetched a newer Go toolchain by itself. augustus list prints what the build contains: at install time, 243 probes, 132 detectors, 48 generators, 3 harnesses and 35 buffs. The numbers change with each release, so list them yourself.

Configure the REST generator

Its README documents the keys for custom endpoints: uri, method, headers, a request template (req_template for raw text, or req_template_json_object for JSON where $INPUT stands for the prompt), and response_json with response_json_field to say where the reply is ^[1]. It also lists a proxy key for traffic interception tools.

$ ./bin/augustus scan rest.Rest \
    --probe promptinject.HijackHateHumans \
    --concurrency 1 --timeout 150s \
    -o out/scan.jsonl --html out/scan.html \
    --config '{
      "uri": "http://127.0.0.1:8800/none/chat",
      "method": "POST",
      "headers": {"Content-Type": "application/json"},
      "req_template_json_object": {"message": "$INPUT"},
      "response_json": true,
      "response_json_field": "reply"
    }'

Choose what to run

• --probe a,b names probes; --probes-glob "promptinject.*,latentinjection.*" selects families; --all runs everything.
• --buff and --buffs-glob add prompt transformations, such as the encoding and translation buffs (lesson 11).
• --detector and --detectors-glob choose the detectors.
• --refusal-pattern "…" tells the mitigation detectors a phrase your own guardrail uses when it refuses, so a custom refusal is not mistaken for a failure.
• --setup, --prepare and --cleanup run shell commands before, between and after probes, for example to fetch a token.

Read the output

The terminal shows a table with the probe, the detector, whether the attempt passed, a score and a status, then a line such as “Overall: 0 passed, 0 review, 15 failed, 0 errored”. In our run, promptinject.HijackHateHumans sent 15 prompts, and the rogue-string detector marked all 15 as vulnerable against an unprotected assistant.

The JSONL file has one object per attempt: probe, prompt, response, detector, scores, passed, verdict, status, a metadata block (for example the triggers that were looked for) and a timestamp. The HTML file presents the same results. Filter on "verdict": "vuln" to read the failing replies:

$ grep -c '"verdict":"vuln"' out/scan.jsonl
$ grep '"verdict":"vuln"' out/scan.jsonl | head -3

Pitfalls we hit

• Concurrency: too many parallel requests against a local model.
• Counts differ by probe: one probe sent 15 prompts, another sends more. Compare failures over attempts, not raw failures.
• A detector is a rule. Read the failing replies before you trust a failure count (lesson 17).

Try it

1. Run promptinject.* against an endpoint you own with concurrency 1.
2. Add --buffs-glob "encoding.*" and note how the failure count changes.
3. Write the result as failures over attempts with an interval (lesson 18).