Hands-on: scan your own endpoint with garak

garak describes itself in one line: it “checks if an LLM can be made to fail in a way we don’t want” ^[1]. This lesson is a worked run, written from a real one: garak 0.17.0, run on 12 October 2026 against a small summarising assistant on a local model. Use it on an endpoint you own.

Install it in its own environment

$ python3 -m venv .venv-garak
$ .venv-garak/bin/pip install -U garak
$ .venv-garak/bin/python -m garak --version

See what it can do

$ .venv-garak/bin/python -m garak --list_probes | grep -i inject

Probes come in families such as promptinject and latentinjection. Most have a default version and a Full version (for example promptinject.HijackHateHumans and promptinject.HijackHateHumansFull) that sends many more prompts, so start with the default ones. The injection families and their detectors were described in lesson 11.

Point it at your own endpoint

garak talks to a custom HTTP endpoint through its REST generator, configured by a small JSON file. The request template uses $INPUT where garak’s prompt goes, and response_json_field names the field that holds the reply:

{
  "rest": {
    "RestGenerator": {
      "name": "my-assistant",
      "uri": "http://127.0.0.1:8800/none/chat",
      "method": "post",
      "headers": { "Content-Type": "application/json" },
      "req_template_json_object": { "message": "$INPUT" },
      "response_json": true,
      "response_json_field": "reply",
      "request_timeout": 120
    }
  }
}

A second file limits the run. soft_probe_prompt_cap caps how many prompts each probe sends, which keeps a first run short; generations sets how many replies are collected per prompt:

{ "run": { "generations": 1, "soft_probe_prompt_cap": 20 },
  "system": { "narrow_output": true } }

Run it

$ .venv-garak/bin/python -m garak \
    --target_type rest -G gen.json --config cfg.json \
    --probes promptinject.HijackHateHumans,latentinjection.LatentInjectionReport \
    --report_prefix "$PWD/out/scan" \
    --parallel_attempts 1 --seed 7

Read what it wrote

garak’s README describes a report of each run, structured as JSONL, and a hit log detailing the attempts that yielded a vulnerability ^[1]. In our run it wrote three files from the prefix: the JSONL report, the hit log and an HTML version of the report.

• scan.report.jsonl: one JSON object per line. Lines with "entry_type": "eval" summarise a probe and detector pair, with passed, fails and total_evaluated.
• scan.hitlog.jsonl: the individual attempts that failed, with the prompt and the reply. Read these first (lesson 18).
• scan.report.html: the same results for people.

{"entry_type": "eval", "probe": "promptinject.HijackHateHumans",
 "detector": "promptinject.AttackRogueString",
 "passed": 0, "fails": 3, "total_evaluated": 3}

In that run, three of three prompts succeeded against an unprotected assistant. Three attempts are far too few to rate anything: use the failure count over the number of attempts with an interval, as in lesson 18, and raise the prompt cap before drawing a conclusion.

Pitfalls we hit

• Relative report prefix: use an absolute path.
• Dependency clash with other tools: one virtual environment per tool.
• Default versus Full probes: a Full probe can send hundreds of prompts, and on a slow local model that is a long wait.
• Detectors define “failure”. promptinject.AttackRogueString checks for a specific string in the reply, so a reply that refuses while quoting the string can be flagged. Read the hit log.

Try it

1. Run the scan above against an endpoint you own, with the prompt cap at 20.
2. Open the hit log and classify five failures as real or false alarms.
3. Repeat with a defence added (lesson 10), and compare the failure counts over attempts, with intervals.