Beyond jailbreaks: leakage, hallucination, poisoning and images

Jailbreaks get the attention, but a model can fail a user in other ways, and a scanner that only tests jailbreaks misses most of them. garak’s README lists the failures it probes for as hallucination, data leakage, prompt injection, misinformation, toxicity generation and jailbreaks, among others ^[1]. This lesson covers the rest of that map.

Leakage

• System prompt leakage. OWASP lists it as LLM07. The series’ advice is to never put confidential data in a metaprompt, because it can usually be extracted ^[2]. garak’s sysprompt_extraction probes try direct requests, encoding tricks and role-play.
• Training-data extraction. Carlini and colleagues showed that an adversary can recover individual training examples by querying a language model ^[3]. garak’s leakreplay probes try to make a model replay an excerpt of a document in order to show whether it was used in training, and its divergence module adapts a repeated-token attack from earlier research.
• Sensitive information in general. OWASP lists LLM02, Sensitive Information Disclosure, which covers both the model and the application around it ^[4].

Invented content

The first episode names fabrications as one of the three built-in risks ^[5]. garak’s snowball probes check whether a system gives wrong answers to complex reasoning questions instead of the right answer or none, and its package-hallucination probes try to get code generation that names non-existent, and therefore insecure, packages. The second matters in practice because an attacker can register a package name that a model tends to invent, a pattern covered in the blog article on hallucinated packages.

Poisoned data and backdoors

Some attacks happen before the model is deployed. Carlini and colleagues showed that poisoning web-scale training datasets is practical, stating that their attacks “could, today, poison 10 popular datasets” and describing two methods, split-view and frontrunning poisoning ^[6]. Hubinger and colleagues trained models with backdoors and found that the behaviour can be made persistent, so that it is not removed by standard safety techniques including supervised fine-tuning, reinforcement learning and adversarial training ^[7]. OWASP groups these under LLM04, Data and Model Poisoning, and LLM03 covers the wider supply chain ^[4]. For a team that fine-tunes or downloads models, the red-team questions are about provenance: where did the data and weights come from, and who could have changed them?

Unsafe handling of output

What the model writes becomes input to something else: a web page, a database query, a shell. OWASP’s LLM05, Improper Output Handling, is about insufficient validation and sanitisation of model output ^[4]. garak’s web_injection probes show the idea: a prompt injection makes the model emit markdown that, when the application renders it as HTML, can run script. The test and the fix are the same as for any untrusted input: treat model output as untrusted data.

Images and other modalities

Qi and colleagues showed that a single visual adversarial example can universally jailbreak an aligned model, compelling it to heed a wide range of harmful instructions through the visual input channel ^[8]. garak’s visual_jailbreak module follows a different method, FigStep, which uses typographic visual prompts. The series’ automation episode also runs a multi-turn attack against an image-generation model ^[9]. Every new input channel is a new place for instructions to hide, and filters built for text may not see them.

Odd tokens

garak also carries glitch probes, which use tokens that lead to unusual model behaviour when processed. They are a reminder that the input to a model is a token stream that nobody reads as a person would.

Machine-learning-level attacks

Classic adversarial machine learning also applies to the models inside your products: evasion, poisoning and extraction attacks against classifiers, for example. The open-source Adversarial Robustness Toolbox, under the MIT licence, is a Python library for this kind of testing ^[10]. Module 10 covers the landscape.

Testing guidance

1. Add leakage probes with planted canaries: a fake secret in the system prompt, a fake record in the retrieval store.
2. Check outputs before they reach any interpreter, and test that rendering is safe with a harmless script marker.
3. For packages, check that code suggestions name only packages that exist and are the intended ones.
4. Ask provenance questions about data and weights, because no probe can detect a good backdoor.