How generative models work, and where they break

Attacks make sense once you know what the model is doing. Earlier machine-learning systems were built for one job, such as classifying emails or recognising faces. Generative models, in the words of the series’ second episode, do not classify or score: they generate. They sample from learned distributions and produce new text, code, images or audio by recombining patterns from training data, fitted to the prompt and context. A useful picture is a very capable autocomplete ^[1].

Training happens in stages

Traditional models usually have a single training stage. Generative models go through several, and the order matters for security.

1. Pre-training. The model reads very large collections of text and code and learns to predict what comes next, mostly without human labels. This takes months and large amounts of computing. It also bakes in whatever bias, misinformation or harmful content was in the data, so “the unsafe stuff goes in early”, as the episode puts it.
2. Post-training. Supervised fine-tuning turns a general language model into one that answers questions and follows instructions.
3. Safety post-training. Here the model is taught what to say and what not to say. Refusals and topic avoidance are mostly layered on at this stage, not built in from the start, which is why they can seem brittle: small changes in wording, tone or context can break them ^[1].
4. Red teaming, just before release, as part of a break-fix loop: testers find failures, builders patch them, and testers try again.

This is only the model side. Application developers add their own testing, guardrails and mitigations, which later lessons examine.

Tokens, vectors and attention

Models do not read letters or words. A tokenizer breaks text into tokens: whole words, parts of words, punctuation, numbers or emoji. Each token becomes a vector, a list of numbers that places it near related concepts, so “cat” and “kitten” sit close together. A transformer then lets the model attend to many parts of the input at once, tracking relationships across long stretches of text.

Two consequences matter to a red teamer. The way text is chopped into tokens can be exploited to slip content past filters that match strings. And the same ability to model context across a whole sequence is what makes models vulnerable to attacks that tamper with that context. The next lessons take these in turn.

Why safety training fails: two modes

Wei, Haghtalab and Steinhardt studied why jailbreaks succeed against models that have been safety-trained. They hypothesise two failure modes: “competing objectives and mismatched generalization” ^[2].

• Competing objectives: the model is trained both to be capable and helpful and to be safe, and a prompt can set these goals against each other, so that following instructions wins over refusing.
• Mismatched generalisation: safety training does not generalise to a domain in which the model nonetheless has capabilities, so the model understands an unusual input that its safety training never covered.

The authors argue that these modes point to a need for “safety-capability parity” and that scaling alone will not fix them ^[2]. This is the theory behind much of what follows. Many single-turn techniques work by setting capabilities against safety (personas, framing) or by moving to a form the safety training saw little of (encodings, unusual languages).

Outputs vary, so tests repeat

Generation is sampled, so the same input can give different outputs. A test that fails once may pass the next time, and the reverse. The series’ discussion of automation returns to this: repeatable, code-driven workflows are valuable precisely because the models are not always deterministic ^[3]. Lesson 14 shows how to report results honestly under that variation.