Direct prompt injection, step by step

Most real attacks on language-model applications begin with prompt injection. The series’ third episode explains it with a typical application, an email summarisation assistant, and the same structure applies to a support bot, a scheduling agent or an HR assistant ^[1].

The anatomy of an AI application

The model is not only responding to what someone types. It processes everything the application passes in, which usually has three parts:

• The system prompt, also called the metaprompt: hidden instructions that guide behaviour, tone and output format, such as “you are an email summarisation bot”.
• The user input: what the person types, such as “what are my emails?”.
• Retrieved data: emails, web search results, documents, tool responses or database output that the application fetches.

All of it is joined in a fusion step and flattened into a single token sequence, the context window. The model does not inherently know where one part ends and the next begins. It sees a stream and predicts what follows. That is what makes injection possible: if an attacker can influence any part of the input, they may override instructions, extract information or hijack behaviour.

The attack

In a direct injection the attacker changes the user input. Instead of a question, the user types an instruction, for example to send emails to an external account, which is a form of data exfiltration. After the fusion step the model sees the system prompt and the new instruction together, and it does not know which came from the developer and which from the user. The effect spreads: what it replies, what actions it triggers through tools or agents, and what instructions it carries forward ^[1].

A case from the series

The episode recounts a 2023 case in which a car dealership’s chatbot was told to agree with everything and to close every message with a phrase calling the reply a binding offer, and was then asked to sell an SUV for one dollar. The bot agreed, repeatedly. As the presenter tells it, the model was not connected to any payment system, so there was no financial loss, but the screenshots spread widely and the dealership took the bot down. The lesson drawn is general: these models are trained to follow instructions, not to understand intent, so “it didn’t take code or elevated access, just clever prompting” ^[1].

What the standards say

OWASP defines a prompt injection vulnerability as occurring “when user prompts alter the LLM’s behavior or output in unintended ways”. It separates direct injections, where a user’s prompt directly alters the model’s behaviour, from indirect ones, where content from external sources does (the next lesson) ^[2]. Its prevention list names seven measures:

1. Constrain model behaviour.
2. Define and validate expected output formats.
3. Implement input and output filtering.
4. Enforce privilege control and least-privilege access.
5. Require human approval for high-risk actions.
6. Segregate and identify external content.
7. Conduct adversarial testing and attack simulations.

OWASP adds a caution that matters for how you report results: given the stochastic nature of the models, “it is unclear if there are fool-proof methods of prevention for prompt injection” ^[2].

What a tester checks

• Can the developer’s instructions be overridden by user text, and how easily? Test the whole system through its real interface, not only the model.
• Can the system prompt be extracted? The series’ rule of thumb is never to put confidential data in a metaprompt, because it can usually be extracted ^[3].
• Can anything the model says or does be changed to something the developer did not intend, such as an action, a data flow or a policy?

To practise safely, plant a canary: a fake secret such as a made-up token in the system prompt of your own test application, and an automated check that detects when it appears in a reply. Microsoft’s open-source AI Red Teaming Playground Labs follow this pattern with twelve challenges, run locally with Docker Compose against an Azure OpenAI or OpenAI endpoint you provide. The direct-injection pair is credential exfiltration (labs 1 and 7, easy and medium) and extracting a secret from the metaprompt (labs 2 and 8): the first asks you to get a model to reveal the contents of a protected file, the second a secret word held in its hidden instructions ^[4].