Mitigations and guardrails: spotlighting and defence in depth

Study Microsoft’s spotlighting techniques, the layers that surround them, the open-source guardrail toolkits, and why no single defence is enough.

此页面暂时只有英文版。

After five lessons on attacks, the series’ seventh episode turns to defence. Its running example is the email assistant: an attacker’s message contains a payload asking the assistant to search for sensitive messages and send them out. The question is how the application can help the model tell trusted input from untrusted input ^[1].

Spotlighting

Spotlighting, published by Microsoft researchers in March 2024, is described as “a family of prompt engineering techniques that can be used to improve LLMs’ ability to distinguish among multiple sources of input”. Its key idea is to apply a transformation to an input that provides “a reliable and continuous signal of its provenance” ^[2]. The series presents three variants, each building on the last.

The three spotlighting variants as presented in Episode 7.
Variant	What the application does	Known weakness
Delimiting	Marks the start and end of untrusted text with special tokens and tells the model never to obey instructions between them	An attacker can add their own delimiters or close the block
Data marking	Interleaves a special character through the whole untrusted text and tells the model that the marked text is data	Languages written without spaces can defeat a marker placed between words
Encoding	Transforms the untrusted text, for example with base64, and tells the model to decode it for the task without following it	Needs a model capable enough to read the encoding

In the authors’ experiments, spotlighting reduced the attack success rate from above 50 per cent to below 2 per cent with minimal impact on task efficacy ^[2]. Treat that as evidence that the idea works under the paper’s conditions, not as a guarantee for your application, and note that the episode itself lists how each variant can be bypassed.

Defence in depth

Spotlighting is one layer. The sources in this course converge on several more:

Input and output filtering, including Microsoft’s four layers for Skeleton Key: input filtering, a hardened system message, output filtering and abuse monitoring ^[3].
OWASP’s prompt injection list: constrain behaviour, validate output formats, filter, enforce least privilege, require human approval, segregate external content and test adversarially ^[4].
Prompt classification and modification before the model, which Anthropic found helpful against many-shot jailbreaking, alongside fine-tuning that only delayed the attack ^[5].
Permission limits on tools (the previous lesson).

Open-source guardrail toolkits

Two open-source projects are widely used for the filtering layer. NeMo Guardrails, under the Apache 2.0 licence, describes itself as “an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems” and offers input, dialog, retrieval, execution and output rails ^[6]. Llama Guard, from Meta researchers, is an instruction-tuned model that classifies both prompts and responses against a safety taxonomy ^[7]. Lesson 15 shows how to compare such guardrails fairly. A caution: a guardrail is also software that can be probed and bypassed, so it needs red teaming like everything else.

The honest conclusion

OWASP says it is unclear whether fool-proof prevention exists for prompt injection, given how these models work ^[4]. The practical position is therefore layered defence plus continuous testing: each change to the model, the prompt, a guardrail or a tool is a reason to re-run your attack suite. Red teaming is how you learn whether a mitigation actually holds.

要点

Spotlighting gives the model a signal of where input came from, through delimiting, data marking or encoding.
Each spotlighting variant has known bypasses, so it is a layer, not a fix.
Combine filtering, constrained behaviour, least privilege, human approval and monitoring.
NeMo Guardrails and Llama Guard are open-source building blocks, and they need testing too.
Re-run your attack suite after every change.

自我检测

1. What problem does spotlighting address?
- Telling trusted from untrusted input — 正确。
- Reducing model size
- Speeding up inference
- Translating text
It gives a signal of provenance so the model can distinguish sources.
2. Which variant interleaves a marker through the untrusted text?
- Delimiting
- Data marking — 正确。
- Encoding
- Caching
Data marking places a special character throughout the data.
3. Why is spotlighting not a complete fix?
- Each variant has bypasses, and attacks adapt — 正确。
- It is illegal
- It needs no model
- It only works on images
The episode itself lists how each variant can be bypassed.
4. What should you do after changing a guardrail?
- Re-run your attack suite — 正确。
- Nothing
- Delete the logs
- Turn off monitoring
A change can open or close gaps, so retest.

用 FireAI 动手做

在你自己的 Mac 上练习这节课的内容。

来源

在你的 Mac 上实践

免费试用全部功能 17 天，无需绑定银行卡。

下载 Mac 版文档