Перейти к содержимому
← AI red teaming: techniques, open-source tools and automated pipelines

Урок 9 из 18 · 8 мин

Agents, tools and excessive agency

Learn why giving a model tools turns a bad sentence into a bad action, what OWASP calls excessive agency, and how to test the permissions behind an agent.

Пока эта страница на английском.

A chat reply that is wrong is a problem. A tool call that is wrong is an incident. Every diagram so far has had an output that triggers actions through agents or tools, and in several of them the model’s output also becomes new instructions fed back into the model [1]. This lesson is about that action layer, which decides how much damage a successful injection can do.

The chain: injection plus a tool

Combine the previous two lessons with a capability. A hidden instruction in an email arrives through retrieved data. The model, which cannot tell data from instructions, follows it. The model has a tool that can read mail and one that can send it. The result is the series’ exfiltration example, where the model searches for particular messages, encodes what it finds and sends it to an attacker-controlled address, then summarises the email as usual and says nothing [2]. No exploit code was needed; the permissions did the work.

What OWASP calls excessive agency

OWASP’s 2025 list includes LLM06, Excessive Agency. It starts from the observation that an LLM-based system is often granted agency by its developer, “the ability to call functions or interface with other systems via extensions”, and names three root causes [3]:

  • Excessive functionality: an extension offers operations the application does not need.
  • Excessive permissions: an extension has broader access to downstream systems than its task requires.
  • Excessive autonomy: high-impact actions happen without verification.

Its prevention advice reads like a least-privilege checklist: minimise extensions and their functionality, avoid open-ended extensions, minimise permissions, run extensions in the user’s own context, require user approval, apply complete mediation, and sanitise inputs and outputs. OWASP adds that logging, monitoring and rate limiting can limit damage but do not remove the vulnerability itself [3].

How to test an agent

  1. Inventory. List every tool the agent can call, with its permissions, the data it can reach and any approval gate. You cannot assess what you have not listed.
  2. Ask the three questions per tool: is the function needed, are the permissions minimal, is a person confirming the high-impact actions?
  3. Plant a harmless action marker. In a sandbox, give the agent a tool that writes a fixed line to a log file you own, then use indirect injection (lesson 4) to try to make it call that tool from a document. If it does, a real tool would have been called too.
  4. Check the boundaries. Can the agent act in another user’s context? Can it reach systems outside its task? Can a tool’s output become a new instruction?
  5. Record the result as a permission finding: the right fix is usually to remove capability, not to add more prompt text.

Defences that scale with the risk

  • Remove tools the task does not need. Capability you never grant cannot be abused.
  • Narrow permissions so that a hijacked agent can do little.
  • Put a person in the loop for sending, deleting, paying and other irreversible actions.
  • Make outbound behaviour visible, so an unexpected connection or message stands out.

Главное

  • A tool turns a successful injection from a wrong sentence into an action.
  • OWASP’s excessive agency has three root causes: functionality, permissions and autonomy.
  • Least privilege and human approval limit the damage when the model is fooled.
  • Test agents by inventorying tools and planting harmless action markers in a sandbox.

Проверьте себя

  1. 1. What are the three root causes of excessive agency?

    • Excessive functionality, permissions and autonomy — Верно.
    • Speed, cost and size
    • Tokens, vectors and attention
    • Latency, logging and caching

    OWASP lists these three in LLM06.

  2. 2. What does an agent’s tool inventory help with?

    • Assessing what a hijacked agent could do — Верно.
    • Training the model
    • Lowering cost
    • Choosing a font

    The reachable actions set the ceiling on damage.

  3. 3. What is a safe action marker?

    • A tool that writes a fixed line to a log you own — Верно.
    • A tool that emails customers
    • A tool that deletes files
    • A real payment

    It shows whether the agent can be steered into a call without harm.

  4. 4. Which control best limits irreversible damage?

    • A person approving high-impact actions — Верно.
    • A longer system prompt
    • A faster model
    • More logging only

    OWASP lists user approval; logging alone does not prevent the action.

Примените на практике в FireAI

Примените этот урок на своём Mac.

Источники

Примените это на своём Mac

Все функции бесплатно на 17 дней, без банковской карты.

Скачать для Mac Документация