Lesson 3 of 7 · 9 min
Evasion and data poisoning: how models are fooled, and hardened
Attackers can fool a model at decision time (evasion) or corrupt what it learns (poisoning). Learn how each works in principle and the defensive practices that make models harder to fool.
A machine learning model does not understand a file or a connection the way a person does. It sees features: measurable properties such as structure, sequences of actions, metadata or timing. Its verdict is a function of those features. That is both its strength, because it generalises to variants it has never seen, and its weakness, because its view of the world is only as good as the features and data it was given.
Evasion: changing the input, not the goal
In an evasion attack, the adversary keeps what their code does but changes how it looks to the model. The research literature calls these changes perturbations: modifications to the input that alter the model’s features without breaking the input’s function. In image recognition, the classic demonstration is a picture altered so slightly a person sees no difference, yet the model labels it as something else entirely. In security, the equivalent is a harmful program presented in a way the model has not learned to associate with harm. NIST AI 100-2 classifies this as an evasion attack; MITRE ATT&CK has long documented the older, non-ML cousin of the idea as obfuscation (T1027) and impairing defences (T1562).
We deliberately stop at the concept. This course does not describe how to craft evasive samples. What matters for a defender is the lesson it teaches: a model that relies on surface features can be steered, so a good defence looks at things an attacker cannot easily change without giving up their goal, such as what a program actually does and where it sends data.
Poisoning: corrupting what the model learns
Poisoning attacks the training stage instead of the decision stage. If an adversary can influence the data a model learns from, or the feedback it receives, they can teach it the wrong lesson: to ignore a family of attacks, or to trust a specific pattern that works as a hidden trigger. The risk grows whenever a model learns from data it did not carefully vet, such as public datasets, user-submitted samples or its own previous outputs.
| Attack | When it strikes | Main defences |
|---|---|---|
| Evasion | At decision time, one input at a time | Behavioural signals, feature choices that are costly to fake, ensembles, adversarial testing by the defender, layered detection |
| Poisoning | During training or retraining | Data provenance and review, limiting who can submit training data, anomaly checks on new data, never training on the model’s own unchecked outputs |
How defenders harden models
- Know your data: record where every training sample came from, who labelled it and when (provenance). Treat third-party datasets and downloaded models as untrusted until reviewed.
- Robust training: include hard, realistic variants during training so the model learns what matters rather than superficial quirks.
- Ensembles and diversity: combine models that look at different features, plus rules, so fooling one does not fool all.
- Red-team your own models: authorised internal testing, before attackers do it for you, is how teams find blind spots safely.
- Monitor after deployment: a sudden drop in detections for a known family, or a change in what the model flags, deserves investigation.
- Keep humans in the loop for high-impact decisions and for anything the model is unsure about.
Where FireAI fits
FireAI’s Autopilot is built to resist the self-poisoning loop: its decision model never learns from its own decisions, only from a person’s answers and corrections, and a broken code signature always goes to the underlying model rather than a shortcut. “Forget What FireAI Learned” resets what it has learned if its habits ever look wrong. It also judges connections on facts that are hard to fake without giving up the attack, such as the app’s verified code signature and where data is actually being sent, rather than on appearance alone.
Key takeaways
- Evasion changes an input at decision time; poisoning corrupts what a model learns.
- Models judge features, so defences should rely on signals an attacker cannot cheaply fake.
- Data provenance, robust training and ensembles make models harder to fool.
- Never let a model retrain on its own unchecked outputs.
Check yourself
1. What is the key difference between evasion and poisoning?
- Evasion targets training data; poisoning targets decisions
- Evasion fools the model at decision time; poisoning corrupts what it learns during training — Right.
- They are two names for the same attack
- Poisoning only affects images
Evasion manipulates a single input when the model decides; poisoning manipulates the data or feedback the model learns from.
2. Which practice most directly reduces poisoning risk?
- Returning precise scores
- Tracking the provenance of training data and limiting who can contribute it — Right.
- Using a larger model
- Retraining the model on its own verdicts
Poisoning needs influence over training data. Knowing and controlling where data comes from removes that lever.
3. Why is it risky for a security model to retrain on its own outputs?
- It uses more electricity
- Any mistake it makes, including one an attacker induced, becomes part of what it learns — Right.
- It makes the model slower
- It is illegal
Self-training turns errors into lessons. An attacker who gets one decision wrong can make the mistake permanent.
Do it with FireAI
Put this lesson into practice on your own Mac.
- Autopilot: FireAI decides the easy connections for you — Let FireAI clear the easy decisions on its own, and always see why.
- Use Autopilot safely: what it decides, and how to correct it — Turn Autopilot on with a clear view of what it’s doing, and fix anything it gets wrong.
- Investigate a connection — Decide with the facts in front of you, not a vague warning.
- Threat lists (opt-in) — Check your traffic against public threat data without sending it anywhere.
- The World map — See where your data actually goes, not just a hostname you’d have to look up yourself.
- Security modes: Home, Coffee shop, Paranoid, Under attack — Match FireAI’s strictness to where your Mac actually is, in one tap.
Sources
- NIST AI 100-2 E2025: Adversarial Machine Learning taxonomy
- MITRE ATLAS
- MITRE ATT&CK T1027: Obfuscated Files or Information
- MITRE ATT&CK T1562: Impair Defenses
- UK NCSC: Principles for the security of machine learning
- UK NCSC and partners: Guidelines for secure AI system development
Put it into practice on your Mac
Try every feature free for 17 days, no card needed.