Evasion and data poisoning: how models are fooled, and hardened

A machine learning model does not understand a file or a connection the way a person does. It sees features: measurable properties such as structure, sequences of actions, metadata or timing. Its verdict is a function of those features. That is both its strength, because it generalises to variants it has never seen, and its weakness, because its view of the world is only as good as the features and data it was given.

Evasion: changing the input, not the goal

In an evasion attack, the adversary keeps what their code does but changes how it looks to the model. The research literature calls these changes perturbations: modifications to the input that alter the model’s features without breaking the input’s function. In image recognition, the classic demonstration is a picture altered so slightly a person sees no difference, yet the model labels it as something else entirely. In security, the equivalent is a harmful program presented in a way the model has not learned to associate with harm. NIST AI 100-2 classifies this as an evasion attack; MITRE ATT&CK has long documented the older, non-ML cousin of the idea as obfuscation (T1027) and impairing defences (T1562).

We deliberately stop at the concept. This course does not describe how to craft evasive samples. What matters for a defender is the lesson it teaches: a model that relies on surface features can be steered, so a good defence looks at things an attacker cannot easily change without giving up their goal, such as what a program actually does and where it sends data.

Poisoning: corrupting what the model learns

Poisoning attacks the training stage instead of the decision stage. If an adversary can influence the data a model learns from, or the feedback it receives, they can teach it the wrong lesson: to ignore a family of attacks, or to trust a specific pattern that works as a hidden trigger. The risk grows whenever a model learns from data it did not carefully vet, such as public datasets, user-submitted samples or its own previous outputs.

How defenders harden models

• Know your data: record where every training sample came from, who labelled it and when (provenance). Treat third-party datasets and downloaded models as untrusted until reviewed.
• Robust training: include hard, realistic variants during training so the model learns what matters rather than superficial quirks.
• Ensembles and diversity: combine models that look at different features, plus rules, so fooling one does not fool all.
• Red-team your own models: authorised internal testing, before attackers do it for you, is how teams find blind spots safely.
• Monitor after deployment: a sudden drop in detections for a known family, or a change in what the model flags, deserves investigation.
• Keep humans in the loop for high-impact decisions and for anything the model is unsure about.

Where FireAI fits

FireAI’s Autopilot is built to resist the self-poisoning loop: its decision model never learns from its own decisions, only from a person’s answers and corrections, and a broken code signature always goes to the underlying model rather than a shortcut. “Forget What FireAI Learned” resets what it has learned if its habits ever look wrong. It also judges connections on facts that are hard to fake without giving up the attack, such as the app’s verified code signature and where data is actually being sent, rather than on appearance alone.