The adversarial machine learning landscape

Ten years ago, most defensive tools matched known patterns: a file hash, a signature, a blocked address. Today, endpoint detection and response (EDR) products, email filters, network firewalls and security information and event management (SIEM) platforms increasingly use machine learning (ML) models to judge whether something looks dangerous. That shift made detection far better at recognising new variants of old attacks. It also created a new kind of target. If a model decides what gets flagged, an attacker who understands the model can try to change its decision.

This is the field of adversarial machine learning: the study of how ML systems can be attacked, and how to defend them. It is not science fiction. The US National Institute of Standards and Technology (NIST) publishes a dedicated taxonomy of these attacks (NIST AI 100-2), and MITRE, the organisation behind the ATT&CK framework that security teams use to describe attacker behaviour, maintains ATLAS, a knowledge base of real-world tactics and techniques against AI systems.

Where the models sit in a security stack

• Endpoint: a classifier scores files and process behaviour on laptops and servers.
• Network: a model scores connections, domains or traffic patterns for signs of command and control or data theft.
• SIEM and security analytics: models rank alerts, spot unusual logins and group related events so analysts see what matters first.
• Email and web: models judge whether a message or page is phishing.

Each of these models takes an input (a file, a connection, a log line) and produces an output (a score or a label). The attacker’s goal is always one of a few things: make a malicious input look benign, make benign inputs look malicious so defenders drown in noise, corrupt what the model learns, or learn enough about the model to copy or bypass it.

The four families of attack

This course looks at each family from the defender’s side. The goal is not to teach anyone how to evade a detector. It is to understand why detectors can fail, so you design systems that still catch an attacker when one layer is fooled.

Why security models are an attractive target

A security model is valuable to an attacker precisely because it stands between them and success. Three properties make it tempting. First, security products are widely deployed, so the same model protects thousands of organisations; a technique that works once may work everywhere. Second, many models can be queried: the attacker can submit a sample and see what happens, the way a lock-picker practises on the same lock you bought. Third, a model’s mistakes are quiet. A missed detection does not ring a bell. That silence is the subject of a later lesson.

Practical takeaways for defenders

• Treat every ML component as part of your attack surface: list which models make security decisions and who can reach them.
• Ask vendors how their models are hardened against evasion and poisoning, and how quickly they retrain.
• Never let one model be the only thing standing between an attacker and your data.
• Map AI-specific risks with ATLAS and NIST AI 100-2 so they sit next to your ATT&CK coverage, not in a separate silo.

Where FireAI fits

FireAI uses on-device models to explain connections and, in Autopilot, to answer routine connection prompts on a Mac. It is designed with the lessons of this course in mind: its decision model never learns from its own decisions, it never allows an unsigned app on similarity alone, and every decision becomes a visible rule a person can review and undo. It is one network layer, not an antivirus, and it works best alongside the other layers this course describes.