Probing the decision boundary, and how defenders spot it

A detection model turns an input into a score, then compares that score with a threshold. Above it, the sample is blocked or flagged; below it, the sample passes. Mathematicians call the frontier between the two outcomes the decision boundary. You never see it directly, but it shapes every verdict the model gives.

An attacker who cannot read a model’s internals can still learn about it by watching its behaviour. This is called black-box probing: submit an input, observe the verdict, change something, submit again. Each answer is a small clue about where the boundary lies. With enough answers, a pattern emerges. NIST’s adversarial ML taxonomy lists this kind of query-based access as one of the main conditions that make evasion and model-extraction attacks possible.

Why query access matters so much

Probing depends entirely on being allowed to ask many questions and getting useful answers back. Three things help the attacker: an unlimited number of queries, answers that are detailed (a precise score rather than a simple yes or no), and nobody noticing that the same person is asking the same kind of question over and over. Remove any one of those and probing becomes far slower and far more visible.

Probing leaves traces

The good news for defenders is that boundary probing is rarely subtle when you look at the right place. Legitimate users submit a variety of unrelated files or requests. Someone mapping a threshold submits many samples that are almost identical, differing in small, methodical ways. That pattern is itself suspicious, and MITRE ATLAS lists reconnaissance of ML systems and gaining ML model access as distinct stages of an attack, which gives your team something concrete to hunt for.

On a single computer, the equivalent signal is an unfamiliar program repeatedly contacting a scanning or verdict service, or trying many variants of the same connection. Watching which programs talk to which services, and how often, turns quiet reconnaissance into something a person can see.

Designing so the boundary matters less

• Do not publish your exact thresholds or feature weights, and avoid returning raw scores to untrusted callers.
• Combine a statistical model with rules and behavioural checks, so crossing one boundary is not enough to pass.
• Log every query to scoring services with who asked and when, and alert on unusual volume or repetition.
• Retrain and re-evaluate regularly; a moving target is harder to map.
• Assume a determined attacker will eventually find some gap, and make sure another layer is watching when they do.

Where FireAI fits

FireAI does not expose a scoring service that others can query: its models run on the Mac and answer only that Mac’s own connections. Its Investigate page shows a risk score out of 100 with the reasons behind it to the person who owns the Mac, not to the program being judged. And because every new app must ask before connecting, a program that suddenly starts talking to unfamiliar services is visible on the world map instead of silent.