The FireAI Security Blog

By FireAI Security & Research Team · Published

Advanced Threat Detection on macOS: What the Layers Actually Do

Advanced Threat Detection on macOS: What the Layers Actually Do

Every product that calls itself “advanced threat detection” for the Mac is built on a small number of mechanisms, most of them documented by Apple. Knowing which mechanism a given feature relies on tells you what it can genuinely see, and what it is blind to. This article walks through those layers one by one: what a signature is, what notarization checks, what Apple’s Endpoint Security framework exposes to security software, what behavioural analysis means in practice, and what a model watching network connections can and cannot judge.

The short version is that no single layer sees everything. A file scanner never sees a connection. A network monitor never sees a file. Products that are honest about that boundary are more useful than products that blur it.

Layer one: signatures (XProtect)

The oldest detection technique is a signature: a pattern that matches a known piece of malicious software. On the Mac this job belongs to XProtect. Apple’s Platform Security Guide, in the page Protecting against malware, describes XProtect as signature-based, and says it runs at three moments: when an app is first launched, when an app has been changed on disk, and when XProtect’s signatures are updated.

Apple is candid about the reach of this approach. The same page notes that XProtect’s rules are “more generic than a specific file hash, so it can find variants that Apple hasn’t seen”, and describes a remediation engine that removes infections after the fact. What a signature cannot do, by construction, is match something nobody has written a rule for yet. That is the gap every other layer exists to narrow.

Layer two: notarization and Gatekeeper

The second layer runs before an app opens for the first time. Apple’s page on Gatekeeper and runtime protection explains that Gatekeeper verifies a downloaded app is from an identified developer, has been notarized by Apple, and has not been altered, and that it asks for the user’s approval before opening downloaded software. Notarization itself is a scan Apple performs on Developer ID-signed software before it ships; Apple’s guide adds that it can later issue a revocation ticket for software found to be malicious, even if it was previously approved.

This layer is a check at the door. It answers “do we know who built this, and did Apple find anything obviously wrong with it?” It does not follow the app once it is inside. An app that passes the door and then behaves badly, or an app whose behaviour changes after an update, is out of Gatekeeper’s sight from that point on.

Layer three: the Endpoint Security framework

Third-party detection products on the Mac, the category usually called EDR (endpoint detection and response), are largely built on one Apple API: the Endpoint Security framework. Apple introduced it in macOS 10.15 so that security software could receive a stream of system events from user space (a process being launched, a file being opened or written, a task being created) instead of loading a kernel extension, which Apple has been retiring.

This is the layer that sees process trees: which process spawned which, with what arguments, touching which files. It is also the layer where “behavioural analysis” usually lives. Behavioural detection means comparing what a process does against known attacker techniques rather than against a known file. The public reference for those techniques is the MITRE ATT&CK matrix for macOS, which catalogues tactics from initial access and persistence through command and control and exfiltration, each broken into specific, observable techniques.

Behavioural analysis has a real cost: an app that legitimately writes launch agents, reads the keychain or spawns shell commands looks, from the event stream alone, much like one doing it for bad reasons. Every EDR vendor tunes that trade-off differently, and none of them publish enough detail for an outsider to compare. What can be said with confidence is that this layer requires either a paid product or a fair amount of expertise to run. For individuals, the non-profit Objective-See Foundation publishes free, open-source tools built on the same Apple frameworks, such as TaskExplorer for inspecting running processes and KnockKnock for listing what is set to launch automatically.

Layer four: memory and file content

Two kinds of analysis sit below even the Endpoint Security layer: inspecting a file’s contents in depth (static analysis, which is what an antivirus scanner does) and inspecting a running process’s memory. These are the only ways to know what a program actually contains, and they are the domain of antivirus and forensic tooling. It is worth saying plainly, because marketing copy often implies otherwise: a firewall, including FireAI, does none of this. FireAI does not scan files, does not read process memory and is not an antivirus.

Layer five: the network

The last layer is the one the others cannot reach. Nearly everything an attacker wants to do after getting code onto a Mac involves the network: fetch a second stage, report in to a control server, send stolen data out. In the ATT&CK matrix those steps have their own columns, command and control and exfiltration, precisely because they are a distinct, observable phase.

A network layer sees a different set of facts than the layers above. For each new outbound connection it knows which signed binary opened it, the destination host, IP and port, the protocol, whether the connection is encrypted, and the timing. That is enough for a reviewer, human or model, to make several useful judgements:

  • Destination reputation: is the host or IP on a public blocklist such as abuse.ch, Spamhaus, Phishing Army or OpenPhish, or is it a Tor exit node? Threat-intelligence feeds are a list lookup, not a guess, and they can be applied locally without sending the traffic anywhere.
  • First-seen binary: is this the first time this code signature has ever opened a connection on this Mac? A freshly installed, unsigned tool that immediately connects to an unfamiliar IP is a different risk than Safari connecting to a CDN.
  • Port and protocol oddities: a text editor talking on port 4444, an app using raw IP addresses instead of hostnames, or a connection to a country the app has no business in.
  • Plain-HTTP credentials: if a connection is unencrypted, the payload is readable on the Mac before it leaves, and a card number, password or API key travelling in clear text can be stopped at that moment.

This is what FireAI’s Private AI review does. A small on-device model (an optional 1.5 GB download) looks at each connection from an unknown app, using exactly the facts listed above, and either blocks it, flags it, or lets it through with a plain-language reason you can read and reverse. The judgement is made on the Mac; the traffic is never sent to HisnLabs or anyone else for analysis.

The limits are just as concrete. A network reviewer cannot see file contents, so it cannot tell you a download is hostile before it runs. It cannot see memory, so it cannot detect an injected payload inside a trusted process, and if hostile code hides behind a signed, allowed app, its traffic inherits that app’s permissions. It cannot decrypt TLS, so for an encrypted connection it judges the destination and the pattern, not the payload. And a reputation feed only knows about destinations someone has already reported.

Putting the layers together

Read as a stack, the picture is coherent. Notarization and Gatekeeper decide whether code gets in. XProtect removes what is already known to be bad. Endpoint Security-based tools watch what processes do. Antivirus and forensics read file contents and memory. The network layer watches what leaves. Each answers a question the others cannot, and the useful test for any “advanced detection” claim is simply: which of these layers is it, and what does that layer see?

It also explains why the layers are not in competition. An EDR product and a per-app firewall can run on the same Mac and never look at the same fact: one reads the process tree, the other reads the connection table. A company that already pays for endpoint detection gains little from a second process monitor, but may gain a great deal from being able to say, per app and per destination, what is allowed to leave. A home user with no EDR at all gets, from the network layer alone, the one signal that the Apple layers never surface.

For most Mac owners, the practical combination is Apple’s built-in layers, kept up to date, plus visibility into the one phase Apple does not show you: which apps are connecting where. That is a narrower promise than “stops every threat”, and it is the one that can actually be kept.

How FireAI and HisnLabs fit in

FireAI is the network layer of that stack and nothing more: it does not scan files or read memory, but it is the one place every app’s outbound connection has to pass through, reviewed on your Mac by a model that explains what it saw.

FireAI is HisnLabs’ own product: an on-device AI firewall for Mac. It shows every connection your apps make, in plain language, and lets you decide what leaves your Mac — its AI runs locally, so your traffic is never sent to us or anyone else. HisnLabs’ security research team is the group that keeps that decision-making accurate: cataloguing which domains are ordinary telemetry versus a real product, tracking the country and network behind a connection, and training the on-device model (its Autopilot feature) on real traffic patterns, all without any of it leaving your Mac.

You can read the technical decisions behind it, or try FireAI for 17 days, at FireAI, by HisnLabs.

Sources