Running a Local LLM on Apple Silicon for Security Triage

A model that classifies a network connection as worth a look or not, running on the same Mac that made the connection, changes three things at once: what leaves the machine, what it costs to run, and whether it keeps working when the network itself is the thing under suspicion. All three matter more for a security tool than for most other uses of a language model, which is why this article is specifically about the on-device case rather than calling an API.

Why on-device, specifically for triage

Sending a connection log line to a cloud model for classification means transmitting, for every single decision, which app on your Mac is talking to which host, on which port, right now. None of that is a secret in the way a password is, but it is exactly the kind of metadata a security-conscious setup tries to minimise sharing, and a tool whose entire purpose is deciding what your Mac is allowed to send elsewhere has an obvious reason not to itself depend on sending something elsewhere for every decision it makes. Running the model locally removes that dependency entirely: the log line never leaves the device, because there is no network call in the decision path at all.

The second reason is continuity. A cloud-based triage step is only as available as your internet connection and the vendor’s API, both of which are exactly the things that might be degraded or deliberately cut during a real incident. A model running in local memory keeps answering with the network down, the Wi-Fi off, or a suspected compromise in progress on the same link the API call would have used.

MLX: Apple’s own array framework

MLX is an array framework built by Apple’s machine-learning research team specifically for Apple silicon, with Python, C++, C and Swift APIs. Its own documentation describes a unified memory model as its defining design choice: arrays in MLX live in shared memory, and operations on them can run on any supported device — CPU or GPU — without the model’s weights being copied between separate memory pools first. That matters concretely for a laptop: a discrete-GPU machine has to copy a model’s weights across a bus into GPU memory before it can compute anything, which costs time and doubles the memory footprint; on a Mac’s unified memory architecture, the CPU and GPU already share the same physical memory, so there is nothing to copy.

mlx-lm, the companion package for running language models on MLX, installs with a single command and ships a default model out of the box:

$ pip install mlx-lm
# runs the default model (mlx-community/Llama-3.2-3B-Instruct-4bit)
$ mlx_lm.generate --prompt "How tall is Mt Everest?"
# or name a specific quantized model from the mlx-community hub
$ mlx_lm.generate --model mlx-community/Mistral-7B-Instruct-v0.3-4bit --prompt "..."
# interactive chat session instead of a single prompt
$ mlx_lm.chat
# convert and quantize a model yourself
$ mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 -q

llama.cpp: the portable option

llama.cpp is the older, more widely ported of the two, written in C/C++ with no Python runtime required at inference time. Its own README states the project’s position on this hardware directly: Apple silicon is treated as, in the project’s words, "a first-class citizen — optimized via ARM NEON, Accelerate and Metal frameworks," and its build documentation confirms that on macOS, the Metal GPU backend is enabled by default, with a build-time flag to disable it if you specifically want CPU-only inference.

$ git clone https://github.com/ggml-org/llama.cpp
$ cd llama.cpp
$ cmake -B build
$ cmake --build build --config Release
# Metal is on by default on macOS; add -DGGML_METAL=OFF to the first
# command above if you need to force CPU-only inference
$ llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# or run it as a local server instead of a one-shot command
$ llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

llama.cpp works from GGUF files, a single-file model format that bundles the quantized weights and everything needed to run them; the command above pulls one straight from a Hugging Face repository by name. MLX, by contrast, keeps closer to a native Python workflow, converting and quantizing models into its own format ahead of time. Neither is strictly better: llama.cpp is the one to reach for if you want a single compiled binary with no Python dependency, MLX is the one to reach for if you are already building the rest of your pipeline in Python and want first-class access to Apple’s unified memory model from there.

Quantization: what you actually trade for a smaller model

Quantization stores each model weight in fewer bits than the 16 or 32 the model was trained in, shrinking both the file on disk and the memory needed to run it, at some cost to output quality. llama.cpp’s own quantize documentation lists exact bits-per-weight figures for each scheme it supports, which is a more precise way to reason about the trade-off than the usual shorthand of "4-bit" or "8-bit":

Multiplying a model’s parameter count by its bits-per-weight figure gives a rough memory estimate for the weights alone: a 7-billion-parameter model at Q4_K_M’s 4.89 bits per weight needs approximately 7,000,000,000 × 4.89 ÷ 8 bytes, or about 4.3 GB, before accounting for the context window and intermediate activations, which add more on top depending on how much text you feed it. That is a calculation from the cited bits-per-weight figure, not a number either project publishes directly, and actual memory use will run higher once a real prompt and its context are loaded. The practical guidance that follows from the table is simple: for a task like classifying one connection log line at a time, where the input is short and the required output is a small structured verdict, a Q4_K_M-class model is very often the right trade — noticeably smaller and faster than Q8_0 or F16, without dropping to the extreme-compression tier where output quality degrades more sharply.

Neither project publishes a memory requirement per Mac configuration, so the practical approach is to work backward from the estimate above and leave real headroom: macOS itself, your browser and whatever else is running all need unified memory too, and a model that just barely fits with nothing else open will swap or stall the moment you switch to another app. On a Mac with a modest amount of unified memory, that argues for staying toward the smaller end of the table above — a few billion parameters at Q4_K_M rather than a much larger model at the same quantization — and reserving the larger, higher-precision options for machines with memory to spare. For a narrow classification task like the one this article is about, a smaller model asked a precise question tends to be both faster and, in practice, more consistent than a larger model given a vague one.

Both projects are under active development, and the honest comparison is less "which is better" than "which fits your pipeline." llama.cpp compiles to a single binary with no Python runtime required at inference time, which matters if you want to embed it inside another piece of software without shipping a Python interpreter alongside it. MLX assumes you are already working in Python (or Swift, for which it also ships bindings) and rewards that with tighter integration into Apple’s own machine-learning stack and the unified-memory model described above. A security tool built as a standalone macOS application, which is the situation FireAI itself is in, sits closer to the llama.cpp end of that spectrum; a research notebook exploring which prompt pattern works best sits closer to the MLX end.

A prompt pattern for triaging one connection

The narrower the question you ask a small local model, the more reliably it answers. For a single connection, that means giving it exactly the fields a human reviewer would look at — nothing more, nothing inferred — and asking for a structured verdict rather than free-form prose:

System: You review one outbound network connection at a time. You are
given only the fields listed below. Do not assume anything not stated.
Respond with exactly two lines: a verdict (allow, ask, or block) and a
one-sentence reason a non-expert could understand.

Connection:
  app: UpdaterHelper.app
  code_signature: unsigned
  destination: 91.203.xxx.xxx:4444
  protocol: TCP
  threat_feed_hit: none
  first_seen: yes (no prior rule for this app)

Verdict:

The fields that matter are the ones a code-signing check and a threat feed can actually produce without guessing: whether the binary is signed and by whom, where it is trying to connect and on what port, whether that destination shows up on a threat feed, and whether this is the first time this app has tried to connect at all. Asking for a fixed two-line output, rather than an open-ended explanation, makes the result easier to log, easier to compare across thousands of connections, and much harder for the model to pad with hedging language that sounds authoritative without saying anything checkable.

A small model will occasionally ignore the requested format anyway — three lines instead of two, an extra caveat, a verdict word that is not one of the three you asked for. Treat that as an engineering problem, not a modelling one: validate the output against the exact shape you expect, and if it does not match, either re-ask or fall back to the safest verdict (ask, meaning show a human) rather than trying to parse a looser answer. A triage step that fails safely on malformed output is far more useful than one that occasionally produces a confident-looking but unparseable line and silently drops it.

Run this pattern across a full day of connection attempts rather than one at a time and the shape of the workload changes: most connections are from apps with an existing rule and never reach the model at all, a smaller number are genuinely first-seen and get a verdict, and only a fraction of those verdicts are anything other than a routine allow. The model’s real job, at that point, is not to be a security expert; it is to cut a long list of first-seen connections down to the small subset a person actually needs to look at, which is a much more achievable target for a few-billion-parameter model than open-ended security judgement would be.

Where this goes wrong

• A small local model can produce a confident, well-written, entirely wrong reason. Nothing about running locally changes that risk; it only changes where the mistake happens, not whether it can happen.
• Context windows are finite, and a long, noisy log does not fit. Summarising or pre-filtering before the model sees the data introduces its own failure point — you can lose the one line that mattered before the model ever gets a chance to look at it.
• The model only ever sees what the log line contains. It cannot see inside encrypted traffic, it cannot verify a threat feed is current, and it cannot know your intent — a connection from a tool you just installed on purpose looks identical to one from a tool you have never heard of.
• A verdict is not an action. Nothing here should block, delete or silently allow a connection on its own; a person still needs to see the reason and confirm it, and be able to reverse the call if the model got it wrong.

That last point is not a limitation specific to a small quantized model running on a laptop — it is true of every automated security decision, local or cloud, small model or large. The value of running it locally is what it removes from the equation (a network dependency, a recurring bill, a third party receiving your connection metadata), not a claim that it removes the need for a human in the loop.

Where FireAI fits this pattern

FireAI, HisnLabs’ firewall for Mac, ships something built on the same idea, at a narrower scope than a general-purpose chat model: an optional local model, an additional download of roughly 1.5 GB, that runs entirely on the Mac and reviews connections from apps that have no rule yet. It requires macOS 14 or later on Apple silicon, applies public threat feeds locally rather than checking them against a remote service, and shows the reason for its decision in the same permission prompt it uses to ask you to allow or deny the connection — every one of those decisions becomes a visible, editable rule, and any of them can be undone. It is worth being precise about what that is and is not: it is not the open-ended, few-billion-parameter chat model this article has been describing, and it is not an antivirus or a VPN — it does one narrow classification job, locally, and leaves the final call to whoever reads the prompt.

How FireAI and HisnLabs fit in

FireAI’s own connection reviewer is this exact bet, made narrower still: an optional, roughly 1.5 GB on-device model, macOS 14 and up, Apple silicon only, reviewing one thing (should this unknown app reach this destination) with a visible reason and an undo button — and, like the triage pattern in this article, it still needs a person to confirm anything consequential.

FireAI is HisnLabs’ own product: an on-device AI firewall for Mac. It shows every connection your apps make, in plain language, and lets you decide what leaves your Mac — its AI runs locally, so your traffic is never sent to us or anyone else. HisnLabs’ security research team is the group that keeps that decision-making accurate: cataloguing which domains are ordinary telemetry versus a real product, tracking the country and network behind a connection, and training the on-device model (its Autopilot feature) on real traffic patterns, all without any of it leaving your Mac.

You can read the technical decisions behind it, or try FireAI for 17 days, at FireAI, by HisnLabs.