Every guardrail vendor will tell you their product works. The good news is that you do not have to take anyone’s word for it. Researchers have published open collections of test messages that anyone can run, and a careful afternoon with them tells you more than any brochure. This tutorial shows the method we use at HisnLabs, so you can apply it to any guardrail: one you are building, one you are buying, or two you are comparing.
It is written for engineers, but the rules in the middle are for everyone. A manager who knows them can read any vendor’s results and see at once whether they mean something.
The one idea behind every step
For testing purposes, a guardrail is a function: text goes in, a score between 0 and 1 comes out, meaning “how likely is this an attack?”. A threshold turns the score into a decision. Each public test set is a list of texts with known answers. You run the function on every text and count three things.
- Caught: attacks that scored at or above the threshold, out of all attacks.
- Wrongly blocked: safe texts that scored at or above the threshold, out of all safe texts.
- Time added: how long one call takes, for the typical message and for the slow tail, on stated hardware.
That is the whole measurement. Everything else in this tutorial is about doing it honestly.
Four rules that make the numbers mean something
1. Never test on what the guard was trained on
Many public test sets are built from the same jailbreak collections that guard models learn from. If a set overlaps with the training data, the guard has seen the answers, and its score says nothing about new attacks. Check every test set against the training sources, and leave out any set that overlaps. If you are comparing a vendor’s model, read its model card for the data it was trained on, and say so next to its number.
2. Choose the threshold before you look at the tests
Pick the threshold on a separate validation set, according to a rule you write down in advance, for example “the highest threshold that blocks no more than a fixed small share of safe validation messages”. Then run the tests once. Moving the threshold until the test score looks good inflates the result and will not hold on real traffic.
3. Always report both sides
A guard that blocks everything catches every attack. That is why a “caught” figure on its own is meaningless. Every caught figure needs a wrongly-blocked figure next to it, measured on a set that contains safe texts. Meta’s CyberSecEval 2 authors make the same point and propose a false refusal rate for it [5]. Our article on guardrails that don’t annoy your users explains why the second number is often the one that decides whether a guard stays switched on.
4. Say what each set measures, and report it separately
Test sets measure different things. Some contain harmful requests, such as asking for dangerous instructions, which is a content-safety question. Others contain manipulation, attempts to override the assistant’s instructions, which OWASP calls prompt injection [7]. A guard built for one will look weak on the other by design. Report per set and per language group, never one blended number.
Report every rate with its n and a confidence range
“96 in 100 caught” on 55 attacks and on 5,000 attacks are different claims. With few messages, chance alone moves the result a lot. The honest way to show this is a confidence range. For rates like these, the Wilson score interval is a good default: it “can be safely employed with small samples and skewed observations”, and it does not collapse to a zero-width range when a rate is near 0 or 1, as the simpler normal approximation does [6]. A few lines of Python are enough:
from math import sqrt
def wilson(k, n, z=1.96):
"""95 in 100 confidence range for k successes out of n."""
p = k / n
d = 1 + z * z / n
centre = (p + z * z / (2 * n)) / d
half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
return round(centre - half, 3), round(centre + half, 3)
print(wilson(53, 55)) # (0.877, 0.99)
print(wilson(4800, 5000)) # (0.954, 0.965)
print(wilson(0, 250)) # (0.0, 0.015)These numbers are for a hypothetical guard, to show the effect. The same observed rate, about 96 in 100, gives a range from about 88 to 99 in 100 on 55 attacks, and from about 95 to 97 on 5,000. The last line matters for the wrongly-blocked side: a guard that blocked none of 250 safe messages may still block up to about 1.5 in 100 in real use. A vendor who shows “0 wrongly blocked” without the n is not telling you much.
Step 1: GuardBench, for a broad first look
GuardBench, from Bassani and Sanchez, was presented at EMNLP 2024. It gathers 40 safety evaluation datasets behind one Python library, and its paper also shared new prompt moderation datasets in German, French, Italian and Spanish [1]. The library is open source under the EUPL-1.2 licence; each dataset keeps its own licence, and some need a sign-up or a Hugging Face login [2]. Install it in its own environment:
# a separate environment for the test tools
python -m venv .venv-guardbench
.venv-guardbench/bin/pip install guardbenchYou give the library one function, which receives a batch of conversations and returns one probability per conversation that it is unsafe. Here it calls a guard that exposes a small scoring endpoint; replace the URL with your own guard or a vendor’s API.
from guardbench import benchmark
import requests
GUARD_URL = "http://127.0.0.1:8810/classify" # your guardrail's scoring endpoint
def moderate(conversations, **kwargs):
# conversations: a list of chats, each a list of {"role": ..., "content": ...}
# return one float per chat: the probability that it is unsafe
return [requests.post(GUARD_URL, json={"text": c[-1]["content"]}).json()["score"]
for c in conversations]
benchmark(
moderate=moderate,
model_name="my-guard 0.1.0",
batch_size=16,
datasets=["xstest", "jbb_behaviors"], # choose the sets that match what the guard is for
metrics=["f1", "recall"],
)Two choices matter here. First, pick the sets that match the guard’s job, not every set: a prompt-injection guard and a content-safety guard should be scored on different sets. Second, include XSTest. Its 250 safe prompts that sound dangerous, next to 200 unsafe contrasts, make it one of the best public measures of wrongly blocked messages [3]. GuardBench computes its metrics at a fixed threshold of 0.5; compute your own pre-chosen threshold from the saved scores too, and say which one each number uses.
Step 2: CyberSecEval prompt injection, for manipulation
Meta’s Purple Llama project publishes CyberSecEval under the MIT licence. Its prompt-injection tests assess susceptibility to attacks in which “a portion of the LLM prompt coming from untrusted user input contains malicious instructions intended to override the LLM's original task” [4]. There is an English file and a machine-translated multilingual file. Every row is an attack, so this set measures “caught” only; pair it with a set of safe texts, such as XSTest or your own.
There are two ways to use it. The quick way is to score each attack directly with your guard, as in Step 1, and count how many cross your threshold, per language. The full way uses Meta’s own runner: it sends each system prompt and attack to a chatbot, and a second model acting as judge decides whether the chatbot was tricked.
# get the tests
git clone --depth 1 https://github.com/meta-llama/PurpleLlama
cd PurpleLlama
# run the prompt-injection tests against a chatbot, with a judge model
python3 -m CybersecurityBenchmarks.benchmark.run --benchmark=prompt-injection --prompt-path="$DATASETS/prompt_injection/prompt_injection.json" --response-path="$DATASETS/prompt_injection/prompt_injection_responses.json" --judge-response-path="$DATASETS/prompt_injection/prompt_injection_judge_responses.json" --stat-path="$DATASETS/prompt_injection/prompt_injection_stat.json" --judge-llm=<JUDGE> --llm-under-test=<MODEL>Run it twice: once with the chatbot alone, and once with the same chatbot behind the guardrail, counting a blocked message as “not tricked”. The difference between the two runs is what the guardrail adds, which is the question a buyer actually cares about. Name the judge model in your report: its own mistakes are part of the result.
Step 3: test in your users’ languages
Public sets are mostly English. If your users write in Arabic, build a held-out set of your own: real-style safe messages and attacks in Modern Standard Arabic, Gulf and Egyptian dialects, Arabizi and mixed Arabic and English, each also in a fully vowelled form and a randomly vowelled form. Never train on it. Report each group separately: a guard that does well in English and poorly in Arabizi is a different product for a bank in Dubai than its average suggests.
What a trustworthy report contains
- For each set: its name, a link, its licence, and whether it is public or your own held-out split.
- Per language group: the caught rate and the wrongly-blocked rate, each with n and a 95 in 100 confidence range.
- The model name and version, the date, and the rule used to choose the threshold.
- The time added per message, typical and slow tail, with the hardware and message length.
- Any red-team runs, with how many attacks got through, and a link or script to reproduce everything.
When you compare two guardrails, use the same sets, the same day, the same hardware and the same threshold rule for both, or state that both used 0.5.
How we apply this to Guardrino
Guardrino, the AI firewall HisnLabs is building for businesses, is in development, and its product page has a measured-results section that stays empty on purpose. It appears only once a release has been tested by exactly this method, with every number carrying its n and its range. Until then we describe what Guardrino is designed to do, and nothing more. For the design itself, see our guide to AI guardrails; for the reasons behind the project, read why we are building Guardrino for the Middle East.
How Guardrino and HisnLabs fit in
Guardrino’s release reports are planned to follow every rule in this tutorial, and its product page shows no results until the first one exists. Until then, the simulation below shows only what Guardrino is designed to do, not how well it does it.