Research · Arabic, side by side
Arabic guardrail comparison: Guardrino vs English prompt guards on Arabic, dialects and Arabizi
We ran Guardrino and other prompt-attack classifiers on the same messages: well-known guards built for English, and guards other teams built for Arabic. This page gives every count, how the test was run, what it cannot tell you, and the data file behind it.
Guardrino is in development and offered through watch-only pilots. These are the results of one model version on our own test set, not a promise about your traffic: a pilot measures the same things on your own messages.
The results
Each cell gives the count and the share it makes, computed from the counts when the page is shown.
Арабська: порівняння
Створено для арабської. У наших власних тестах Guardrino перехопив майже 9 з 10 атак арабською, а найвідоміші англійські засоби захисту — близько 1 з 7.
| Модель | Перехоплено атак арабською (з 7 874) | Перехоплено атак арабізі (з 1 523) | Заблоковано безпечних повідомлень (з 31 348) |
|---|---|---|---|
| Guardrino-t7 v7 | 6 814 з 7 874 (86,5%) | 1 319 з 1 523 (87%) | 804 з 31 348 (2,6%) |
| Guardrino-t7 v7, налаштований на частку хибних блокувань ArabGuard Та сама модель із суворішим порогом, щоб вона блокувала так само мало безпечних повідомлень, як ArabGuard. | 6 520 з 7 874 (82,8%) | 1 214 з 1 523 (80%) | 519 з 31 348 (1,7%) |
| Ara-Prompt-Guard V1 (NAMAA, Arabic) Створено для арабської іншою командою, оцінено тим самим способом. | 2 033 з 7 874 (25,8%) | 99 з 1 523 (7%) | 615 з 31 348 (2,0%) |
| ArabGuard (Egyptian) Створено для арабської іншою командою, оцінено тим самим способом. | 1 553 з 7 874 (19,7%) | 2 з 1 523 (0%) | 519 з 31 348 (1,7%) |
| HARIS Створено для арабської іншою командою, оцінено тим самим способом. | 561 з 7 874 (7,1%) | 0 з 1 523 (0%) | 459 з 31 348 (1,5%) |
| Prompt Guard 2 (Meta) | 1 128 з 7 874 (14,3%) | 73 з 1 523 (5%) | 882 з 31 348 (2,8%) |
| Protect AI v2 | 894 з 7 874 (11,4%) | 95 з 1 523 (6%) | 589 з 31 348 (1,9%) |
Метод: власний тестовий набір HisnLabs з 13 249 атак і 31 348 безпечних повідомлень, переважно арабською. Це дає перевагу моделі, навченій на схожих даних, як наша, тож це не незалежний тест. Поріг кожної моделі налаштовано так, щоб на валідаційній вибірці вона блокувала 1 з 100 безпечних повідомлень. Дати тесту: 26–27 жовтня 2026 р.. Зовнішні моделі створені для англійської, для одного діалекту або переважно для сучасної стандартної арабської; це не критика на їхню адресу. Одне відкрите джерело в тестовому наборі (arabic-guardrail, 2 271 атак) може входити до навчальних даних інших моделей, а його навчальна частина була в наших; без нього Guardrino-t7 перехоплює 4 596 з 5 603 атак арабською (82%).
How we tested
The test set is HisnLabs’ own (“cells” in the data file): 13,249 attacks and 31,348 safe messages, mostly Arabic, with Arabizi and some English. Every model saw the same messages. Tested 26–27 October 2026.
Arabic attacks caught and Arabizi attacks caught: how many attacks written in Arabic script, and in Arabizi (Arabic in Latin letters and digits), the model flagged. Higher is better. Safe messages blocked: how many harmless messages it flagged by mistake. Lower is better. Read the two together: a guard can catch more attacks simply by blocking more messages.
Thresholds: a classifier gives each message a score, and a threshold turns that score into block or allow. Comparing models at their default thresholds would mostly compare how strict each one is. So each model’s threshold was set on a separate validation sample to block about one in 100 safe messages, then used unchanged on the test set. Each row’s threshold is in the data file.
Guardrino by dialect
Most Arabic attacks in the set carry a dialect label. Our row, split by dialect (the other models’ dialect counts are in the data file):
| Dialect | Arabic attacks in the set | Caught by Guardrino |
|---|---|---|
| Gulf | 1,918 | 1,642 of 1,918 (85.6%) |
| Egyptian | 1,834 | 1,573 of 1,834 (85.8%) |
| Levantine | 559 | 540 of 559 (96.6%) |
| Maghrebi | 654 | 624 of 654 (95.4%) |
| Modern Standard Arabic | 2,526 | 2,102 of 2,526 (83.2%) |
383 Arabic attacks have no dialect label: they count in the Arabic column only.
Why there are two Guardrino rows
The second row is Guardrino-t7 v7 at a stricter threshold (0.9818), chosen so that it blocks no more safe messages than ArabGuard: 519 of 31,348, against 519 for ArabGuard. Read it against ArabGuard. With the false blocks level, the difference in attacks caught is not bought by blocking more safe messages.
A public source that may overlap
One public dataset in the test set, arabic-guardrail, gives 2,271 of its attacks. It also gives 2,216 of the safe messages. Other models may have been trained on it, and its training split was in our own training data, so it can flatter any of them. Without it, Guardrino catches 4,596 of 5,603 Arabic attacks (82%).
The table without that source
The same rows with arabic-guardrail taken out of the test set, both its attacks and its safe messages. Only rows whose data includes this split are shown.
| Model | Arabic attacks caught (of 5,603) | Arabizi attacks caught (of 1,037) | Safe messages blocked (of 29,132) |
|---|---|---|---|
| Guardrino-t7 v7 | 4,596 of 5,603 (82.0%) | 844 of 1,037 (81%) | 744 of 29,132 (2.6%) |
| Ara-Prompt-Guard V1 (NAMAA, Arabic) | 1,568 of 5,603 (28.0%) | 97 of 1,037 (9%) | 612 of 29,132 (2.1%) |
| ArabGuard (Egyptian) | 1,208 of 5,603 (21.6%) | 2 of 1,037 (0%) | 516 of 29,132 (1.8%) |
| HARIS | 339 of 5,603 (6.1%) | 0 of 1,037 (0%) | 429 of 29,132 (1.5%) |
What this comparison cannot tell you
- It is our own test, not an independent one. The test set is mostly Arabic and close to the data Guardrino was trained on, which favours our model. We publish the counts so that others can check them, and we would welcome an independent test.
- One public source may overlap with training data, ours and other teams’ (see above). That is why the table without it is shown.
- Every number depends on the threshold rule. With a different false-block budget every row changes, and the order of close rows can change too.
- Some dialect sets are small: 559 Levantine and 654 Maghrebi attacks. A few dozen messages move those rows noticeably, so read them as a rough indication.
- The outside models were built for English, for one dialect or mainly for Modern Standard Arabic. The table shows what happens when they meet text they were not built for. It is not a judgement of those models.
- It measures one version, Guardrino-t7 v7, tested on 27 October 2026. Guardrino is in development: later versions are measured again, and your own traffic may differ. A watch-only pilot measures it on your messages.
The data
The file behind this page holds counts only, never percentages: for each model, the attacks it caught and the safe messages it blocked in each column, its threshold, the counts by dialect and the counts without the overlapping source. The page computes every share from it when shown.
Main fields: rows (one per model, each count as [caught, out of]; rows marked hidden stay in the file but are not shown here), columns, max_fpr (the share of safe messages each threshold was set to block), date_from and date, and overlap.