The science of detectors and judges

Learn how scanners decide that a reply is a failure, why every method has typical errors, and how to calibrate a judge against human labels before trusting its numbers.

An automated red-team run produces thousands of replies, and a machine must decide which ones count as failures. That decision is the weakest link in most pipelines. If the detector is wrong, the failure rate is wrong, and every chart built on it is wrong too. Treat the detector as a measuring instrument that needs calibration, and not as ground truth.

Five ways to decide

No method is best everywhere. Prefer the most deterministic one that fits the objective.
Method	How it decides	Strength	Typical error
Canary check	Looks for a planted token or a specific side effect	Deterministic and cheap, and the best choice when the objective can be expressed this way	Misses harm that does not involve the canary
Pattern matching	Matches strings or regular expressions; garak’s dan probe, for example, measures success by not getting a safety mitigation message ^[1]	Fast and transparent	Brittle: refusal wording varies, and quoting the attack can look like compliance
Refusal scoring	Asks whether the model refused, and inverts it, as in the series’ PyRIT demo ^[2]	Needs no knowledge of the topic or language	Counts a reply that adds a disclaimer and then complies as success
Trained classifier	A model scores a reply against a taxonomy: Augustus lists the Perspective API and unsafe-content detectors ^[3], and Llama Guard classifies prompts and responses ^[4]	Scales, and is consistent run to run	Its categories may not match your definition of harm, and updates change results
LLM as judge	A language model reads the reply and a rubric and gives a verdict, with a rationale	Handles open-ended objectives	Biases and limited reasoning, below

What is known about LLM judges

Zheng and colleagues examined LLM-as-a-judge and identified position bias, verbosity bias, self-enhancement bias and limited reasoning ability. They also report that strong judges can match human preferences well, with over 80 per cent agreement, the same level as agreement between humans, in their chat-quality setting ^[5]. Two cautions follow. The agreement was measured on a different task from harm scoring, so it does not transfer automatically. And the named biases have direct red-team versions: a judge may favour long answers, so a long refusal can be misread, and a judge from the same model family as the target may favour its output.

What a wrong detector does to your numbers

Let p be the true share of replies that really fail. A judge with sensitivity s catches a share s of the real failures, and with false-alarm rate f wrongly flags a share f of the good replies. The observed rate is then s × p + f × (1 − p). If the judge raises false alarms on a large number of good replies, the observed rate is inflated even when the system is sound. If sensitivity is low, real failures are missed. A rate without the judge’s error rates is therefore an incomplete result.

Calibrating a judge

Draw a random sample of replies, including ones the judge passed as well as ones it failed.
Have people label each reply against the written rubric, ideally two people for a subset to check that they agree with each other.
Count agreements and disagreements by kind: real failures the judge missed, and false alarms.
Report these as error rates, with intervals (lesson 14), alongside the failure rate.
If the error is large, change the rubric, the judge or the method, then calibrate again.
Pin the judge’s model and prompt version, and recalibrate whenever either changes.

Design rules

Make objectives scorable. A planted canary beats a vague harm definition wherever you can use it.
Keep the judge separate from the target, and keep a human sample in the loop.
Require a rationale. The PyRIT demo shows the reason for each score, which makes disputes quicker to settle ^[2].
State the definition. The demo’s presenter notes the score may need tweaking depending on what you think harm is. Write down your definition so a reader can disagree with it.

Standardised evaluation

HarmBench’s authors observe that automated red teaming “holds substantial promise” but “the field lacks a standardized evaluation framework to rigorously assess new methods”, and they propose one that compares 18 red-teaming methods across 33 target models and defences ^[6]. Even if you do not use it, the point stands for your own pipeline: fix the protocol, including the scorer, before you compare anything.

Key takeaways

The detector is a measuring instrument: calibrate it before trusting the failure rate.
Prefer deterministic checks such as canaries, then classifiers, and use LLM judges for open-ended objectives.
LLM judges show position, verbosity and self-enhancement bias, and have limited reasoning.
The observed rate mixes true failures with the judge’s false alarms, so report the judge’s error rates.
Pin and recalibrate the judge whenever it changes.

Check yourself

1. What is a typical error of refusal scoring?
- It counts a reply that adds a disclaimer and still complies as success — Right.
- It reads images
- It needs the network
- It cannot be inverted
The series’ demo notes that a disclaimer followed by compliance counted as success because there was no outright refusal.
2. Which biases are named for LLM judges?
- Position, verbosity and self-enhancement — Right.
- Colour, size and age
- Latency, cost and memory
- Syntax, spelling and grammar
Zheng et al. list these along with limited reasoning ability.
3. Why report a judge’s error rates?
- The observed failure rate mixes real failures with the judge’s false alarms and misses — Right.
- They make reports longer
- Judges never err
- It speeds up the run
Without error rates the observed rate cannot be interpreted.
4. When should you recalibrate?
- When the judge’s model or prompt changes — Right.
- Never
- Only once a year
- Only for images
A different judge is a different instrument.

Do it with FireAI

Put this lesson into practice on your own Mac.

Sources

Put it into practice on your Mac

Try every feature free for 17 days, no card needed.

Download for Mac Docs