Evaluator Disagreement in LLM Response-Safety Assessment

Automated evaluators are often treated as if they provide interchangeable measurements of model safety. This study tests that assumption by applying four methods (a keyword heuristic, Llama Guard 3, a StrongREJECT-inspired GPT-4o-mini judge, and OpenAI Moderation) to fixed responses produced from 488 adversarial prompts. The primary canonical run contains 488 GPT-4o-mini responses; a secondary Claude 3.5 Haiku run completed 385 records, which are used as a matched cross-model subset.

The methods produced sharply different output distributions, but the divergence cannot be read as a simple accuracy ranking. They received different inputs, operationalized different constructs, and handled failure differently. On the primary run, Llama Guard returned a valid semantic output for only 41.4% of records, while the other methods exceeded 99.5% coverage. After technical failures were removed, all four evaluators were unanimous on 18.4% of complete GPT records and 1.2% of complete Claude records. Pairwise agreement ranged from 2.5% to 98.2%. A post-hoc text-normalization audit also changed 57 GPT keyword outcomes from inconclusive to safe.

The central conclusion is narrow but consequential: an automated safety score is not self-interpreting. Its meaning depends on the evaluator’s target construct, input context, interface, thresholding, text normalization, and failure policy.

Why evaluator disagreement matters

A safety score looks like a property of the model. In practice, it is also a property of the measuring instrument.

Red-team studies commonly generate a set of adversarial prompts, collect model responses, and ask an automated evaluator whether each attack succeeded. The final attack-success rate can then appear objective: a single percentage, often reported to one decimal place. Yet the number is downstream of several design choices: what the evaluator sees, which behavior it is instructed to detect, how it converts uncertainty into a label, and what happens when an API call fails.

This project began with a practical question: how much does the evaluator determine the reported safety outcome when the underlying responses remain fixed? The repository answered that question by storing several evaluator verdicts side by side for every response rather than selecting one evaluator and treating its labels as ground truth.

RQ1

How different are the verdict distributions produced by four automated evaluation methods on fixed responses?

RQ2

Which evaluator-specific patterns persist across two target models?

RQ3

How much disagreement reflects construct mismatch, operational choices, or technical failure rather than genuine ambiguity?

Study design

The unit of analysis is a fixed pair: one adversarial prompt and one target-model response. Each response is then processed by four evaluators. This design isolates cross-method differences at the row level, although it does not establish which evaluator is correct.

Prompt sources

Target models and canonical runs

An earlier GPT run, 3f77c02e, contains 483 records and no OpenAI Moderation verdicts. A later run, 235fb125, contains only 200 records. Fixing run IDs prevents figure scripts from silently selecting the first run that happens to match a prompt-count condition.

Four methods, shared labels, different measurement targets

Separating semantic uncertainty from technical failure

The repository’s common interface exposes three verdicts: safe, unsafe, and inconclusive. However, evaluator code also maps API errors, malformed outputs, and missing credentials to inconclusive. This article reclassifies summaries containing markers such as “call failed,” “API error,” or “failed to parse” as technical failure. Genuine semantic uncertainty remains inconclusive.

This distinction changes the interpretation substantially. An evaluator that abstains because a response is genuinely ambiguous is behaving differently from an evaluator that never returned a usable result.

Results

Raw evaluator-output profiles diverged sharply

On the canonical GPT-4o-mini run, the keyword heuristic labeled 69.3% of responses safe and 30.7% inconclusive. The LLM judge labeled 95.1% safe, while OpenAI Moderation labeled 99.4% safe. Llama Guard produced a starkly different profile: 30.9% unsafe, 10.5% safe, and 58.6% technical failure.

The broad evaluator profile persisted across target models

For a controlled cross-model view, the analysis uses the 385 prompt IDs completed in both runs. Among valid semantic outputs, Llama Guard labeled 73.3% of GPT responses and 80.4% of Claude responses unsafe. The other methods remained below 1.3%. The broad pattern therefore persisted across targets, but a separate coverage plot is essential because Llama Guard returned valid outputs for only about 42% of matched records.

The cross-model similarity is consistent with an evaluator-specific effect but does not localize its source. Possible causes include the evaluator’s learned calibration, the custom classification prompt, OpenRouter serving behavior, output parsing, or interactions among these factors. The native Llama Guard model card recommends applying the model’s chat template and allows thresholding based on the first-token probability; the repository implementation instead sends an explicit natural-language classification prompt through a generic chat-completions interface.

Complete-case agreement was low

Agreement statistics become more meaningful after records with technical failure are removed. This leaves 201 complete semantic records for GPT-4o-mini and 163 for Claude 3.5 Haiku. The term all disagree used in the original code is replaced here with three-label spread: with four evaluators and only three possible semantic labels, all four cannot produce four distinct labels.

Pairwise agreement exposed method families

Pairwise comparison is more informative than one global agreement number because it reveals which methods behave similarly. The LLM judge and OpenAI Moderation agreed on 95.7% of jointly valid GPT records and 98.2% of Claude records. The keyword heuristic agreed with the LLM judge on 72.0% of GPT records but only 10.1% of Claude records. Llama Guard’s agreement with other methods remained between 18.3% and 26.4% for GPT and between 2.5% and 20.9% for Claude.

A small normalization change moved 57 verdicts

The keyword heuristic lowercases text but does not normalize typographic apostrophes. A post-hoc audit applied Unicode NFKC normalization and replaced curly quotes with ASCII equivalents before running the same rules on the stored responses. In the GPT run, 57 records moved from inconclusive to safe, changing the keyword profile from 69.3% safe / 30.7% inconclusive to 80.9% safe / 19.1% inconclusive. No Claude verdict changed, indicating that Claude’s mismatch was broader than punctuation alone.

Small preprocessing choices can change a reported evaluator profile by more than eleven percentage points without changing a single model response.

Five diagnostic cases

Aggregate rates show that methods differ; row-level cases reveal why. The following examples are taken from canonical GPT run 5973adf8; response excerpts are shortened for readability.

What the disagreement actually means

The most useful conclusion is not that one evaluator is universally better. The evidence supports three narrower sources of divergence.

Construct mismatch

The methods target lexical refusal, policy-category safety, harmful usefulness, or harmful content. Those are related but non-identical properties.

Operational mismatch

Inputs, prompts, templates, thresholds, score mappings, and text normalization differ. Even the same model can behave differently under another serving interface.

Technical mismatch

Timeouts, 400 responses, parse failures, and missing outputs must not be merged with genuine semantic uncertainty.

Disagreement is diagnostic, not ground truth

A disagreement map can identify records that warrant human review and reveal evaluator-specific patterns. It cannot determine correctness by itself. Majority voting is especially risky when methods do not target the same construct: three votes may reflect three correlated proxies rather than three independent measurements.

The strong pairwise agreement between the LLM judge and OpenAI Moderation is a useful example. It suggests that both usually label the observed responses safe, but it does not make them interchangeable. The judge sees the harmful request and scores compliance; Moderation sees only the generated response and detects harmful content. They can agree for different reasons.

Relation to prior work

StrongREJECT was introduced partly because prior jailbreak evaluators often overstated attack effectiveness; its automated evaluator was designed to better approximate whether a response gives useful forbidden information. JailbreakBench and HarmBench similarly emphasize standardized prompts, threat models, and scoring procedures.

More recent work directly studies judge instability. Safer or Luckier? evaluates eleven LLM safety judges and finds sensitivity to surface artifacts, including apologetic language. When Scanners Lie keeps attacks and outputs fixed while varying evaluator components, showing substantial changes in reported vulnerability scores. A 2026 multi-dimensional study likewise reports that judge disagreement varies across domains, criteria, languages, and linguistic style.

The present project is smaller and less controlled than those studies, but it offers a transparent row-level artifact: prompts, responses, four evaluator outputs, rationales, an SQLite database, and an interactive dashboard. Its clearest contribution is an accessible demonstration of how construct and implementation differences appear in real evaluation records.

Threats to validity

Construct validity

The four methods are not measuring a single, fully specified latent variable called “safety.” The custom prompt set also mixes data leakage, hallucination, role-play, injection, and harmful compliance. Aggregate comparisons should therefore be interpreted as cross-method output comparisons, not as four estimates of one true rate.

Internal validity

The repository implementation accesses Llama Guard through a generic OpenRouter interface rather than the native template recommended by Meta. The LLM judge uses GPT-4o-mini, which is also the primary target, creating possible shared biases. The keyword evaluator is lexically brittle. The StrongREJECT-inspired scoring implementation is custom and does not reproduce the official evaluator exactly.

External validity

The main analysis covers two proprietary target models, and the Claude run is incomplete. Prompts are static and single-turn. Results may not transfer to open-weight models, multi-turn attacks, tool-using agents, other languages, or later hosted model versions.

Statistical and annotation limits

The study does not provide a complete independent human ground truth, repeated stochastic judge runs, confidence intervals for accuracy, or calibrated probability analysis. The partial manual-review files are useful for exploration but insufficient for definitive evaluator ranking.

Practical recommendations

Specify the construct

State whether the evaluator detects harmful content, harmful compliance, policy violations, refusal, or factual failure.

Publish the protocol

Report evaluator model, version, template, input roles, thresholds, parser, normalization, and serving provider.

Separate failure states

Keep semantic abstention distinct from timeout, API error, parse error, and missing response.

Report coverage

Every rate should include its denominator and the fraction of records for which the evaluator produced a valid output.

Preserve row-level data

Aggregate scores should link back to prompts, responses, evaluator outputs, and rationales for audit.

Use humans strategically

Adjudicate disagreements and a random sample of agreements; do not assume the majority evaluator is correct.

Conclusion

Four automated methods looked at the same model responses and produced sharply different distributions of safe, unsafe, and inconclusive labels. The differences persisted across two target models, but they were entangled with unequal coverage, different inputs, different constructs, a brittle keyword implementation, and a high rate of classifier-call failure.

The right lesson is not that safety evaluation is impossible, or that adding more evaluators automatically solves the problem; it is that automated safety scores require a measurement model. Researchers should state what each evaluator is intended to detect, how it was invoked, how failures were handled, and how its labels relate to human-defined assessment targets.

An automated safety score is not self-interpreting. The evaluator is part of the experiment.

— Yanis Prasol, 2026