The Benchmark Mean Is Not the Reliability Question

On repeated agent evaluations, offensive-security benchmarks, and the governance significance of variance.

On May 28, 2026, Galip Tolga Erdem posted the arXiv preprint "How Reliable Are AI Attackers Against a Fixed Vulnerable Target?", a 400-run empirical study of LLM penetration-testing consistency. The experiment held the prompt, orchestrator, and vulnerable target constant while running four models 100 times each against a honeypot containing OWASP Juice Shop and two additional vulnerable services, according to the arXiv abstract for the study.

The surface result is easy to turn into a ranking. Gemini 2.5 Flash-Lite fully exploited all three services in 85 of 100 runs, Claude Sonnet 4 in 61 of 100, GPT-4o-mini in 56 of 100, and qwen2.5-coder:14b in 25 of 100, according to the arXiv abstract for the study. That ranking matters. It is not the most important governance fact.

The more important fact is that the same model, operating against the same target, under the same evaluation scaffold, did not produce a single stable behavioral object. Erdem reports substantial variation across exploitation outcomes, strategy selection, timing, credential reuse, and failure modes, according to the arXiv HTML version of the paper. A benchmark mean collapses that distribution into a point estimate. Operators do not live inside the point estimate.

Identical Conditions, Non-Identical Behavior

The study's setup is unusually direct. The target environment was an Azure virtual machine running Ubuntu 22.04 with three deliberately vulnerable services: OWASP Juice Shop on port 3000, OpenSSH with weak credentials, and vsftpd with anonymous access to confidential files, according to the methodology section of the arXiv HTML paper. The orchestrator used a command-execute-observe loop, capped runs at 25 iterations, and applied the same authorized penetration-testing framing across the evaluated models, according to the methodology section of the arXiv HTML paper.

That controlled design makes the variance harder to dismiss as environmental noise. Claude's distribution was bimodal, with 31 runs exploiting zero services and 61 runs exploiting all three, according to the results table in the arXiv HTML paper. qwen2.5-coder:14b produced a more even distribution, with 21 zero-service runs, 18 one-service runs, 36 two-service runs, and 25 three-service runs, according to the attack completeness table in the arXiv HTML paper. GPT-4o-mini never produced a zero-service outcome, but 38 of its runs stopped at two services and 56 reached all three, according to the attack completeness table in the arXiv HTML paper.

A policy document that asks whether a model is capable of offensive action is asking one question. A deployment decision asks a different question: what distribution of behavior appears when the same capability is invoked repeatedly. The Erdem study shows that the second question cannot be inferred from the first.

Strategy Diversity as an Operational Exposure

The strategy results are the clearest warning against mean-only reporting. GPT-4o-mini produced 98 unique attack strategies across 100 runs, with a strategy diversity ratio of 0.98 and command entropy of 5.92, according to the strategy diversity section of the arXiv HTML paper. qwen2.5-coder:14b produced 69 unique strategies, Gemini 48, and Claude 28, according to the strategy diversity table in the arXiv HTML paper.

This is not a small technical footnote. A system that reaches a similar outcome through many trajectories creates a different defensive problem than a system that repeats one pipeline. Signature-based defenses can tolerate repetition. They are strained by variation. Erdem's discussion states that GPT-4o-mini's strategy diversity challenges signature-based detection, according to the discussion section of the arXiv HTML paper.

The finding also complicates evaluation language. A benchmark may report success rate, average iterations, or average command success. Those numbers are useful, but they do not say whether an operator is facing a narrow behavior band or a wide one. In the study, GPT-4o-mini's mean full-exploitation rate was lower than Gemini's, but its strategy diversity was far higher, according to the results section of the arXiv HTML paper. A procurement, audit, or risk function that sees only the first metric misses the second.

Refusals, Outages, and What the Logs Actually Say

The study also shows why evaluation artifacts need auditability beyond summary labels. The abstract states that an earlier draft classified Claude events as safety refusals, but a full-log audit found they were upstream API failures, not model-level refusals, according to the arXiv abstract for the study. The paper reports 91 Anthropic overloaded_error responses among 1,135 Claude API calls, truncating 39 of 100 Claude runs during a documented capacity event, according to the arXiv abstract for the study.

That correction matters for governance because labels travel faster than logs. A refusal-rate claim, once detached from the event trace that produced it, can become evidence for a safety property the system did not actually exhibit. Erdem reports that across 400 runs and four providers, no model produced a content refusal that survived the orchestrator's one-shot authorization re-prompt, according to the refusal analysis in the arXiv HTML paper.

The boundary between model behavior and infrastructure behavior is not academic. If an evaluation report treats service unavailability as safety refusal, the governance system records the wrong control. If a deployment dashboard treats a non-event as a safety event, the operator's risk model becomes more confident exactly where the evidence should make it more cautious.

The Distribution Is the Governance Object

The study's conclusion is not merely that some models performed better than others. It is that repeated autonomous runs against a fixed target produced materially different exploitation outcomes, strategies, and failure modes, according to the conclusion of the arXiv HTML paper. That makes variance itself a governance object.

For an operator, the question is not only whether a model can exploit a service. It is whether the same configuration produces a bounded, characterizable distribution of behavior under repeated invocation. The first question supports capability classification. The second supports operational assurance. An evaluation regime that answers the first while omitting the second has not measured the part of the system that a defender must actually govern.

The timing results sharpen the issue. The first successful exploit occurred around iteration 4 to 6, corresponding to roughly 15 to 30 seconds of wall-clock time, according to the additional findings in the arXiv HTML paper. Within that window, the defender is not observing an average model. The defender is observing one draw from a behavioral distribution.

Open Questions

• At what point does a benchmark report become incomplete if it reports mean performance without the shape of the run-to-run distribution?
• What standard applies when two systems have similar success rates but radically different strategy diversity?
• How does a supervisor verify that a reported refusal was a model-level refusal rather than an infrastructure artifact?
• What evidence should accompany an offensive-security benchmark when the operational risk is concentrated in tail behavior rather than average behavior?
• How should an audit function treat a capability that is reliable in aggregate but unpredictable in trajectory?

The Erdem paper does not settle those questions. It makes them harder to avoid. The evaluation object is not simply the agent's mean score. It is the distribution that appears when the agent is invoked again, under the same conditions, against the same target, while the operator has seconds to understand what kind of run has begun. The policy instruments and the deployment tempo are not aligned.