A Red-Team Benchmark That Names the Audit Axes

A June 1, 2026 benchmark measured attack success rates ranging from 32% to 81% against production frontier models, using authorization boundary attacks embedded in ordinary tool responses. The paired guard model cut the panel average to 2.4%. The benchmark's own versioning discipline is as notable as its results.

On June 1, 2026, researchers Hiskias Dingeto and William Leeney published AgentRedBench, a dynamic, LLM-driven red-teaming benchmark targeting a specific and underexamined threat: indirect prompt injection carried inside the content that tool-use agents read back from third-party integrations such as Gmail, Salesforce, or Jira, content the user neither writes nor controls. According to the paper's abstract, existing benchmarks under-measure this threat because most cover only a handful of integrations, replay the same attack payload across runs, and train open-source guard models on chat-style data rather than on the tool-response content where the actual attacks live.

AgentRedBench addresses this with 215 scenarios built around "subtle underspecified authorization," attacks positioned at the boundary of what a user's original request actually authorizes, spanning 24 enterprise integrations across nine functional families and five attack types. Tested against an eight-model panel spanning Anthropic, OpenAI, and Google systems, the no-guard attack success rate ranged from 32% for Claude Sonnet 4.6 to 81% for Gemini 3 Flash. The researchers also released AgentRedGuard, a guard model trained on an integration-diverse corpus of adversarial tool-response content, which cut the panel's average attack success rate from 69.9% to 2.4%, at a false-positive rate of 0.37%, outperforming existing open-source guard baselines including Llama Guard, PromptGuard 2, and ProtectAI on both detection and false-positive axes. Cross-integration and cross-attack-type holdout tests confirmed the improvement generalizes beyond the training subset.

Underspecified Authorization as a Category

The paper's core conceptual contribution is naming "subtle underspecified authorization" as a distinct attack category, separate from the more commonly discussed direct prompt injection, where malicious instructions are inserted crudely and often detectably into content an agent processes. Underspecified authorization attacks instead exploit ambiguity in what a user's original request actually permits: a request to "check the calendar and reschedule if needed" contains an implicit boundary around what counts as "if needed" that a malicious integration response can exploit without ever looking like an injected command. This is a harder detection problem than filtering for suspicious instruction-like text, because the exploit lives in the gap between the user's intent and the literal scope of their words, not in an obviously foreign instruction.

The wide variance in no-guard attack success rate, 32% to 81% across an eight-model panel, is itself a governance-relevant finding independent of the guard model's performance. A 49-percentage-point spread among current frontier models on the same benchmark means that a deploying organization's actual exposure to this attack class depends heavily, and non-transparently, on which model backs its agent, a dependency most procurement and compliance processes are not currently built to evaluate.

What the Benchmark's Versioning Discipline Signals

The paper's methodological choice to keep the canonical scenario set out of public training corpora, evaluated instead through a maintainer-mediated channel with immutable versioning, is a direct response to a well-documented failure mode in AI benchmarking: once a benchmark's test cases become part of a model's training data, performance on that benchmark stops measuring the underlying capability and starts measuring memorization of the specific test cases. This is the same integrity problem that governance frameworks face when a compliance artifact can be gamed once its evaluation criteria become fully public and static. AgentRedBench's maintainer-mediated channel is an attempt to preserve the benchmark's evidentiary value over time, which is a different and more demanding standard than simply publishing a fixed test set once.

The CISA-led Five Eyes advisory on agentic AI names accountability risk as a category where agentic system architecture can obscure what caused a particular action. AgentRedBench and AgentRedGuard operate one layer upstream of that accountability question: before an enterprise can trace what caused an action, it needs a way to measure how often an integration-borne attack succeeds against its specific model and guard configuration in the first place. A 2.4% panel-average attack success rate with AgentRedGuard deployed is a meaningful reduction from 69.9%, but it is not zero, and the paper does not claim it is a complete solution to the underlying authorization-ambiguity problem the benchmark measures.

What Remains Outside the Benchmark's Scope

AgentRedBench measures whether an attack succeeds in causing an agent to take an unauthorized action within the scope of a single scenario. It does not, based on the public abstract, measure whether an enterprise deploying a guard model like AgentRedGuard can subsequently produce an auditable record proving the guard was active, correctly configured, and evaluated every relevant tool response at the time an incident is later investigated. A benchmark that measures attack success rate is answering "how often does this fail." It is not, by itself, answering "how would anyone outside the deployment verify that the guard was actually running when it mattered."

Open Questions

• Given a 49-percentage-point spread in no-guard attack success rate across current frontier models, what disclosure obligation, if any, should apply to vendors regarding their models' relative exposure to underspecified-authorization attacks?
• How does an enterprise verify, after an incident, that AgentRedGuard or a comparable guard model was actually active and evaluating the specific tool response involved, rather than reconstructing that fact from the deploying organization's own logs?
• Does AgentRedBench's maintainer-mediated versioning scheme scale as the benchmark grows past 215 scenarios and 24 integrations, or does gatekeeping the canonical scenario set create its own bottleneck?
• What happens to the 2.4% residual attack success rate under AgentRedGuard in a real production environment, where integration diversity likely exceeds the 24 integrations covered in the benchmark's training corpus?
• At what point does "subtle underspecified authorization" become a standard category regulators and auditors reference, comparable to how "prompt injection" is now broadly understood?
• Who is accountable when a guard model with a 0.37% false-positive rate blocks a legitimate action at scale, given that even a small false-positive rate compounds across a high-volume agentic deployment?

The governance artifact is retained. The governance function is not.