Domain-Conditioned Safety and the Reproducibility Question

A June 3, 2026 paper measured zero successful multi-step attacks against current frontier models on a 793-episode browser benchmark. The same model weights fell to the identical attack style, applied to coding tasks, up to 100% of the time.

On June 3, 2026, researcher Nicholas Saban published Domain-Conditioned Safety in Frontier Computer-Using Agents, a paper that challenges a widely cited category of finding in the agentic AI red-teaming literature: reported prompt-injection attack success rates of 42% to 98% against computer-using agents. According to the paper's abstract, those headline numbers cluster on retired models and on the single most-vulnerable model in each study's panel, a sampling pattern that can make an attack look more broadly effective than it is against the models organizations are actually deploying today.

To test this, the paper introduces CUA-HandCrafted, a public benchmark of 793 episodes spanning 24 multi-step web tasks, 56 attack templates, 8 attack families, and 4 system-prompt configurations, built by reproducing published attack techniques as hand-crafted templates and running them against current frontier systems rather than retired ones. Against Claude Sonnet 4.6 and GPT-5.4, the paper reports zero successful multi-step attacks out of 140 attempts, with a Clopper-Pearson 95% upper confidence bound of 2.60%. A prompt ablation test indicates this resistance is a property of the model weights themselves, not merely a function of a well-constructed system prompt. Yet the same model weights, tested on a sister coding-agent benchmark called SkillBench, fell to hand-crafted skill-injection attacks at rates up to 100%.

The Claim: Safety Hardening Does Not Generalize Across Modality

The paper's central argument is that the literature's high, oft-cited attack success rates are largely an artifact of reinforcement-learning-optimized injection text, adversarial strings specifically tuned against a target model, rather than evidence that the underlying attack categories themselves are broadly effective against current frontier systems. More consequentially, the paper argues that frontier safety hardening is domain-conditioned: resistance built into a model applies specifically to the browser surface, which has been the most heavily targeted and most heavily red-teamed attack surface to date, and does not transfer to other computer-using-agent modalities such as coding assistants, where the same weights remain highly vulnerable to the same style of attack.

If this finding holds up under scrutiny, it reframes a significant share of prior red-teaming results. A 98% attack success rate reported against a browser-based agent in an earlier study, if achieved using RL-optimized injection text against a since-hardened or retired model, says less about the general vulnerability of computer-using agents than it appears to say. Meanwhile, a 0% success rate against the current frontier on browser tasks says nothing about the same model's vulnerability in a coding context, where this paper found up to 100% success using comparable attack techniques.

The Reproducibility Argument

The paper's most pointed methodological claim is aimed not at any specific prior result but at a practice across the field: reporting attack techniques without releasing the optimized injection strings used to achieve a headline number, or extrapolating safety conclusions from browser-domain testing to other computer-using-agent modalities, makes published attack success rate numbers unreproducible, according to the paper. This is a direct challenge to how red-teaming results have been communicated in the recent agentic AI literature: a percentage without the underlying adversarial artifact and without domain-scoped claims is not independently verifiable, and this paper argues it should not be treated as though it were.

This reproducibility critique lands in a period when red-teaming results have functioned as a primary form of public evidence about agentic AI risk, cited by vendors, researchers, and policymakers alike. If a meaningful share of those headline numbers cannot be reproduced without access to withheld optimized strings, and if the numbers that can be reproduced do not generalize beyond the specific modality tested, the evidentiary base for claims about agentic AI safety is narrower than the raw percentages suggest.

What Domain-Conditioning Means for Governance

The CISA-led Five Eyes advisory on agentic AI advises organizations to assume agentic AI systems may behave unexpectedly and to plan deployments accordingly until security practices, evaluation methods, and standards mature. Domain-conditioned safety hardening is a specific, empirically grounded version of that caution: a model's demonstrated safety on one surface, even a heavily tested one, provides no basis for assuming equivalent safety on a different surface the same model can operate in. A governance framework or procurement standard that certifies a model's safety based on browser-domain red-teaming results, without separately testing coding-domain or other modality-specific attack surfaces, would be extending a safety claim well past the boundary this paper's own evidence supports.

Open Questions

• If safety hardening is domain-conditioned rather than general, what testing regime would be required to certify a model's safety across every computer-using-agent modality it is deployed to operate in, rather than the one most heavily red-teamed?
• Should published red-teaming results be required to include the optimized injection strings used to achieve a headline attack success rate, as this paper's reproducibility argument implies, and who would enforce that disclosure standard?
• How many previously reported high attack success rates in the agentic AI literature would fail to replicate against current frontier models if retested using this paper's methodology?
• Does the 100% attack success rate on SkillBench indicate that coding-agent safety is meaningfully behind browser-agent safety, or that SkillBench itself tests a fundamentally different and harder threat model?
• At what point does an organization deploying a computer-using agent across multiple modalities, browser and coding tasks alike, need to treat each modality as a separately unverified safety surface rather than assuming uniform protection?
• What accounts for the field's apparent tendency to concentrate red-teaming effort on the browser surface specifically, and does that concentration itself create a false sense of comprehensive safety coverage?

The governance artifact is retained. The governance function is not.