A Full-Lifecycle Offensive-AI Benchmark

A June 3, 2026 benchmark from a UC Berkeley-led team evaluates AI agents across the entire vulnerability lifecycle, discovery, proof of concept, and patch, using 920 real vulnerabilities across 139 open-source projects. It is the direct benchmark family Microsoft cited when it disclosed its own vulnerability-scanning harness three weeks earlier.

On June 3, 2026, a research team led by Dawn Song at UC Berkeley, including Tianneng Shi, Robin Rheem, Dongwei Jiang, Mona Wang, and a dozen other coauthors, published CyberGym-E2E, a benchmark designed to close a specific gap in how AI agent cybersecurity capability is measured. According to the paper's abstract, existing cybersecurity evaluations of AI systems are limited in scale or scope and fail to capture the end-to-end lifecycle of real-world software vulnerability discovery and remediation. CyberGym-E2E addresses this by evaluating agents across the full lifecycle, discovery, proof-of-concept generation, and patch generation, using an automated, agent-enhanced pipeline that transforms open-source vulnerability data into realistic evaluation environments. The benchmark currently comprises 920 real-world vulnerabilities across 139 different open-source projects, and the paper has been accepted at ICML 2026.

The benchmark's lineage matters. CyberGym, the earlier benchmark family this paper extends, is the same 1,507-task public benchmark that Microsoft cited when it disclosed MDASH, its multi-agent vulnerability-scanning harness, on May 13, 2026, reporting an 88% score. CyberGym-E2E is not the same evaluation Microsoft used, but it comes from the same research lineage and addresses the same domain, offensive and defensive AI capability against real software vulnerabilities, with a deliberately more comprehensive scope: not just discovery, but the full chain through proof of construction and remediation.

Why Full-Lifecycle Evaluation Changes the Governance Question

A benchmark that measures only vulnerability discovery answers a narrower question than a benchmark that measures discovery, proof-of-concept construction, and patch generation together. Discovery alone tells a governance framework whether an agent can find a bug. The full lifecycle tells a different and more consequential story: whether an agent, or a coordinated system of agents, can independently complete every step from finding a flaw to producing a validated exploit to generating a fix, without a human in the loop at any intermediate stage. That is the capability profile of an actor that could, in principle, operate on either side of the same lifecycle: a defender patching vulnerabilities before disclosure, or an attacker constructing a working exploit before a patch exists.

CyberGym-E2E's scale, 920 vulnerabilities across 139 projects, and its automated pipeline for generating new evaluation environments from open-source vulnerability data, mean the benchmark itself is a capability multiplier. A benchmark this comprehensive is also, necessarily, a training and evaluation resource that lowers the barrier to building a system with dual-use capability across the same lifecycle it measures. The paper does not frame its contribution this way; it frames CyberGym-E2E as addressing a measurement gap in AI cybersecurity capability. But a measurement gap and a capability gap are adjacent problems, and closing the former tends to narrow the latter as a side effect.

Cross-Referencing the Harness Question

MDASH and CyberGym-E2E sit on opposite sides of the same evaluation question. MDASH is a production system Microsoft credited with real findings against real Patch Tuesday vulnerabilities, evaluated in part against the earlier CyberGym benchmark. CyberGym-E2E is the more comprehensive academic benchmark, built by researchers with no described operational relationship to Microsoft's system, that extends the same evaluation family to cover the full lifecycle MDASH's disclosure did not fully detail. Microsoft's disclosure described discovery, validation, deduplication, and proof construction inside MDASH's own pipeline. It did not describe an equivalent, externally verifiable patch-generation stage. CyberGym-E2E's explicit inclusion of patch generation as a benchmarked capability suggests the field is moving toward evaluating the complete lifecycle as a matter of course, which will make disclosures like Microsoft's, framed only around discovery, look partial by comparison going forward.

This convergence is not evidence of coordination between the two efforts. It is evidence that the full lifecycle, not just discovery, is becoming the standard unit against which offensive and defensive AI cybersecurity capability gets measured, following the same broader pattern in this arc of harnesses and benchmarks becoming the object of governance interest rather than individual models.

What the Benchmark Does Not Address

CyberGym-E2E measures whether an agent can complete the full lifecycle successfully. It does not, based on the public abstract, address the governance question that follows directly from a high score on such a benchmark: what happens when a system capable of full-lifecycle vulnerability handling is deployed outside a research or defensive context, and what artifact would allow an outside party to verify which lifecycle stages a given deployed system is actually permitted, and monitored, to execute in production. A benchmark score is a capability measurement, not a deployment control.

Open Questions

• As full-lifecycle benchmarks like CyberGym-E2E become the standard, what governance distinction, if any, separates a system benchmarked for defensive patch generation from the same system redirected toward offensive exploit construction?
• What oversight applies to the automated, agent-enhanced pipeline CyberGym-E2E itself uses to generate new evaluation environments, given that pipeline is itself a form of agentic capability operating on vulnerability data?
• How does a benchmark of this scale and comprehensiveness get audited for whether its own evaluation environments introduce vulnerabilities into the broader open-source ecosystem it draws from?
• When a production system like MDASH is compared against an academic benchmark like CyberGym-E2E, who defines the equivalence between the production system's internal validation stages and the benchmark's formal evaluation criteria?
• At what point does full-lifecycle vulnerability-handling capability, once benchmarked and published, become a capability governance frameworks need to track independent of which organization deploys it?
• What happens to the benchmark's evidentiary value once it, like other agent benchmarks, becomes a target for training-data contamination or gaming by systems optimized specifically to score well on it?

The governance artifact is retained. The governance function is not.