Verification Cost Is a Named Harness Metric

A May 27, 2026 paper asks, in its own title, what the best harness for cybersecurity AI is. The answer it finds is that no single harness wins, and the paper prices the alternative in dollars and hours.

On May 27, 2026, a research team published Towards Cybersecurity SuperIntelligence (CSI), a paper whose subtitle asks directly: what is the best harness for cybersecurity AI. The paper's answer is that the question is malformed. Cybersecurity systems have been converging on a single execution scaffold per agent, an iterative shell loop driven by a large language model, but scaffolds are not interchangeable, rarely interoperable, and no single scaffold dominates across all challenge types, according to the paper's abstract. The authors, Víctor Mayoral-Vilches, Francesco Balassone, María Sanz-Gómez, Paul Zabalegui Landa, Daniel Sánchez Prieto, Marina Oteiza Álvarez, Davide Quarta, and Martin Pinzger, benchmarked five scaffolds, named CSI::Claude, CSI::Codex, CSI::GCAI, CSI::Mistral, and CSI::CAI, against the 33 Cybench challenges, holding the underlying model fixed. The best individual scaffold solved 15 of 33 challenges, 45.5%. The union of four scaffolds solved 17 of 33, 51.5%. The fifth scaffold, CSI::Mistral, solved only 10 of 33 on its own but contributed one exclusive solve no other scaffold found.

The paper's central architectural claim is a blackboard system: a shared substrate where scaffold-specialized agents run in parallel and exchange intermediate findings. That blackboard configuration solved 19 of 33 challenges, 57.6%, a 27% relative gain over the single best scaffold, achieved 25% faster, 20.2 hours versus 26.8 hours, at comparable cost, $5,480 versus $5,122.

Cost as a First-Class Metric

The dollar figures are the detail that separates this paper from a purely academic exercise. Cybersecurity SuperIntelligence reports not just accuracy but time and cost, treating verification expense as a metric with the same standing as solve rate. That framing has a direct governance implication: if the cost of achieving higher coverage through harness composition is comparable to the cost of a single scaffold, then cost is not the barrier to adopting more robust, multi-scaffold architectures. Availability of the architecture is the barrier. Most cybersecurity AI deployments today run a single scaffold because that is the default pattern the tooling ecosystem has converged on, not because a single scaffold is verified to be sufficient.

This is the fourth paper in three weeks to treat "harness" as the first-class substrate object rather than treating the underlying model as the unit of analysis. The pattern is now dense enough to be a signal in its own right: independent research groups, publishing in the same narrow window, are converging on the observation that the orchestration layer, not the model, is where cybersecurity AI capability and cybersecurity AI risk both actually live. A production system disclosed the same week, Microsoft's MDASH vulnerability-scanning harness, reinforces the pattern from the deployment side: Microsoft credited its multi-agent pipeline, not a specific model, with finding 16 of May's Patch Tuesday vulnerabilities.

What "No Single Scaffold Dominates" Implies for Governance

If no single scaffold dominates across challenge types, then any compliance framework that certifies a specific scaffold configuration as safe, or evaluates a vendor's cybersecurity AI product against a fixed benchmark using a fixed scaffold, is testing a narrower claim than the one that matters operationally. The paper's own results show that scaffold performance is task-dependent: CSI::Mistral, the weakest individual performer at 10 of 33, still contributed a unique solve the stronger scaffolds missed entirely. A governance regime built around "does this scaffold pass benchmark X" would miss the diversity effect the paper's blackboard architecture is designed to exploit.

The blackboard architecture itself introduces a new governance surface. When scaffold-specialized agents exchange intermediate findings through a shared substrate, the correctness of any single agent's output now depends on the correctness of what other agents wrote to that shared substrate. An error introduced by one scaffold can propagate to every other scaffold reading from the blackboard, a dynamic the CISA-led Five Eyes advisory on agentic AI names directly in its Structural risk category: the interconnected structure between agents increases attack surface and complexity, and the failure of one component propagates through the topology. The CSI paper demonstrates the performance benefit of that interconnection. It does not describe a mechanism for isolating or attributing a fault that originates on the blackboard itself.

The Verification Cost the Paper Does Not Price

The paper prices the cost of running the benchmark: dollars and hours to solve 33 Cybench challenges under five different scaffold configurations. It does not price a different cost: the cost of verifying, independently, that a given blackboard run's 19 solved challenges were solved correctly, and not merely reported as solved by an agent whose validation step was itself wrong. Cybench challenges have known correct answers, which makes benchmark verification tractable. Real-world vulnerability discovery, the domain MDASH and similar production systems operate in, does not come with a ground truth the way a benchmark does. The paper's cost metric measures the harness's own operating expense. It does not measure the expense an independent auditor would incur to verify the harness's output without simply trusting the harness's own validation stage.

Open Questions

• If cost and time are now standard metrics for harness comparison, when will verification cost, the cost of an independent party confirming a harness's output is correct, become an equally standard metric?
• How does a compliance framework certify a multi-scaffold blackboard architecture whose behavior emerges from the interaction of specialized agents, rather than from any single evaluable component?
• What happens when an error is written to the blackboard by one scaffold and propagates to every other scaffold before any validation step catches it?
• Given that no single scaffold dominates across challenge types, what is the correct unit for a regulator or auditor to evaluate: the individual scaffold, the blackboard architecture, or the full deployed system including whichever scaffolds happen to be composed at a given time?
• At what point does treating "harness" as the unit of analysis require its own dedicated evaluation standard, separate from existing model evaluation frameworks?
• Does the 27% relative performance gain from blackboard composition, documented here, create commercial pressure toward more interconnected multi-scaffold systems faster than governance frameworks for auditing those systems can be built?

The governance artifact is retained. The governance function is not.