A Skill Is Code and Instruction at Once, and No Scanner Reads Both

A June 2026 runtime-verified benchmark finds that the strongest detector for malicious agent skills collapses precisely where code and instruction meet.

AI coding agents such as Claude Code and Gemini CLI increasingly extend themselves through third-party skills: markdown packages that bundle natural-language instructions, executable scripts, and tool permissions into a single installable unit. A paper posted to arXiv on June 5, 2026, titled "MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills," by Wenbo Guo, Wei Zeng, Chengwei Liu, Xiaojun Jia, Yijia Xu, Lei Tang, Yong Fang, and Yang Liu, identifies a structural reason this category of software supply chain has been hard to secure: a skill is simultaneously code and agent-facing instruction, and that duality makes its risk "neither pure code nor pure prompt."

The paper opens by noting a gap in the evaluation landscape itself: detection tools built for this hybrid space have never been measured against verified ground truth, leaving their real effectiveness unknown, and evaluations that rely only on samples found in the wild are structurally biased toward whatever attack pattern happens to be common at collection time rather than the full space of what is possible.

Building Ground Truth the Field Did Not Have

To close that gap, the authors constructed MalSkillBench, described as the first runtime-verified benchmark of malicious agent skills. The dataset comprises 3,944 malicious skills labeled along a three-dimensional taxonomy spanning 108 cells, alongside 4,000 matched benign skills for comparison. Of the malicious skills, 3,214 were produced through a closed-loop Generate-Verify-Feedback pipeline that admits only samples whose malicious behavior actually fires inside a Docker sandbox under system-call monitoring and an LLM judge, rather than samples that merely look malicious on inspection. The remaining 703 were collected in the wild.

The verification-yield numbers themselves carry a signal that anticipates the paper's later detection findings: code injection reached a 94.5 percent verification yield, meaning the vast majority of attempted code-injection samples could be confirmed to actually misbehave when run. Prompt injection reached only 75.8 percent. The paper describes this gap as "the same fragility that later makes it hard to detect," meaning prompt-based attacks are harder to even confirm as malicious under controlled execution, a difficulty that recurs when detection tools attempt to classify them after the fact.

What the Wild Sample Reveals

The in-the-wild portion of the dataset was narrow in a way that itself carries a warning. It was dominated by a single cryptocurrency-theft campaign, accounting for 86.6 percent of one behavior pattern and 81 percent originating from just two accounts. Beneath that dominant campaign, the authors identify a small but architecturally new tail of skills attacking the agent control plane directly, a category distinct from the credential-theft pattern that made up the bulk of wild samples. A detection strategy tuned primarily on what is common in the wild would be tuned almost entirely on one campaign, and would have limited exposure to the structurally novel attacks the paper flags as an emerging tail.

The Detector Collapse

The paper's central empirical finding concerns detection performance itself. The strongest skill-specific detector tested reached 98.4 percent recall on code injection, a strong result on its own terms. On prompt-injection and agent-control-plane attacks, the same detector collapses. The paper further reports that wild-only scoring, evaluating detectors solely against in-the-wild samples rather than the full verified benchmark, swings detector rankings by up to 66 recall points, meaning the choice of evaluation set alone can make a mediocre detector look like the best available option or vice versa.

The paper's diagnosis of why detection fails on the hybrid cases is structural rather than incidental: supply-chain scanners and prompt-injection defenses each see only half of a skill, and no combination of the two recovers the joint code-instruction relationship that defines the risk. A supply-chain scanner inspects the executable component. A prompt-injection defense inspects the natural-language instruction component. Neither is built to reason about how the two interact, and the paper's finding is that this interaction, not either component alone, is where the risk that evades detection actually lives.

Extending the Fog

This finding sits directly inside the Five Categories advisory's description of Design and Configuration risk, which warns that unvetted third-party components "may carry excessive or unintended privileges when integrated into agent workflows." A skill is precisely such a third-party component, and this paper demonstrates empirically that the tooling most likely to be deployed to vet it, code scanners and prompt-injection filters, structurally cannot see the property that makes a skill dangerous. The advisory names the risk category. This paper measures the specific detection gap inside it, and finds the gap is not a matter of insufficiently mature tooling catching up over time, but a matter of two tool categories examining two different halves of one artifact and never reconciling what they see.

The paper's own conclusion states the implication without qualification: "Detecting malicious skills therefore requires reasoning jointly over task intent, code, and instructions." That joint reasoning capability, per the paper's own detector results, does not yet exist at a level that holds up against prompt-injection or agent-control-plane attacks, even though it holds up well against code injection alone.

Open Questions

• What standard applies when an organization has deployed both a supply-chain scanner and a prompt-injection filter and reasonably assumes the combination covers third-party agent skills, when this paper finds neither component sees the joint relationship that defines the risk?
• How does a supervisor evaluate a detector's real-world readiness when wild-only scoring can swing its ranking by up to 66 recall points relative to full verified benchmark scoring?
• At what point does the dominance of a single cryptocurrency-theft campaign in wild data distort the field's collective sense of what a malicious skill typically looks like?
• Who is responsible for auditing the architecturally new tail of agent-control-plane attacks the paper identifies, given that these fall outside the pattern most current tooling is tuned to catch?
• What artifact would let an enterprise verify, prior to installing a third-party skill, that its code and its natural-language instructions do not jointly encode a capability neither component reveals alone?
• Does the 108-cell taxonomy generalize to skill ecosystems beyond the agent platforms this benchmark was built against, or does it require reconstruction for each new agent framework?

The loop closed around an oversight function that was never instrumented.