Why a scanner can look strong and still fail

Start with the deployment question

Before choosing a score, state what a detector is supposed to do. Is it reviewing a familiar internal library, triaging unknown public packages, or blocking an installation? Those uses tolerate different kinds of error. A benchmark is useful only when its operating conditions inform that decision.

Hold out more than random rows

MaliciousSkillBench distinguishes random, structural-disjoint, and source-disjoint protocols. Closely related samples can make a detector appear more general than it is. Keep the split rule, duplicate handling, and source relationship attached to any reported result.

Count the benign review burden

Record both missed malicious samples and benign packages flagged for review. A tool that catches attacks by flagging almost everything may create an unusable queue. Conversely, few warnings can reflect low sensitivity. Compare the trade-off at a stated configuration rather than selecting one favorable number.

Make scanner claims reproducible

Keep the package snapshot, scanner revision, policy, model configuration, and scoring rules. Read maintainer limits and note whether optional services receive package content. Re-run a meaningful local sample before adopting a published configuration. This guide proposes review questions; it does not publish an independent scanner ranking.

Worked example: a review queue that catches almost everything

A hypothetical detector flags 95 of 100 malicious packages and 80 of 100 benign packages. The malicious recall looks attractive, but reviewers now face 175 alerts, most of which do not distinguish benign from malicious very well. A second detector with fewer alerts may still miss important cases. Neither trade-off can be judged from one headline score.

Work through the evidence

Report the confusion matrix, threshold, class balance, source relationships, and operating context. Use held-out sources when the deployment involves unfamiliar packages. Explain whether the tested unit is a file, a complete skill, or a revision. Group near-duplicates appropriately so copies of the same pattern do not silently dominate the result.

Decision criteria and a useful review record

Choose a configuration against a stated review budget and consequence of misses. Preserve errors and unprocessed files rather than discarding them. A detector comparison should use compatible artifacts and settings or clearly label the differences. The dataset shelf links resources for this work; it is not an independently validated ranking of their current tools.