dsh-skill-pack-security poison-sample benchmark results

Deterministic malicious-pattern benchmark. Regenerate with pnpm --dir provider run build && node benchmark/run.mjs (zero new dependencies; the checks are the shipped provider/src/vet engine, run from the built provider/lib).

Method

• Detector: the three malicious-pattern checks from provider/src/vet/checks.ts — installScriptsCheck (download+exec, obfuscated eval, credential/global-config touches), networkExfilCheck (exfil/receiver domains in script files), and obfuscationCheck (dynamic eval + base64/hex payloads).
• Detection = the check's verdict is fail (the state that blocks installation under the deny gate policy). A warn is not counted as a detection.
• Labels: label: true = a malicious sample of that class; label: false = a benign plugin file (no lifecycle scripts, harmless URL, or clean readable code).
• Placeholders only: every URL/domain is a harmless placeholder (example.com, webhook.site, pastebin.com, …) used purely as a detection signal; nothing is contacted.

Per-class metrics

Class	TP	FP	FN	TN	Detection rate (recall)	FPR	Precision	F1
install-scripts	8	0	0	8	1.000	0.000	1.000	1.000
network-exfil	4	0	2	5	0.667	0.000	1.000	0.800
obfuscation	5	0	1	5	0.833	0.000	1.000	0.909

Overall

Aggregate	Precision	Recall	F1
Macro (mean of the three classes)	1.000	0.833	0.903
Micro (pooled TP/FP/FN across classes)	1.000	0.850	0.919

• Samples: 38 total (20 malicious, 18 benign).
• Overall false-positive rate (micro): 0.000.

Per-sample notes

• Missed malicious samples (false negatives): network-exfil/ne-pos-03, network-exfil/ne-pos-04, obfuscation/ob-pos-01.
• Flagged benign samples (false positives): none.

Gap vs OSV / Socket (honest)

This benchmark measures only the self-built static heuristics. OSV and Socket are database/telemetry-backed: OSV matches lockfile versions against published advisories, and Socket layers maintainer/adversary telemetry (install-script provenance, package age, typosquat scores) on top of static analysis. The gap, in both directions:

• Coverage: the self-built checks catch the shapes they were written for (download+exec, decoded-then-executed payloads, credential/global touches, exfil domains, eval+base64). They have no vulnerability database, so a known CVE in a pinned dependency is invisible to install-scripts/network-exfil/obfuscation — that evidence belongs to the sbom check, which delegates to osv-scanner/npm audit when those CLIs are present (not exercised here, which needs a real on-disk project).
• Precision: the heuristics are pattern-based, so renamed or lightly re-encoded payloads evade them, while Socket's telemetry can also flag reputational risk the heuristics cannot see. The FPR reported here is only the pattern-level false-positive rate on a curated benign set, not a whole-ecosystem FPR.

Known limitations (honest)

• Shape, not intent. These checks fire on syntactic indicators; they do not execute anything or model adversarial intent.
• Single-file heuristic view. The dataset feeds one file (or a package.json) per sample; the real engine walks a whole package, and cross-file payloads are out of scope here.
• Telegram/Discord webhook domains are dead entries. EXFIL_DOMAINS lists api.telegram.org/bot and discord.com/api/webhooks (host + path), but the check compares them against the bare URL host, so those two receiver domains never match (ne-pos-03, ne-pos-04).
• Obfuscation base64 must be inline. The base64-blob regex matches a literal directly inside atob()/Buffer.from() or a 100+ char standalone blob, so const s='<short-base64>'; eval(atob(s)) is missed (ob-pos-01).
• No evasion corpus. Obfuscation samples use plain atob/base64; re-encoded, split, or novel obfuscation is not in this regression set.