CryptoSkill Evaluation Framework

April 3, 2026 · View on GitHub

Design Philosophy

Evaluation should be cheap by default, deep when it matters. Most of the 800+ skills can be scored with static analysis (zero LLM cost). Only fund-moving skills (exchanges, DeFi, wallets, trading) need expensive LLM-as-judge evaluation. The framework produces a single 0-100 Quality Score per skill, decomposed into transparent dimensions.

Inspired by: npm quality scores (26 attributes, 3 aspects), Minara crypto-skill-benchmark (76 scenarios, LLM-as-judge), ClawHub repository context scoring (70% codebase + 30% metadata), SkillsMP maintenance tiers, GoPlus AgentGuard (6-detector security pipeline, trust registry), SkillCompass (6 dimensions with hard security gate), SkillTester (formal utility + security formulas), OWASP Agentic Skills Top 10, Chrome Web Store (install velocity + retention), and Agent Skills Directory (dmgrok, 0-100 quality scoring).

Key Research Findings

SourceKey Insight
ClawHub43.4% of skill repos have zero stars, 66.5% zero forks. 5 of top 7 most-downloaded skills were confirmed malware at peak.
SkillsMPMaintenance recency is the "strongest trust indicator". Minimum quality gate: 2 GitHub stars.
SkillCompass80% of skills lacked rejection boundaries (trigger dimension). Mean score 73.8/100 for top 100 ClawHub skills.
GoPlus AgentGuard6 security detectors with 160+ credential patterns, 75+ prompt injection patterns, 112+ malicious command patterns.
Oathe ClawMutinyFound 88 dangerous skills where the leading scanner only flagged 7 (91% miss rate). Signature-based detection misses instruction-layer threats.
Snyk ToxicSkills13.4% of scanned skills had critical issues. 36.82% had any flaw. 76 confirmed malicious payloads.
OWASP AST10Recommends "semantic + behavioral multi-tool pipeline" over pattern-matching. Provides Universal Skill Format with ed25519 signatures.
npm (npms.io)26 package attributes feed 11 metrics, then 3 aspects (Quality 40%, Popularity 30%, Maintenance 30%). Recalculated every 15 days.
OpenRouterReal usage data outperforms synthetic benchmarks as a quality signal.
SkillTesterFormal scoring: utility = efficiency-weighted success rate; security = probe pass rate across 3 groups (abnormal behavior, permission boundary, sensitive data).
Agent Skills DirectoryMaintenance 50pts, Documentation 30pts, Provider Trust 20pts. Security gates must pass before inclusion.

Score Architecture

Quality Score (0-100) = Static Score (40%) + Security Score (20%) + Depth Score (40%)
ComponentCostRunsCoverage
Static ScoreFreeEvery bot cycle (6h)All 800+ skills
Security ScoreFreeEvery bot cycle (6h)All 800+ skills
Depth Score~$0.10/skillOn change or weeklyFund-moving skills get LLM judge; others get heuristic

Layer 1: Static Score (40 points max)

Computed purely from file analysis. No LLM, no API calls. Runs on every skill every 6 hours.

Dimensions

DimensionPointsHow Computed
Documentation10SKILL.md quality (see checklist below)
Completeness8Required files present (SKILL.md, SOURCE.md, references/)
Freshness7Last update recency (from git log or SOURCE.md)
Provenance8Official classification, GitHub org verification
Structure7Follows naming conventions, proper frontmatter

Documentation Checklist (10 points)

CheckPointsMethod
SKILL.md exists1File check
Has valid YAML frontmatter (name, description)1YAML parse
Description > 50 chars1Length check
Has "When to Use" or trigger section1Text search
Has examples or usage section1Text search
Has references/ directory1Directory check
Word count > 200 (meaningful content)1Word count
Has version in frontmatter1YAML check
Has metadata.openclaw section1YAML check
No placeholder text ("TODO", "TBD", "Lorem")1Text search

Completeness (8 points)

CheckPoints
SKILL.md present2
SOURCE.md present with all fields2
references/ directory with at least 1 file2
scripts/ directory (if skill is executable)1
_meta.json present1

Freshness (7 points)

Based on the source repo last commit date:

RecencyPoints
Within 1 month7
Within 3 months5
Within 6 months3
Within 1 year1
Older than 1 year0

Provenance (8 points)

CheckPoints
Classification = OFFICIAL4
Source GitHub org has 10+ public repos2
Source repo has 5+ stars1
License declared1

Structure (7 points)

CheckPoints
Directory name is kebab-case1
Official skills follow project-official-name pattern2
Correct category placement2
No hardcoded API keys or secrets in SKILL.md1
Valid YAML frontmatter (parses without error)1

Layer 2: Security Score (20 points max)

Static security analysis. No LLM needed. Flags critical issues.

Dimensions

DimensionPointsHow Computed
Credential Safety6Regex scan for API keys, private keys, mnemonics
Code Safety6Scan scripts/ for dangerous patterns
Permission Scope4Analysis of allowed-tools in frontmatter
Supply Chain4External dependency count, URL destination checks

Security Gate (Binary)

Independent of score. A FAIL flag if any of:

  • Private key or mnemonic pattern found in any file
  • Dynamic code execution with user input in scripts
  • Fetches to non-HTTPS URLs
  • Obfuscated code detected (base64 encoded strings > 100 chars)

Permission Risk Tiers

Allowed ToolsRiskDeduction
Read, Glob, Grep onlyLow0
+ Edit, WriteMedium-1
+ BashHigh-2
+ Bash with network accessCritical-3

Security Scanning Depth (informed by GoPlus AgentGuard + Snyk + OWASP AST10)

Tier 1 — Pattern-Based (runs every 6h, free):

  • 160+ credential patterns (API keys, tokens, private keys, DB strings)
  • 75+ prompt injection patterns (including base64, zero-width obfuscation)
  • 112+ malicious command patterns (RCE, reverse shells, encoded payloads)
  • Data exfiltration: sensitive path access, HTTP/DNS tunneling, clipboard scraping

Tier 2 — Behavioral Analysis (runs weekly, free):

  • Declared vs. needed permissions comparison (permission abuse detection)
  • URL destination analysis (malicious domains, suspicious TLDs, phishing)
  • Instruction-layer threat scan (markdown behavioral directives — catches the 91% that signature scanners miss per Oathe research)

Tier 3 — Deep Scan (on new skill addition):

  • AST dataflow analysis for scripts/ (Python, JS)
  • Binary/minified code detection
  • External dependency resolution and vulnerability check

Layer 3: Depth Score (40 points max)

Two tracks depending on skill category:

Track A: LLM-as-Judge (Fund-Moving Skills)

For categories: exchanges, defi, trading, wallets, payments

Uses the crypto-skill-benchmark methodology: send scenarios to an LLM simulating the skill, then judge responses.

DimensionPointsWhat it measures
Safety12Confirmation before fund moves, amount precision
Coverage10Breadth of operations and chains
Robustness8Adversarial input handling, scam detection
Routing6Intent-to-action mapping accuracy
UX4Output completeness

Scenario tiers:

  • Basic (5 scenarios): swap, send, balance, price, deposit ($0.05/skill)
  • Standard (20 scenarios): + perps, limit orders, scam tokens ($0.15/skill)
  • Full (76 scenarios): All core + adversarial ($3/skill)

Track B: Heuristic Depth (Non-Fund-Moving Skills)

For categories: chains, analytics, identity, prediction-markets, social, ai-crypto, dev-tools, mcp-servers

No LLM needed. Scored by content analysis:

DimensionPointsHow Computed
Actionability12Has scripts/ or CLI commands? Has API endpoints documented?
Specificity10References specific contracts, addresses, or API URLs?
Examples8Has concrete usage examples with expected outputs?
Error Handling6Documents error cases, edge cases, limitations?
Integration4References other skills, MCP servers, or tools?

Category-Specific Bonus Dimensions

Certain categories get bonus checks (up to +5, capped at 40 total):

CategoryBonus CheckPoints
MCP ServersHas install command (npm/docker/SSE URL)+2
MCP ServersHas tool schema or endpoint documentation+3
AnalyticsCites data sources (APIs, contracts, subgraphs)+3
AnalyticsDocuments rate limits and freshness+2
IdentityReferences specific ERC standards+2
IdentityDocuments privacy considerations+3
Dev ToolsHas build/test commands+3
Dev ToolsDocuments supported chains/frameworks+2

Composite Score Formula

quality_score = static_score + security_score + depth_score
# Clamped to 0-100

Grade Tiers

GradeScoreBadgeMeaning
A80-100GreenProduction-ready, well-documented, secure
B60-79BlueGood quality, minor gaps
C40-59YellowUsable but significant gaps
D20-39OrangeMinimal quality, use with caution
F0-19RedNot recommended

Implementation Plan

Phase 1: Static + Security Scoring (Week 1)

  • Add scripts/score-skills.py to compute static + security scores
  • Store scores in docs/skills.json per skill
  • Run in bot cycle after catalog regeneration

Phase 2: Website Display (Week 2)

  • Score badge on skill cards (A/B/C/D/F with color)
  • Safety gate indicator
  • Sort/filter by score
  • Tooltip with dimension breakdown

Phase 3: LLM Depth Scoring (Week 3-4)

  • Integrate crypto-skill-benchmark scenario runner
  • Run basic tier for all fund-moving skills
  • Update weekly

Phase 4: Regression Tracking (Week 5+)

  • JSONL history per skill
  • Alert on score drops >10 points
  • Score trend sparkline on website

Cost Estimate

PhaseSkillsCost/RunFrequencyMonthly
Static + Security800$0Every 6h$0
Heuristic Depth500$0Every 6h$0
LLM Basic (5 scenarios)300$0.05/skillWeekly~$60
LLM Standard (20 scenarios)50$0.15/skillMonthly~$8

Total monthly: ~$70

References

Benchmarks & Scoring

Security

Registries & Marketplaces