EmbedEval

LLM Embedded Domain Knowledge Probe — Do LLMs actually understand embedded firmware, or do they just pattern-match?

EmbedEval measures whether LLMs possess the implicit domain knowledge to write safe embedded C code. It covers Zephyr RTOS, ESP-IDF, STM32 HAL, FreeRTOS, Linux kernel drivers, and Yocto recipes across 267 test cases (219 public + 48 private held-out).

Unlike HumanEval or SWE-bench which test general coding, EmbedEval tests knowledge that only embedded engineers would have: interrupt safety, cache coherency, DMA alignment, power management, and real-time constraints — without telling the LLM what to check.

Live leaderboard · Methodology · Roadmap · Contribute

Result write-ups are published on EdgeLog, the maintainer's embedded and edge engineering blog. Start with what the firmware benchmark actually measured.

---

Key Insight: The Implicit Knowledge Gap

Most benchmarks tell the LLM exactly what to do. EmbedEval tells the LLM what to build but not how to make it safe:

Prompt:  "Implement DMA transfer from src to dst buffer.
          Use a callback to signal completion."

What an embedded engineer knows (not in prompt):
  - Buffer must be cache-line aligned (__aligned(32))
  - Cache flush before DMA start
  - Cache invalidate after DMA complete
  - Completion flag must be volatile (shared with ISR)
  - Flag checked AFTER synchronization, not before

Measured impact: Explicit prompts ("use volatile") pass at ~95%. Implicit prompts (derive from domain knowledge) pass at ~60%. This 35%p gap means current benchmarks overestimate LLM capability in embedded domains.

---

Leaderboard

n=3 aggregate pass@1 means:

Model	pass@1 (n=3 mean)	95% CI	Stability	Weakest Category	Strongest	Cases
Sonnet 5	67.0%	[63.7%, 70.2%]	82.1%	dma (31%), isr-concurrency (41%), threading (49%)	boot, adc, pwm (100%)	263 (2026-07-19)
Sonnet 4.6	68.0%	[64.4%, 71.3%]	87.1%	isr-concurrency (23%), dma (31%), threading (33%)	adc, device-tree, pwm (100%)	233 (2026-04-12)
Haiku 4.5	56.9%	[53.2%, 60.6%]	73.0%	dma (8%), isr-concurrency (38%), memory-opt (33%)	boot, device-tree, pwm (100%)	233 (2026-04-12)

Sonnet 5 vs Sonnet 4.6 — no measurable improvement. On the 232 cases common to both runs (isolating model change from the case-set growth), majority-vote pass@1 is 68.1% → 67.2% = −0.9%p: statistically tied, inside 4.6's own 66.1–70.4% run range. Sonnet 5 is the most stable model measured (stdev 0.29%p) but not more capable on embedded firmware; the weakest categories (dma, isr-concurrency, threading) are unchanged.

Sonnet vs Haiku: 11.1%p overall (CIs don't overlap — statistically significant).

See detailed comparison: docs/BENCHMARK-COMPARISON-2026-04-05.md (§10 = Sonnet 5) · delta report: docs/BENCHMARK-DELTA-sonnet5-vs-sonnet46.md See analysis & conclusions: docs/LLM-EMBEDDED-CONSIDERATIONS.md

Category Heatmap

All 23 categories (Sonnet 4.6 vs Haiku, 233 cases, n=3 run, 2026-04-12). Sonnet 5's per-category profile tracks 4.6 closely — same weakest three (dma, isr-concurrency, threading):

Category          Sonnet   Haiku    Gap      What it tests
----------------  -------  -------  -------  --------------------------
adc               100%      50%    +50%p     ADC read, sampling patterns
ble                82%      45%    +37%p     BLE stack API, connection mgmt
boot               90%     100%    -10%p     Boot sequence + Kconfig
device-tree       100%     100%       0      DT syntax, node references
dma                31%       8%    +23%p     Cache alignment, DMA lifecycle
gpio-basic         67%      83%    -16%p     GPIO config, device_ready() checks
isr-concurrency    23%      38%    -15%p     ISR safety, volatile, barriers
kconfig            90%      60%    +30%p     Build config generation
linux-driver       70%      70%       0      Kernel module, syscall interfaces
memory-opt         67%      33%    +34%p     Memory domains, slab allocators
networking         75%      75%       0      Socket lifecycle, error paths
ota                67%      58%     +9%p     OTA update lifecycle
power-mgmt         75%      67%     +8%p     Sleep modes, power domains
pwm               100%     100%       0      PWM duty cycle, device binding (n=1)
security           50%      70%    -20%p     Crypto API, key management
sensor-driver      75%      67%     +8%p     Sensor API patterns
spi-i2c            79%      64%    +15%p     Bus protocol, transfer sequences
storage            54%      31%    +23%p     Flash lifecycle, NVS patterns
threading          33%      33%       0      Mutex ordering, thread safety
timer              83%      50%    +33%p     Timer callback safety
uart               33%      67%    -34%p     UART config, async TX/RX
watchdog           90%      60%    +30%p     WDT feed timing, reset handling
yocto              80%      70%    +10%p     Yocto recipe authoring

---

Quick Start

# Prerequisites: Python 3.12+, uv
pip install uv  # if not installed

# Clone and install
git clone https://github.com/Ecro/embedeval.git
cd embedeval
uv sync

# Run benchmark (L0 + L3 static checks, no Docker needed)
uv run embedeval run --model claude-code://sonnet --cases cases/

# View results
cat results/LEADERBOARD.md

LLM Connection Modes

Mode	Example	Requirement
Claude Code (subscription)	--model claude-code://sonnet	Claude Code CLI installed
LiteLLM (API key)	--model anthropic/claude-sonnet-4-20250514	ANTHROPIC_API_KEY env var
Mock (testing)	--model mock	None

Common Commands

# Filter by category or difficulty
uv run embedeval run --model claude-code://sonnet --cases cases/ -c isr-concurrency
uv run embedeval run --model claude-code://sonnet --cases cases/ -d hard

# Multiple attempts for pass@k
uv run embedeval run --model claude-code://sonnet --cases cases/ --attempts 5

# With compiler feedback (self-correction measurement)
uv run embedeval run --model claude-code://sonnet --cases cases/ --feedback-rounds 3

# Multi-turn agent mode
uv run embedeval agent claude-code://sonnet --cases cases/ --max-turns 5

# Bug fix scenario (LLM diagnoses + fixes seeded bugs)
uv run embedeval run --model claude-code://sonnet --cases cases/ --scenario bugfix

# Temporal filtering (contamination prevention)
uv run embedeval run --model claude-code://sonnet --cases cases/ --after-date 2026-01-01

# Include private held-out cases (separate repo)
uv run embedeval run --model claude-code://sonnet \
    --cases cases/ --private-cases ../embedeval-private/cases/ --include-private

# Only retest cases changed since last run
uv run embedeval run --model claude-code://sonnet --cases cases/ --retest-only

# Validate all reference solutions pass
uv run embedeval validate --cases cases/

# List cases with metadata
uv run embedeval list --cases cases/

# Prompt sensitivity analysis
uv run embedeval sensitivity claude-code://sonnet --sample 30 --variants 3

# Generate safety guide from results
uv run embedeval guide --results results/

---

5-Layer Evaluation Architecture

Each case is evaluated through five progressive layers. Failure at any layer halts evaluation.

  Generated Code
       │
       ▼
  ┌─────────────┐
  │ L0 Static   │  checks/static.py — includes, CONFIG symbols, ISR signatures
  └──────┬──────┘
         ▼
  ┌─────────────┐
  │ L1 Compile  │  west build / idf.py / arm-gcc (Docker or local, skippable)
  └──────┬──────┘
         ▼
  ┌─────────────┐
  │ L2 Runtime  │  native_sim execution with 10s timeout + output validation
  └──────┬──────┘
         ▼
  ┌─────────────┐
  │ L3 Heuristic│  checks/behavior.py — domain-specific pattern analysis
  └──────┬──────┘
         ▼
  ┌─────────────┐
  │ L4 Mutation │  checks/negatives.py — meta-verification (30 cases)
  └─────────────┘

Layer	Method	What it catches	Docker needed
L0	Static pattern matching	Missing headers, wrong CONFIG, bad ISR signatures	No
L1	SDK compilation	Syntax errors, undefined symbols, type mismatches	Yes (or ZEPHYR_BASE)
L2	QEMU/native_sim execution	Segfaults, deadlocks, wrong output	Yes
L3	Domain heuristic checks	Missing volatile, wrong lock order, no error cleanup	No
L4	Mutation testing	Validates that L0/L3 checks themselves are sound	No

Default mode (no Docker): L0 + L3 provide strong discriminative power. L1/L2 auto-skip when EMBEDEVAL_ENABLE_BUILD is unset.

See METHODOLOGY.md for detailed architecture diagrams and implementation details.

---

24 Categories, 6 Platforms

Platform Coverage

Platform	Cases	Build System	Evaluation
Zephyr RTOS (native_sim)	145	west build	Full (L0-L4)
Zephyr RTOS (qemu_arm)	8	west build	L0-L3
ESP-IDF	5	idf.py build	L0-L1, L3
STM32 HAL + FreeRTOS	5	arm-none-eabi-gcc	L0-L1, L3
Linux kernel	8	kbuild	L0, L3
Yocto/Embedded Linux	8	bitbake	L0, L3

Categories

Peripheral & Communication: gpio-basic, uart, adc, pwm, spi-i2c, dma, ble, networking

Concurrency & Timing: isr-concurrency, threading, timer, watchdog

System Configuration: kconfig, device-tree, boot, ota, power-mgmt

Safety & Resources: security, storage, sensor-driver, memory-opt

Platform-Specific: yocto, linux-driver

---

Scoring

Metrics

Metric	Description
pass@1	First-attempt accuracy (primary metric)
pass@k	Unbiased estimator from Chen et al. (2021): 1 - C(n-c,k) / C(n,k)
95% CI	Wilson score confidence interval on pass@1
Embed Gap	EmbedEval pass@1 minus HumanEval pass@1 (negative = harder than general coding)

Aggregation Dimensions

Results are sliced by model, category (23), difficulty tier (easy/medium/hard), evaluation tier (sanity/core/challenge), and reasoning type (api_recall, rule_application, cross_domain, system_reasoning).

Report Outputs

A single benchmark run generates:

File	Content
results/LEADERBOARD.md	Model comparison, category heatmap, layer pass rates
results/<model>-results.json	Full machine-readable report (model name is slug-sanitized)
results/runs/<date>_<model>/report.md	Per-case failure analysis with patterns
results/SAFE_GUIDE.md	Risk-tier guidance for embedded engineers (auto-generated by run)
results/TEST_RESULTS.md	Per-case test status tracker (incremental run history)

---

Evaluation Modes

Mode	What it measures	CLI
Single-shot	Raw first-attempt accuracy	embedeval run
Multi-attempt	pass@k across N samples	embedeval run --attempts 5
Feedback	Self-correction on L0/L1 errors	embedeval run --feedback-rounds 3
Agent	Multi-turn iterative refinement	embedeval agent --max-turns 5
Bug fix	Diagnose + fix seeded mutations	embedeval run --scenario bugfix
Context Quality	Effect of team's CLAUDE.md / system prompt	embedeval run --context-pack ./CLAUDE.md + embedeval context-compare
Harmful-case triage	Distinguish real regressions from check brittleness	embedeval harmful-inspect --bare X --expert Y

Context Quality Mode quantifies how much your team's implicit-context files actually help the LLM and how much room is left to improve them. context-compare reports per-case effect (H/Hm/F/P) and token cost; harmful-inspect sub-classifies harmful cases by failure layer so you know whether to edit the pack (L1+ = likely real) or the static checks (L0 = likely brittleness). See docs/CONTEXT-QUALITY-MODE.md for the workflow and metric interpretation.

---

Case Structure

cases/isr-concurrency-003/
├── metadata.yaml           # id, category, difficulty, platform, reasoning_types, ...
├── prompt.md               # Task prompt (functional requirements, no safety hints)
├── reference/main.c        # Verified correct solution
├── src/main.c              # LLM-generated code goes here during evaluation
├── context/                # Additional context files (optional)
├── checks/
│   ├── static.py           # L0: required includes, struct layout, ISR signature
│   ├── behavior.py         # L3: volatile qualifiers, lock ordering, ISR safety
│   ├── expected_output.txt # L2: expected program output pattern (optional, 119/185)
│   └── negatives.py        # L4: mutation tests to validate checks (optional, 30/185)
├── CMakeLists.txt          # Zephyr build config
└── prj.conf                # Zephyr Kconfig

Design principles:

1. Self-contained — each case is a standalone Zephyr/ESP-IDF/STM32 project
2. Deterministic — all checks are regex/pattern-based, no LLM-as-judge
3. Implicit knowledge — prompts describe what to build, not how to make it safe
4. Reference verified — every case has a reference solution that passes all layers

---

Contamination Prevention

• 48 private cases in a separate repository — never exposed to LLM training data
• Temporal cutoff — --after-date filter for training data freshness analysis
• Content-hash tracking — --retest-only detects modified cases for efficient re-evaluation

---

Project Structure

embedeval/
├── src/embedeval/           # Core library (21 modules)
│   ├── cli.py               # Typer CLI entry point
│   ├── runner.py            # Case discovery, filtering, benchmark orchestration
│   ├── llm_client.py        # LiteLLM + claude-code:// + mock providers
│   ├── evaluator.py         # 5-layer evaluation pipeline
│   ├── scorer.py            # pass@k (unbiased) + Wilson 95% CI
│   ├── reporter.py          # JSON, Markdown, failure analysis, safe guide
│   ├── models.py            # Pydantic models (EvalResult, BenchmarkReport, ...)
│   ├── check_utils.py       # Scope-aware check utilities
│   ├── agent.py             # Multi-turn agent evaluation
│   ├── bugfix.py            # Bug fix scenario (mutation-based)
│   ├── sensitivity.py       # Prompt sensitivity analysis
│   ├── difficulty.py        # IRT difficulty calibration
│   ├── ablation.py          # Layer contribution ablation study
│   ├── failure_taxonomy.py  # Automated failure classification (8 patterns)
│   ├── safety_guide.py      # Risk-tier safety guide generation
│   └── test_tracker.py      # Incremental retest tracking
├── cases/                   # 219 public test cases
├── tests/                   # 1277 pytest tests
├── docs/
│   ├── METHODOLOGY.md                    # Full benchmark methodology + architecture diagrams
│   ├── CONTRIBUTING.md                   # How to add new test cases
│   ├── LLM-EMBEDDED-CONSIDERATIONS.md    # Research insights + practical guidance
│   ├── LLM-EMBEDDED-FAILURE-FACTORS.md   # 42-factor failure taxonomy (6 categories)
│   ├── LLM-EMBEDDED-DEVELOPMENT-GUIDE.md # End-to-end workflow + knowledge base
│   └── BENCHMARK-COMPARISON-2026-04-05.md # Haiku vs Sonnet detailed comparison
├── scripts/
│   ├── sync_docs.py              # Auto-sync README/METHODOLOGY counts (mandatory at wrapup)
│   └── ...                       # Other analysis and verification scripts
├── external_benchmarks.yaml # HumanEval/SWE-bench reference scores
├── Dockerfile               # Zephyr SDK build environment
├── Dockerfile.esp           # ESP-IDF build environment
├── Dockerfile.stm32         # STM32 HAL build environment
└── .github/workflows/       # CI + benchmark dispatch + case validation

---

Development

uv run pytest                               # Run all tests
uv run ruff check src/ tests/              # Lint
uv run ruff format src/ tests/             # Format
uv run mypy src/                           # Type check
uv run embedeval validate --cases cases/        # Validate reference solutions
uv run embedeval validate-metadata --cases cases/  # Validate metadata consistency
uv run embedeval categories --cases cases/     # Show case counts per category
uv run python scripts/sync_docs.py         # Sync README/docs counts (run before commit)

---

Key Research Findings

Documented in LLM-EMBEDDED-CONSIDERATIONS.md:

1. Implicit vs Explicit Gap — 35%p pass rate drop when removing safety hints from prompts
2. 4-Level Implicit Knowledge Model — C language → RTOS patterns → Hardware constraints → System safety
3. Failure Distribution — Sonnet: 31% L2 + 31% L3. Haiku: 43% L0 + 24% L1. Bigger models fail later (safety, not syntax).
4. General vs Embedded — 56% of failures are general SW problems (error paths), 44% are embedded-specific (HW constraints)
5. Model Size Sensitivity — Category-level gaps reach up to 50%p (e.g., adc); overall n=3 Sonnet–Haiku gap is 11.1%p (CIs non-overlapping)
6. 6 LLM Failure Patterns — happy path bias, semantic mismatch, resource imbalance, order violation, cross-platform hallucination, missing safety guards
7. "3AM Paranoia" Layer — 8 categories of field knowledge (timer overflow, flash wear, sensor plausibility, radio corruption) that LLMs structurally cannot learn

---

Known Limitations

• Platform bias — 81% Zephyr, with ESP-IDF and STM32 at 5 cases each
• L3 precision — Static heuristic checks are regex-based; true semantic verification needs L1/L2
• Single-file scope — Cases test single-file code generation, not multi-file project scaffolding
• Difficulty calibration — Assigned labels may not match empirical difficulty (IRT calibration module in src/embedeval/difficulty.py)

See METHODOLOGY.md for our complete self-assessment.

---

Comparison with Related Work

Dimension	HumanEval	SWE-bench	EmbedAgent (ICSE'26)	EmbedEval
Domain	General Python	Python SWE	Arduino/ESP32/RPi	Embedded (Zephyr/ESP-IDF/STM32/FreeRTOS/Linux/Yocto)
Cases	164	2,294	126	233
Platforms	1	1	3	6
Verification	assert	pytest	Wokwi sim	5-layer pipeline
Contamination	None	PR-based	HW combos	Separate private repo + temporal
Scoring	pass@k	% resolved	pass@1	pass@k + 95% CI + Embed Gap
Unique	—	—	Circuit design	Implicit Knowledge Gap

vs EmbedAgent (ICSE'26). Both target embedded LLM evaluation, but the focus is different. EmbedAgent measures cross-platform programming on hobbyist boards (Arduino / ESP32 / Raspberry Pi Pico) with Wokwi circuit simulation and bundles Programmer / Architect / Integrator role tasks. EmbedEval measures production embedded firmware on Zephyr RTOS, ESP-IDF, STM32 HAL, FreeRTOS, Linux kernel drivers, and Yocto, deliberately withholds safety hints from prompts (the "implicit knowledge gap"), and verifies through five layers including a mutation-testing meta-layer. They are complementary: EmbedAgent for breadth across hobbyist hardware, EmbedEval for depth on production RTOS/driver patterns.

---

Roadmap

See ROADMAP.md for the v0.2/v0.3 plan. Open an issue to influence direction. Briefly: v0.2 broadens model coverage and adds FreeRTOS / Linux driver cases; v0.3 adds multi-file scaffolding and cross-platform migration; v1.0 freezes the schema.

---

Contributing

See CONTRIBUTING.md for case authoring. In brief:

• New case: create cases/<category>-<NNN>/ with metadata.yaml, prompt.md, reference/main.c, plus checks/static.py and checks/behavior.py. Verify with uv run embedeval validate --cases cases/ -c <category>.
• New model: run uv run embedeval run --model <litellm-id> --cases cases/ for n=3, then PR results/runs/<date>_<model>/ along with an updated LEADERBOARD.md.
• Methodology critique: open an issue using the methodology-question template.
• Interop contract: see HILOOP-HANDOFF.md — producer-side schemas, stability tiers, and breaking-change protocol for every artifact consumed by downstream tools (Hiloop transpile, evidence injection).

Issue templates: model evaluation request, case contribution, methodology discussion (under .github/ISSUE_TEMPLATE/).

---

Citation

If EmbedEval is useful for your work, please cite:

@misc{embedeval2026,
  title  = {EmbedEval: A Benchmark for LLM-Generated Embedded Firmware},
  author = {{EmbedEval Contributors}},
  year   = {2026},
  url    = {https://github.com/Ecro/embedeval},
  note   = {Open benchmark with 267 cases across Zephyr, ESP-IDF, STM32 HAL,
            FreeRTOS, Linux kernel drivers, and Yocto. Measures the implicit
            knowledge gap in LLM-generated embedded code.}
}

---

License

Apache 2.0 — see LICENSE.