Eval-core Runner

eval-core is the mdpr-skill evaluation runner. It compares MDPR output with and without accepted skill hints or approved packs, while preserving MDPR as the deterministic runtime.

Flow

Markdown source
  -> MDPR baseline build
  -> sourceSha256 and manifest metrics
  -> schema/boundary checked agent-hint.json
  -> optional approved mdpr.pack.json
  -> MDPR guided build with hints and/or pack
  -> review-core coherence and visual policy review on both outputs
  -> sufficient-context check for guided review evidence
  -> regression gate
  -> deck coherence evidence and design decision trace
  -> mdpr-skill-eval-v1 report

npm run test:eval-core:e2e runs this flow against the actual local MDPR CLI from .cache/mdpr/packages/cli/dist/index.js when that checkout is available. It builds tiny Markdown decks twice, verifies that the guided build accepts a schema-valid hint, verifies that a guided build can apply an approved pack without any agent hint content, checks the emitted eval report, and includes a review regression fixture that proves a successful MDPR CLI build can still fail the review gate when guided artifacts introduce non-editable primary objects.

Gates

schemaSync verifies that the hint manifest is mdpr-agent-hint-v1, was generated by mdpr-skill, has a generatedAt timestamp, contains a hints array, and matches the baseline sourceSha256. It also checks synced MDPR source-of-truth copies for bridge and design rail schemas, including PPTX object maps, selection context, PPT selection, pack and override candidates, theme candidates, and HTML design analysis artifacts.

boundary verifies that hints do not contain final decision fields such as coordinates, colors, typography, z-order, arrows, geometry, exact icon paths, component variants, recipes, or renderer object IDs.

Approved packs are evaluated outside the hint rail. guidedPackPath is passed to MDPR as --pack, while eval-core emits an empty schema-valid hint manifest when no agent hint is supplied. This keeps theme/component token comparison separate from agent final-decision fields.

regression compares baseline and guided metrics. A guided build fails the gate when it increases overflow, coherence warnings, visual errors, text clipping risk, contrast failures, or connector warnings. It also checks thresholded build-time, slide-count, output-size, and minimum-font regressions. MDPR command failures are preserved as adapter failures, not converted into quality regressions.

review compares baseline and guided review-core summaries. A guided build fails the gate when review errors increase, review warnings increase, review findings contain forbidden final-decision fields, or findings lack evidence. The review pass can flag detached captions, orphan evidence, claimless evidence slides, section rhythm drift, raw hex leakage, mixed visual scales, effect budget overuse, accent overuse, and non-editable primary PPTX objects.

sufficientContext checks the guided review findings using an Agentic RAG-style grounding rule. Non-empty evidence is not enough: material review findings must point to concrete MDPR context such as source slide ids, block ids, layout slide ids, region ids, PPTX object-map entries, screenshots, selection paths, design source declarations, diagram ids, or manifest metrics. Weak evidence such as a plain explanation string fails this gate even when the review gate has already counted the finding.

The emitted report also records evidenceRetrieval.baseline and evidenceRetrieval.skillGuided. Each plan contains a fixed corpus catalog and one route per review finding. The route states which corpus ids support the finding, which evidence keys were covered, whether any required support was missing, and which feedbackQueries should be used if a route is missing. When eval has a local MDPR output directory, missing routes also include artifactAttempts: deterministic checks for likely local files such as mdpresent-manifest.json, presentation-ir.json, layout-ir.json, pptx-object-map.json, contact sheets, selection contexts, design analyses, or diagram metrics. A found artifact keeps the route auditable, but it does not turn the route into covered until the finding carries concrete evidence keys. Corpus ids are deterministic report metadata:

• mdpr-manifest
• presentation-ir
• layout-ir
• pptx-object-map
• rendered-artifact
• selection-context
• design-analysis
• diagram-metrics

This is not an agent authority path. It is an audit trail for why sufficientContext passed or failed. Feedback queries are follow-up validation instructions such as inspecting the layout IR, PPTX object map, rendered artifact, design analysis, or manifest metrics. They do not authorize an agent to create coordinates, styles, recipes, or renderer objects.

The report also records deckCoherence and designDecisionTrace. deckCoherence is a deck-level evidence rollup for baseline and guided runs: coherence warnings, overflow, text clipping risk, contrast failures, connector warnings, review finding counts, and missing-evidence counts. Its boundary is evidence-only-not-mdpr-pass-fail; it explains what changed, but it does not override MDPR validation or renderer decisions.

designDecisionTrace records the order of evidence handoff: source, baseline build, guided input, guided build, review, evidence routing, and gate summary. Each step names the owner (mdpr or mdpr-skill) and artifact references. Its boundary is trace-only-not-renderer-instructions, so it must not contain coordinates, renderer object ids, final layouts, theme bindings, or PPTX/PDF pass/fail claims.

Report

The emitted report uses schemaVersion: "mdpr-skill-eval-v1" and records:

• summary.overallStatus
• baseline and guided manifest paths
• baseline and guided run artifacts
• optional baseline and guided profile metadata copied from MDPR manifests
• baseline and guided review summaries
• baseline and guided evidence retrieval route plans
• deck-level coherence evidence
• design decision trace steps
• schemaSync, boundary, regression, review, and sufficientContext gate findings
• thresholds used for the regression decision
• accepted hint path
• optional baseline and guided pack paths

The report is evidence for review. It is not a layout or style authority.