Testing Guide

July 10, 2026 · View on GitHub

This project uses layered verification across two test ecosystems:

  • Unit tests (vitest): parser/model/renderer behavior in isolation.
  • Browser package tests (Playwright): built standalone entry, ECharts registration matrix, and isolated PDF.js Worker rendering in Chromium.
  • E2E tests (pytest + Playwright): structural validation, visual comparison against PowerPoint PDF output, and baseline-driven shape/SmartArt regression.

Unit Tests

pnpm test              # Run all unit tests
pnpm test -- --watch   # Watch mode
pnpm test:coverage     # With v8 coverage report → coverage/

Browser Package Tests

pnpm build
pnpm exec playwright install chromium
pnpm test:browser

These tests load the built standalone browser artifact with a tracked PPTX, initialize every renderer-supported ECharts series through the modular runtime, verify computed overflow behavior for all text-axis combinations, and execute the actual outer-Worker plus PDF.js-worker path. CI runs the PDF test against both supported PDF.js major lines; Node-only imports are not accepted as browser compatibility evidence.

Coverage areas:

  • Parser safety and correctness (ZipParser, relationship parsing, EMU/angle/PCT unit conversion)
  • Shape geometry (preset shape path tests in test/unit/shapes/presets.test.ts)
  • Renderer behavior (batching, windowed mounting, hyperlink safety)
  • Color utilities (HSL/RGB conversion, lumMod/lumOff/tint/shade modifiers)

E2E Tests

Setup

cd test/e2e
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
playwright install chromium

Running

# Start dev servers first (from project root)
pnpm dev:e2e    # Vite :5173 + Python API :8080

# Then run tests
cd test/e2e
pytest -v                       # All E2E layers
pytest test_structural.py -v    # Layer 1: model structure vs ground truth
pytest test_visual.py -v        # Layer 2: HTML screenshots vs PDF (SSIM >= 0.65/slide)
pytest test_regression.py -v    # Layer 3: scores vs stored baselines

The default corpus is testdata/cases. Windows-generated oracle cases live under testdata/windows-cases and can be selected without changing test code:

pytest -v --testdata-source=windows    # Windows-generated cases only
pytest -v --testdata-source=all        # Default + Windows-generated cases

# Equivalent environment override:
PPTX_E2E_TESTDATA_SOURCE=windows pytest test_visual.py -v

Windows case ids are encoded with a win__ prefix during pytest parametrization so baseline/report filenames stay distinct from default cases.

Test Layers

LayerFileWhat it checks
Structuraltest_structural.pyWord coverage, shape count/position from exported model
Visualtest_visual.pySSIM between HTML screenshots and PDF pages
Regressiontest_regression.pyNo score drops > 0.02 SSIM or 2% text coverage vs baselines
Baseline generationtest_oracle_case_generation.pyBaseline case JSON validity and generation pipeline

Baseline-Driven Shape/SmartArt Evaluation

The baseline pipeline is the primary tool for expanding and verifying rendering quality. It generates isolated test cases (one shape or SmartArt layout per slide), renders them, and compares against PowerPoint output.

Running Baseline Evaluation

cd test/e2e
source .venv/bin/activate

# Evaluate all preset shapes (by MsoAutoShapeType ID)
.venv/bin/python3 scripts/run_all_shapes_eval.py --shape-id-min 1 --shape-id-max 200

# Evaluate all SmartArt layouts
.venv/bin/python3 scripts/run_all_shapes_eval.py --smartart-cases-dir oracle/cases-full

# Both at once
.venv/bin/python3 scripts/run_all_shapes_eval.py \
  --shape-id-min 1 --shape-id-max 200 \
  --smartart-cases-dir oracle/cases-full

# Find failures
grep "False" reports/oracle-failures/all-shapes-eval.csv | sort -t, -k2 -n

Output files:

  • reports/oracle-failures/all-shapes-eval.json — full metrics per case
  • reports/oracle-failures/all-shapes-eval.csv — case,ssim,color_hist_corr,fg_iou_tolerant,chamfer_score,fg_iou,passed,needs_review
  • reports/<case>_slide0_{pdf,html,diff}.png — visual comparison images

Generating Ground Truth

Ground truth requires Microsoft PowerPoint (macOS or Windows). A VBA macro creates PPTX files and exports PDFs:

cd test/e2e
.venv/bin/python scripts/one_shot_full_ground_truth.py \
  --macro-host testdata/pptx-macro-host.pptm \
  --cases-dir oracle/cases-full \
  --testdata-dir testdata \
  --shape-id-min 1 \
  --shape-id-max 500

This generates/reuses ground truth for all SmartArt layouts available on the local PowerPoint build plus the specified shape ID range.

Manual Review

For cases that pass automated metrics but have visual nuances:

  1. Open /test/pages/e2e-compare.html
  2. Review side-by-side PDF vs HTML renders with SSIM scores
  3. Save verdicts per slide card
  4. Verdicts persist in reports/oracle-failures/manual-review.json

Visual Evaluation Metrics

The comparison pipeline uses a two-layer metric system. This design was arrived at empirically by testing many metrics against 300+ shape cases.

Pass/Fail Layer (automated)

These two metrics determine pass/fail. Chosen for zero false positives across all 452+ baseline cases:

MetricRangeThresholdWhat it catches
ssim0-1>= 0.95Structural errors: wrong geometry, missing elements, layout shifts
color_hist_corr-1 to 1>= 0.80Color errors: wrong scheme resolution, gradient bugs, tint/shade misapplication

Warning Layer (human review)

ConditionFlagPurpose
ssim < 0.99needsReview: trueFlags near-misses for human inspection without auto-failing

Diagnostic Layer (display only)

These metrics appear in the UI for reference but do not affect pass/fail:

MetricRangeDescription
fg_iou0-1Foreground pixel IoU (non-white pixel overlap)
fg_iou_tolerant0-1FG IoU with 1px morphological dilation
chamfer_score0-11 - normalized Chamfer Distance
mae0-1Mean Absolute Error per pixel

Why Only SSIM + Color Histogram?

  • fg_iou was removed from pass/fail — thin-stroke shapes (brackets, braces, arcs) get ~50% IoU drop from 1px anti-aliasing differences despite correct geometry.
  • chamfer_score was not promoted — its "dilution effect" masks localized errors when most of the shape is correct.
  • SSIM catches all bugs that any other metric catches, but color histogram adds sensitivity to pure-color errors that SSIM misses (e.g., wrong gradient stop colors on an otherwise correctly shaped element).

Previously evaluated but rejected: edge_iou (too noisy), fg_area_ratio (redundant), fg_centroid_distance (no observed failures), patch-based SSIM (can't detect < 1% area defects), LPIPS (heavy PyTorch dependency, marginal improvement).

Color Histogram Correlation Details

  • Computed over H, S, V channels independently (30 H bins, 32 S/V bins), then averaged
  • Only foreground pixels (gray < 245) are compared, ignoring white backgrounds
  • Sparse foreground (< 1.5% coverage) returns 1.0 to avoid anti-aliasing noise on thin-stroke shapes
  • Score of 1.0 = identical color distributions; >= 0.80 = pass threshold

Shape Fix Protocol (TDD Required)

For complex shape regressions (curved arrows, multi-segment geometry, 3D faces):

  1. Isolate one baseline case ID and one slide.
  2. Record baseline metrics (ssim, color_hist_corr, fg_iou, chamfer_score).
  3. Extract ground-truth geometry from ECMA-376 presetShapeDefinitions.xml before editing path code.
  4. Add a failing unit test for the specific mismatch.
  5. Implement minimal geometry patch.
  6. Verify: unit tests pass, baseline metrics do not regress, visual review confirms correctness.
  7. Only then mark the case fixed and move to the next.

Do not use blind parameter tuning. When topology or shape semantics are wrong, derive the fix from the OOXML spec.

Chart Fix Protocol

Chart rendering is validated at two levels:

  1. Add focused unit tests in test/unit/renderer/ChartRenderer.test.ts for OOXML semantics such as stacking mode, axis orientation, per-point data labels, chart color styles, legend data, and combo-axis wiring.
  2. Run targeted oracle comparisons through the E2E API or pytest. Use --testdata-source=windows for Windows-generated chart oracle cases, which include many Office chart defaults not present in lightweight generated decks.

Current chart 3D support is intentionally a 2D fallback for render continuity. Do not treat surface3DChart or 3D perspective/depth mismatches as fixed unless there is a real 3D rendering implementation and corresponding oracle coverage.

Test Pages

PagePurpose
/test/pages/index.htmlUpload preview with model search and lazy thumbnail navigation
/test/pages/render-slide.htmlSingle slide at native resolution
/test/pages/e2e-compare.htmlE2E dashboard with SSIM scores
/test/pages/compare-renderers.htmlRenderer-to-renderer comparison
/test/pages/export.htmlModel JSON tree viewer

URL params for list rendering performance: listStrategy, listBatchSize, windowedInitialSlides, windowedOverscanViewport.

Contribution Requirement

For behavior changes:

  • Add or update at least one relevant test.
  • Keep existing test suite green before opening PR.
  • For shape geometry changes, run the baseline evaluation on affected cases.