Validation status
September 19, 2026 · View on GitHub
This file records observed results, not forecast capability.
- TypeSafe adapter: mocked contract/error/cancellation tests passed. The Jev ledger records attempted requests, received responses, actual reported models, input/output tokens, and unknown usage for incomplete calls. Missing usage is not treated as zero; complete totals stay unavailable when any usage is unknown. Duplicate candidate data was removed from requests; live decision quality and provider-reported savings remain unmeasured.
- Automated suite: both TypeScript projects and 129 tests pass across provider contracts, planning, approval, cancellation, stale snapshots, completion confirmation, privacy boundaries, usage receipts, limits, compact observations, literal preparation, reviewed fills, field-map parsing, voice process lifecycle, and actual MCP stdio client/server communication. This includes 119 core/desktop tests and ten field-map parser tests. Reproduce with
npm run typecheckandnpm test; results below identify what was tested against real native apps. - macOS: Accessibility and dictation helpers compiled with Swift 6 and strict concurrency. A real disposable AppKit fixture passed eight native checks, including compact inspection and preparation without mutation, discovery, app isolation, consumed snapshots, exact text fill, button effects, capture/OCR geometry, and fixture exit. See native evidence.
- Reviewed fills: fake-driver tests pass for immutable approval, all-or-nothing preparation, duplicate/protected/OCR targets, missing or changed native identities, process/window/document/form drift, expiry, duplicate approval, cancellation, uncertain writes, literal readback mismatch, and final detection of changes to earlier fields. A real macOS six-field AppKit fixture passed seven authored cases: blank, prefilled, Unicode, changed remaining value, disappeared field, duplicate field, and rejected write. Independent fixture state confirmed exact success or the expected stopped partial result, with no submit action or provider request. Two initial failed suites are retained: one exposed the menu-bounds validation defect, and one exposed a no-op fixture setter; both were corrected before the passing rerun. See batch evidence and reproduction. This is macOS-only fixture evidence, with no Jev baseline, billing comparison, or Windows batch runtime result.
- Dictation: fake child-process tests cover startup and stop, Unicode output, cancellation, process reservation through close, end-event delivery, stale results, malformed/oversized output, broken pipes, spawn failures, and timeouts. These tests do not use a microphone. A read-only macOS capability probe reported an available recognizer with on-device support for
en_US; it did not start capture or request authorization. Live speech recognition, other languages, and Windows speech runtime behavior remain unverified. - Windows: a real WinForms fixture passed UIA discovery, exact ValuePattern text entry, InvokePattern button effects, selected-window capture, app isolation, stale/consumed snapshot rejection, and access revocation on a Windows runner. Workflow, actual result. This does not establish arbitrary-app, OCR-click, or keyboard-input compatibility.
- Local OCR: recognized text from a local UI image with bundled language data, without a remote OCR service.
- Desktop UI (2026-09-18): the actual app loaded its redesigned renderer and retained the existing connection. The production renderer/preload also passed isolated Electron fixture flows at 1280×820 and 900×660: keyboard app selection, four-app limit, search, focus trapping/restoration, consent, Hybrid setup, loading, provider/runtime errors, approval, separate completion verification, Stop, fresh recovery, and independent scrolling. Long review overflow and initial modal focus were found and fixed. These fixture transitions do not establish live model-driven task success or Windows visual QA. See the design assessment and reproducible fixture.
- Alpha.4 renderer: the production renderer passed browser mock-bridge checks for exact six-field review and receipts, mismatch recovery, held execution and Stop, automatic permission-return refresh, synthetic dictation completion/cancellation, keyboard focus, and scrolling at 900×660. The installed Electron app rendered, but native Computer Use input was unreliable; those attempted interactions are not counted as passed. Mock permission and voice events do not establish OS authorization or speech recognition. See recorded checks.
- Installed Alpha.4 follow-up (2026-09-19): after the user removed and re-added the installed app's Accessibility entry, native access worked and the provider connection was restored. A real guided task executed one approved action, then stopped on TypeSafe response validation. Task completion was not established; the exact rejected response was not retained, so its cause remains unresolved.
- TypeSafe failure investigation: three raw network responses from public synthetic tasks in the official console had probability totals of 1. An earlier copied console JSON response totaled 0.99, but its matching raw response is unavailable. This does not prove an API rounding defect, and the strict validation rules remain unchanged. The source update adds fixed validation codes and bounded numeric summaries to errors and usage receipts, with tests excluding provider text, option IDs, task contents, and credentials. These diagnostics are not installed in the user's Alpha.4 app and do not retroactively identify its failure.
- Website: rendered at 1280px and 390px with no horizontal overflow, meaningful download/developer links, working help/Escape focus restoration, and no relevant console errors.
- Developer context: eight synthetic cases measure exact UTF-8 bytes and
o200k_basetokenizer counts with schema overhead, omissions, negative cases, and stale/error traces disclosed. These are not billed savings or completed desktop tasks. See results and reproduction. - Distribution: earlier Mac arm64, Mac Intel, and Windows x64 installers built and were published as explicitly unsigned prereleases. A later local check found invalid bundle signature integrity in the Alpha.3 Mac app. Alpha.4 packaging adds ad-hoc integrity signatures for the app and native helpers, with
codesign --verify --deep --strictrequired in release CI. This is not Developer ID signing or notarization. The installed Mac app's enabled Accessibility entry initially referenced an older build identity; a fresh helper still reported denied access. The user's later removal and re-addition restored access, as recorded above. Windows publisher signing is not configured. The public project page was verified in the in-app browser; no production installer or benchmark-backed SOTA claim is made. - Packaging: local OCR also passed from inside the packaged application's ASAR/resources layout, with bundled language data.
Update this report with actual build, native smoke, provider-backed run, and release results as they are completed.