ODS Validation Matrix

Last updated: 2026-05-25

This page describes the layered coverage used to validate ODS between changes. It is intentionally sanitized: it publishes hardware classes, operating systems, GPU paths, and test phases without private hostnames, LAN addresses, usernames, or filesystem paths.

Layered Test Surface

ODS uses four standing validation layers:

Layer	Where it runs	Coverage	What it proves
CI matrix	GitHub Actions containers	Installer, shell, Python, PowerShell, dashboard, environment, and distro smoke checks	Fast pull-request safety for syntax, parsing, and cheap regressions
Zero-prereq bootstrap lab	Release harness distro containers	Bare Ubuntu, Debian, Fedora, Rocky, Arch, and openSUSE images with no assumed Git, jq, Python, Docker, or Compose	The public curl bootstrap path can provision its own prerequisites on clean Linux systems
Fleet distro lab	Private lab Docker containers + Incus VMs	10 container distros plus VMs for Ubuntu, Fedora, Rocky, Arch, and openSUSE	Fast distro breadth plus real systemd, network, Docker daemon, Compose, and installer dry-run with Docker enabled
Real hardware fleet	Private local machines	Linux NVIDIA, Linux AMD, Linux ARM NVIDIA, macOS Apple Silicon, and optional Windows laptop targets	Fresh installs, GPU runtime, dashboard, Hermes, UI, lifecycle, ODS Talk, and agent capabilities

Containers are breadth, not user-experience proof. Incus VMs add systemd, kernel, and Docker-daemon realism, but they still do not prove GPU runtime. Physical fleet machines remain the release gate for accelerator and product behavior.

Hardware Surface

Test surface	OS family	Architecture	Accelerator path	Memory class	Fleet role
Linux NVIDIA workstation	Ubuntu 24.04	x86_64	High-memory CUDA GPUs	90 GB+ VRAM per GPU	Primary CUDA install, dashboard, UI, and capability target
Linux AMD unified-memory workstation	Ubuntu 24.04	x86_64	AMD Strix Halo / ROCm-Lemonade path	120 GB+ unified	Primary AMD install/runtime validation target
Linux NVIDIA unified-memory appliance	NVIDIA Ubuntu derivative	aarch64	Grace Blackwell / CUDA path	120 GB+ unified	ARM Linux + NVIDIA appliance validation target
macOS constrained Apple Silicon	macOS	arm64	Native Metal inference + Docker services	16 GB unified	Smoke gate and tight-memory macOS validation target
macOS high-memory Apple Silicon	macOS	arm64	Native Metal inference + Docker services	120 GB+ unified	Large-model macOS validation target
Windows hybrid GPU laptop	Windows 11 + Docker Desktop/WSL2	x86_64	NVIDIA mobile GPU plus Intel integrated GPU	32 GB+ system RAM	Windows installer, Docker Desktop, WSL2, and mobile-GPU validation target when enabled for a run

This standing hardware fleet is the repeatable release surface for GPU and product behavior: it can run in parallel whenever installer, bootstrap, dashboard, agent, model, or extension code changes. The CI matrix, zero-prereq bootstrap lab, and distro lab add repeatable distro evidence between hardware fleet runs. Community and volunteer testers add broader coverage on other GPUs, distros, operating-system versions, storage layouts, and network environments, but those reports are complementary evidence rather than the always-on release gate.

Fleet Phases

The private fleet harness runs these phases and records structured artifacts for each host where the phase is applicable.

Phase	What it proves	Normal cadence
Zero-prereq bootstrap	The public installer can bootstrap clean distro containers without preinstalled developer tools or Docker	Every release-grade run
Regression replay	Previously fixed fleet bugs have not returned	Every full fleet run
Smoke gate	A constrained Apple Silicon target can fresh-install and pass core health before the larger fleet starts	Every full fleet run
Preflight	OS, RAM, disk, Docker, firewall, port conflicts, prior install state	Every install run
Fresh install	The public bootstrap path can nuke prior state and install non-interactively	Every full fleet run
Core verify	Dashboard API, dashboard UI, llama-server models/chat, and Hermes proxy are reachable	Every post-install run
Cloud-mode contracts	Cloud and hybrid modes do not accidentally require local llama-server and still render required configs	Every release-grade run
Dashboard API flows	Model listing, model download/switch, and extension install state transitions	Every post-install run
Hermes auth/chat	Magic-link session auth, gated Hermes access, and seed echo through the agent path	Every post-install run
Browser UI	Dashboard navigation, model/extension surfaces, and Open WebUI model proxy behavior	Default UI target every run; scheduled wider UI sweeps
Capability probes	Chat coherence, web search, file write/read, code execution, skills list, loaded-model identity, context, and ODS Talk/owner-portal surfaces where enabled	Every post-install run, with LLM probes deferred while bootstrap is still active
Lifecycle	Idempotent reinstall, ods restart, and ods doctor after state changes	Every release-grade run; optional for lighter development runs
Release confidence report	The run is summarized into product, capability, lifecycle, and user-facing gates	Every release-grade run

--phase all is intended for a faster full-product development sweep. The private release-grade path adds zero-prereq bootstrap and lifecycle gates so a green release run means more than "fresh install worked once."

Run the release-grade fleet after operational code changes: installer phases, bootstrap, compose stack generation, service wiring, dashboard/API behavior, Hermes, model routing, GPU detection, lifecycle commands, or anything else that can affect a user's install or running stack. Docs-only and cosmetic changes can usually rely on CI and focused documentation checks.

Release Confidence Gates

Release-grade runs render a machine-readable confidence summary plus a human report. The top-level gates are:

Gate	Pass signal
Zero-prereq bootstrap	Bare distro containers can fetch the public installer and provision prerequisites
Install Green	Every enabled hardware host completed a fresh public-path install
Product Green	Core services, cloud contracts, dashboard flows, Hermes auth/chat, and UI checks passed
Capability Green	Full-model capability probes passed, or were explicitly deferred/skipped for documented reasons
Lifecycle Green	Idempotent reinstall, restart, and doctor checks passed on enabled hosts
User Green	The combined release-readiness gate: zero-prereq, install, product, capability, and lifecycle evidence are all clean or explicitly accounted for

This is why a green release fleet pass is stronger than ordinary service health. It exercises the user-visible setup path, first-run behavior, post-install actions, real agent work, and lifecycle recovery after state has changed.

Evidence Receipts

Release notes should cite sanitized evidence from the current candidate, not a private run directory. A useful receipt includes:

• the ODS commit, tag, or release candidate;
• the run date;
• sanitized hardware classes covered;
• distro lab breadth;
• regression replay result;
• install, verify, dashboard, Hermes, UI, capability, and lifecycle summaries;
• skipped, deferred, blocked-by-environment, or not-run phases;
• any known gaps that should not be read as supported behavior.

Private artifacts keep detailed logs, JSON events, and host-specific evidence inside the lab. Public docs should quote the sanitized summary only, especially when a run includes local hostnames, LAN addresses, usernames, or filesystem paths.

Treat Windows evidence as release-relevant only when the Windows target produces preflight, install, verify, dashboard, and UI artifacts for the candidate being claimed. A supported platform can have installer and code support even when it is not included in every default private release-fleet run; release notes should say which hardware classes actually ran.

What This Proves

• Installer OS and package-manager logic is exercised across the major Linux package-manager families.
• The public curl bootstrap path is tested from clean Linux images that do not assume developer tools or Docker are already present.
• Systemd, network, Docker daemon, Docker Compose, and installer dry-run behavior are exercised in disposable Incus VMs for the major Linux families.
• The installer is repeatedly exercised on real machines, not only CI containers and VMs.
• The release path covers heterogeneous GPU vendors, memory sizes, operating systems, and CPU architectures.
• The harness records environment state before install so firewall, Docker, disk, DNS, and port issues can be separated from product bugs.
• The user-facing path is tested beyond service liveness: dashboard actions, model switching, Hermes auth, agent capabilities, ODS Talk surfaces, and regression fixtures are part of the gate.
• Lifecycle behavior is part of release-grade confidence: reinstall, restart, and ods doctor must recover cleanly after state has changed.

What This Does Not Claim

• Every Linux distribution is exhaustively installed on real hardware for every change. CI containers and the distro lab cover broad distro logic and systemd/Docker VM behavior; the physical fleet covers the high-value hardware paths.
• The Incus VM matrix is not GPU validation. GPU runtime claims require real NVIDIA, AMD, Intel, or Apple hardware evidence.
• OS and distro rotation is periodic because reprovisioning real machines is intentionally slower than running the standing fleet. Release notes should call out any rotated distro or OS image that was included for that candidate.
• Intel Arc is still experimental unless a release cites a successful Arc fleet run for that release candidate.
• ODS Talk, LAN, AP-mode, packaged appliance handoff, router, Wi-Fi, mDNS, and client-device behavior require target-mode validation because home networks vary.
• A fresh fleet pass is not a long-term soak test. Bench, thermal, and overnight stability runs are separate evidence.
• A green fleet pass is a strong release signal, not a promise that every unsupported driver, storage layout, firewall, network, or optional hardware combination will work perfectly.

Release Readiness Receipt

Before a release is described as ready, the release notes should cite:

• the ODS version and matching Git tag or release;
• the fleet run date and sanitized hardware classes covered;
• regression replay result;
• zero-prereq bootstrap and distro-lab result summary;
• install/verify/dashboard/Hermes/UI/capability/lifecycle result summary;
• any skipped, deferred, blocked-by-environment, or not-run phases;
• known gaps that should not be read as supported behavior.

The version signal should be internally consistent before publication: manifest.json, the changelog section, the Git tag/release, and any release notes should all name the same version. If a candidate has not been tagged yet, describe it as unreleased or release-candidate evidence rather than as a shipped stable release.