252.md

July 14, 2026 · View on GitHub

The following transcript describes the content of a video about "OpenAgents Assurance," an automated quality assurance and testing framework.

00:00 - 00:30: Introduction to Automated Testing

Speaker: "All right, we’ve got a release build. It’s not pretty. Kind of going through the automated testing here. So, beyond the ProductSpec, we want to have that idea extended through the rest of development. We had done some thinking a few videos ago about an automated QA process."

On-screen Text (X Post - @gokulr):

EVIDENCE LOOP FOR PRODUCTSPEC A Product Spec should not stop at launch.

The common failure mode with product docs is that they describe intent before the work, then disappear once the work starts.

A PR ships, an eval runs, a dashboard moves, a customer complains. BUT the product doc stays frozen.

Then 3 weeks later, nobody knows which acceptance criterion the PR satisfied, which eval run proved the model behavior, or which dashboard showed whether the product bet worked.

To fix this, we just added evidence support to ProductSpec.

The core idea is simple: ProductSpec defines intent. Evidence shows what happened.

00:30 - 01:08: The Concept of a QA Swarm

Speaker: "Like, we want to automate quality assurance and testing. So, this idea of a QA swarm. A swarm of QA agents pointed at your product to get proof it works. A coordinated fleet of autonomous QA agents—scripted scenarios, seeded monkeys, LLM explorers, perf probes—drives your app through a real browser, a real terminal, and (on macOS) the real native window; every discovery distills into a committed, re-runnable e2e test; every run produces an honest CONFIRMED/REFUTED verdict, videos, exact accounting, and a shareable web URL where the whole swarm is visible as a live, cinematic StarCraft-blue board..."

On-screen Text (X Post - @Christopher David):

Episode 246: Dogfooding Khala Code We use Khala Code to fix bugs in Khala Code.

We design a "QA Swarm" for automated quality assurance and testing.

We'll build QA Swarm first for Khala Code, then add it to our Autopilot product suite for businesses.

From Fable: "QA Swarm: point a swarm of QA agents at your product and get proof it works. A coordinated fleet of autonomous QA agents — scripted scenarios, seeded monkeys, LLM explorers, perf probes — drives your app through a real browser, a real terminal, and (on macOS) the real native window; every discovery distills into a committed, re-runnable e2e test; every run produces an honest CONFIRMED/REFUTED verdict, videos, exact accounting, and a shareable web URL where the whole swarm is visible as a live, cinematic StarCraft-blue board — nodes for every agent and scenario, edges that light only when real receipts land."

QA Swarm spec: github.com/OpenAgentsInc/...

01:08 - 02:49: Integrating Evidence into the ProductSpec

Speaker: "Okay. So, we've enjoyed creating the ProductSpec. We've implemented what we designed earlier today. And then we're like, okay, let's connect this with the QA swarm idea. And then as we were starting that, realized that Gokul had extended ProductSpec to encompass evidence. So kind of had ours extend this idea of you are connecting evidence into the ProductSpec. So let's just read this real quick."

Speaker (reading the X post):

"A Product Spec should not stop at launch. The common failure mode with product docs is that they describe intent before the work, then disappear once the work starts. A PR ships, an eval runs, a dashboard moves, a customer complains. But the product doc stays frozen. Then 3 weeks later, nobody knows which acceptance criterion the PR satisfied, which eval run proved the model behavior, or which dashboard showed whether the product bet worked. To fix this, we just added evidence support to ProductSpec. The core idea is simple: ProductSpec defines intent. Evidence shows what happened. Decision Trace records what changed. Related Artifacts now let teams attach evidence directly to ProductSpec IDs. AC-1 can link to the PR, test, release, or code that implemented it. EVAL-1 can link to the eval run or human review record that checked model behavior. SM-1 can link to the dashboard, analytics snapshot, or experiment that measured the post-launch outcome. This matters more as agents write more code. An agent can claim it implemented something. A PR can look complete. A test suite can pass. But the useful question is: which piece of product intent did this evidence satisfy? That is where structured specs start to matter. If AC-2 says the user can export a dashboard with visible filters preserved, the implementation PR should point back to AC-2. If EVAL-1 checks whether an AI support triage model correctly identifies account-risk tickets, the eval run should point back to EVAL-1. If SM-1 measures median time to first human response, the dashboard or analytics snapshot should point back to SM-1. This turns a Product Spec from a planning document into a record of intent plus proof. A few important boundaries: ProductSpec does not run evals. ProductSpec does not collect production traces. ProductSpec does not replace Braintrust, Langfuse, Datadog, GitHub, Linear, or your analytics stack. ProductSpec gives all of those artifacts a stable place to attach. The latest validator now catches stale evidence links. If a Related Artifact points to AC-99 and no AC-99 exists, that is invalid. It also warns when the evidence type looks mismatched, like an eval run attached to a success metric instead of an eval. This is the direction I’m most excited about: Software intent that survives implementation. Evidence that connects back to intent. Decision traces that explain what changed when reality pushed back. Founders and builders: if your team is using AI agents to build software, you should be defining intent via ProductSpec, not just code against the current codebase."

Speaker: "ProductSpec defines intent. Evidence shows what happened. Decision Trace records what changed... ProductSpec does not run evals... ProductSpec does not replace Braintrust, Langfuse, Datadog, GitHub, Linear, or your analytics stack. OpenAgents replaces this stuff! We're coming at GitHub and Linear and, you know, looking into these others as well."

02:49 - 04:08: Introducing "Observer"

Speaker: "So, we have a draft of something we're for now calling AssuranceSpec. And then we're going to turn this into a product. We gotta use from the StarCraft universe, right? So it's going to be called Observer. Proof-design for software built by agents. Observer turns an accepted product spec into a reviewed proof design, compiles it into an immutable verification manifest, and runs it through real QA tools—so when the checks go green, the green means what it claims. Built on AssuranceSpec, a framework-neutral standard for committed verification intent. In development at OpenAgents—designed in the open, dogfooded on our own desktop app before anyone else's code. So there'll be some level of putting this product to QA, like automated software factory process into the OpenAgents desktop app. So instead of, like, purchasing some third party's expensive software factory—sorry Chamath—you're just going to get it running locally using your existing subscription. How about that? This is all just super draft kind of thrown together, but you can kind of browse that to see where we're going with that. That's at openagents.com/observer."

On-screen Text (OpenAgents Website - /observer):

Observer. Proof-design for software built by agents. Observer turns an accepted product spec into a reviewed proof design, compiles it into an immutable verification manifest, and runs it through real QA tools — so when the checks go green, the green means what it claims.

Built on AssuranceSpec, a framework-neutral standard for committed verification intent. In development at OpenAgents — designed in the open, dogfooded on our own desktop app before anyone else’s code.

THE FAILURE MODE — A PASSING SUITE IS NOT PROOF

  • false_green_fixture_assert: The test asserts the fixture. It proves the mock behaves like the mock — not that the project behaves like the spec.
  • false_green_api_mirror: The test mirrors the implementation. Written after the code, it inherits the code’s assumptions — including the wrong ones.
  • false_green_mocked_seam: The real seam is never exercised. Both sides of a client-server or renderer-host boundary are faked, so integration defects pass clean.
  • false_green_coverage_theater: Coverage stands in for behavior. Line-coverage measures execution, not correctness — activity dressed up as evidence.
  • false_green_round_up: Everything rounds up to green. Skipped, stale, flaky, and inconclusive results quietly vanish from the summary.

04:08 - 05:02: OpenAgents Assurance Repository and Wrap-up

Speaker: "And then in the assurance folder... one of the things we're doing is, like, taking the kind of agent tooling that the ProductSpec repo has and just building the same tooling, MCP and stuff, for QA. So the OpenAgents repo will speak—or the OpenAgents desktop app will speak—both ProductSpec and AssuranceSpec, and then anybody else can use this same tooling, but we'll keep it in our OpenAgents mono-repo for now. Gap analysis, blah blah blah. Anyways, feel free to point your agents at this stuff and give us some feedback on any of this. We're going to be, like, more thoroughly testing this for a few iteration cycles before releasing it, but hopefully that should be tomorrow afternoon. See you soon."

On-screen Text (GitHub - OpenAgents Assurance):

OpenAgents Assurance This directory is the canonical home for OpenAgents proof-design architecture: the AssuranceSpec companion standard, Observer, Assurance Manifests, Environment Profiles, adapter contracts, evidence receipts, and the first dogfood plan.

Assurance is deliberately separate from ProductSpec’s portable evidence index, the existing Desktop ProductSpec workroom loop, and QA execution.

  • ProductSpec: commits intent and indexes evidence with Related Artifacts
  • workroom: runs accepted plans, packets, evidence, verification, disposition
  • AssuranceSpec: commits how we intend to know
  • Observer: compiles admitted verification obligations
  • QA tools/Swarm: execute and emit exact Assurance Receipts

On-screen Text (OpenAgents App):

03:36 AM The version bump is now on main at 2858c67d9d, and the repository's first pre-push Desktop gate passed: 1,301 tests, typecheck/build, and the complete Electron smoke. The clean release preflight's eight start/exit rows are green, its final credential row correctly refused until the Apple signing environment is loaded. I'm building the owner-held release environment now and starting the signed/notarized RC11 build.

03:37 AM The signed build has cleared native-addon preparation, the owned Rust audio helper build, asset generation, and Electron packaging. It is now in the slower signing/notarization phase; the release preflight is fully green with the Developer ID and API-key credentials loaded.