ag-ui-validate

August 29, 2026 · View on GitHub

Conformance validator for the AG-UI protocol (Agent–User Interaction Protocol). Point it at an AG-UI endpoint — or feed it a recorded event stream — and it reports every way the stream violates the protocol, with a rule ID, a severity, a location, and a link to the governing spec section.

✖ AGUI203  error  event 42  TOOL_CALL_START id 'call_7' never terminated
✖ AGUI302  error  event 51  STATE_DELTA failed to apply: /items/3: '3' is not a valid index for an array of length 0
✖ AGUI503  error  event 60  Unknown event type 'runStarted' — did you mean 'RUN_STARTED'?
ℹ AGUI902  info   —         None of the 61 events carry the optional timestamp property

2 errors, 0 warnings, 1 info — 3 of 7 AG-UI features exercised

Status: released to both npm (npm install ag-ui-validate) and PyPI (pip install ag-ui-validate). Both packages version in lockstep — same rule catalog, same fixture corpus, byte-identical CLI; see Python below.

Why

AG-UI has SDKs and integrations, but no conformance tooling: nothing tells an implementer your stream is subtly wrong, here's the rule and the spec section. This project is that tool — the AG-UI analogue of what a2a-inspector is for A2A.

Three design commitments make it trustworthy:

Every diagnostic cites the spec. Each of the 46 rules carries a specUrl (and where possible an exact specQuote) pointing at the governing section of docs.ag-ui.com or the WHATWG SSE spec. Behaviour the spec doesn't clearly govern is reported at info severity at most, and logged in docs/spec-questions.md for filing upstream.
The validator never throws. Broken input is its input. Malformed JSON, unknown event types, hostile objects — all diagnostics, never exceptions (fuzz-tested against 50k hostile inputs).
False positives are treated as worse than false negatives. The rules are grounded in @ag-ui/core v0.0.59 and the current docs; where the two disagree, the SDK wins and the discrepancy is recorded.

Quickstart

npm install --save-dev ag-ui-validate

CLI

npx ag-ui-validate http://localhost:8000/agui   # live endpoint (POSTs a RunAgentInput)
npx ag-ui-validate run.jsonl                    # recorded stream (NDJSON/JSONL or SSE capture)
cat run.jsonl | npx ag-ui-validate -            # stdin

Exit codes: 0 clean, 1 findings at or above the --fail-on threshold (error by default, or warnings over --max-warnings), 2 tool failure. Timing-based transport rules are meaningless for recordings, so they are reported as skipped with a reason rather than risking false positives.

Useful flags (see --help for all):

Flag	Effect
`--json` / `--sarif` / `--junit`	machine-readable report on stdout (SARIF 2.1.0 for code scanning, JUnit XML for CI)
`--group`	one line per rule with a count — for large streams with repeated findings (totals stay exact)
`--rule AGUI105=error`, `--off AGUI902`	per-rule severity overrides
`--features shared-state,...`	declare exercised features (enables e.g. AGUI305)
`--max-warnings 0`	fail CI on any warning
`--fail-on <error\|warning\|none>`	severity that triggers a nonzero exit (default `error`; `none` for report-only runs)
`--header "Authorization: Bearer …"`, `--timeout 30`	endpoint options

Validate in CI (GitHub Action)

- uses: langport-dev/ag-ui-validate-action@v1
  with:
    target: http://localhost:8000/agui   # or a recorded .jsonl file
    sarif-file: agui.sarif               # optional: upload via codeql-action

The step fails on error-severity findings, writes a findings table (with a counts-by-category breakdown) to the job summary, annotates the PR inline with ::error/::warning for each finding, and exposes errors/warnings/info/sarif-path/report-path outputs — see langport-dev/ag-ui-validate-action.

Test your agent in Vitest

import "ag-ui-validate/vitest" // registers the matcher (put it in setupFiles)

it("streams a conformant run", async () => {
  const events = await captureRunEvents(myAgent) // however you record them
  expect(events).toBeValidAGUI()
})

The matcher takes an array of events (objects or JSON strings) or a whole JSONL capture as one string. Failures print each finding with its rule ID and spec link. Options mirror the validator: { features, severityOverrides, maxWarnings } — e.g. expect(events).toBeValidAGUI({ maxWarnings: 0 }) to fail on warnings too. The raw matcher function is also exported, so Jest users can expect.extend({ toBeValidAGUI }) themselves.

Validate recorded events (pure, runs anywhere)

import { createValidator } from "ag-ui-validate"

const v = createValidator({
  features: ["shared-state"],            // optional: enables feature-specific rules
  severityOverrides: { AGUI902: "off" }, // optional: tune or disable rules
})

for (const event of events) {
  // feed parsed objects or raw JSON strings — bad JSON is a diagnostic
  const diagnostics = v.feed(event)      // findings, as soon as detectable
}
v.finalize()                             // end-of-stream checks

const { diagnostics, summary, features, skipped } = v.report()

The core is a pure function over an event sequence: zero I/O, zero runtime dependencies, isomorphic across Node 22+, browsers, Deno, and Workers.

Validate a live endpoint

import { validateEndpoint } from "ag-ui-validate/transport"

const { report, status, eventCount } = await validateEndpoint(
  "http://localhost:8000/agui",
  {
    headers: { authorization: "Bearer …" },
    onDiagnostic: (d) => console.error(`${d.severity} ${d.rule} ${d.message}`),
  },
)

The transport layer POSTs a minimal RunAgentInput, consumes the SSE or NDJSON response, streams every frame through the core, and additionally evaluates the transport-level rules that recorded input can't exercise: SSE framing (including the classic missing-data:-prefix bug), Content-Type, keepalive gaps, buffered-not-flushed responses, and mid-run disconnects.

Render a report

The CLI's output formats are plain functions over a Report, importable for your own tooling:

import { formatReportSummary, toSarif, toJUnit } from "ag-ui-validate/report"

Diagnostic shape

{
  "rule": "AGUI203",
  "severity": "error",            // "error" | "warning" | "info"
  "message": "TOOL_CALL_START id 'call_7' never terminated",
  "eventIndex": 42,               // 0-based; -1 for end-of-stream findings
  "eventType": "RUN_FINISHED",    // optional
  "pointer": "/toolCallId",       // optional RFC 6901 pointer into the event
  "relatedEventIndex": 17,        // optional, e.g. the unterminated start
  "specUrl": "https://docs.ag-ui.com/concepts/events#tool-call-events"
}

Python

A native Python port ships in py/ — the same rule catalog, the same fixture corpus, and a CLI with byte-identical flags, error messages, and JSON/SARIF/JUnit output to the TypeScript one above. It's checked against the TypeScript implementation on every PR by Parity CI (see docs/TESTING.md for how, and docs/PYTHON-PORT-PLAN.md for the port's full milestone history).

pip install ag-ui-validate   # or "ag-ui-validate[transport]" for just the endpoint-validating extras

CLI

ag-ui-validate http://localhost:8000/agui   # live endpoint (POSTs a RunAgentInput)
ag-ui-validate run.jsonl                    # recorded stream (NDJSON/JSONL or SSE capture)
cat run.jsonl | ag-ui-validate -            # stdin

Same exit codes and the same flags as the CLI above (see ag-ui-validate --help) — the Python argument parser is a direct, hand-rolled port of the TypeScript one rather than argparse/click, specifically to keep both CLIs' invocation and error messages byte-for-byte identical.

Test your agent in pytest

from ag_ui_validate.pytest_plugin import assert_valid_agui

def test_my_agent_stream(captured_events):
    assert_valid_agui(captured_events, features=["shared-state"], max_warnings=0)

assert_valid_agui accepts a list of events (dicts or JSON strings) or a whole JSONL capture as one string, and raises AssertionError with every finding's rule ID and spec link on failure. It registers automatically as a pytest plugin on install — no conftest.py setup needed. An async counterpart validates a live endpoint directly from a test:

from ag_ui_validate.pytest_plugin import assert_valid_agui_endpoint

async def test_my_live_agent():
    await assert_valid_agui_endpoint("http://localhost:8000/agui", max_warnings=0)

assert_valid_agui has zero runtime dependencies; only the endpoint-validating helpers (assert_valid_agui_endpoint, validate_agui_endpoint) pull in httpx (pip install ag-ui-validate[transport]).

The rule catalog

46 rules, maintained as data in spec/catalog.json, shared by both implementations in this repo (TypeScript and Python) rather than duplicated. Every rule has its own page — spec grounding, rule index (generated from the catalog, drift-checked in CI):

Group	IDs	Examples
Lifecycle	AGUI001–008	run must start with `RUN_STARTED`, terminate with `RUN_FINISHED`/`RUN_ERROR`, nothing after a terminal event
Text messages	AGUI101–106	content without start, unterminated messages, duplicate `messageId`
Tool calls	AGUI201–208	unterminated calls, args that don't concatenate to valid JSON, results referencing unknown calls
State	AGUI301–305	RFC 6902 patch validity, deltas that fail to apply to reconstructed state
Reasoning	AGUI401–402	reasoning content without an open reasoning message
Transport	AGUI501–508	SSE framing, Content-Type, keepalive gaps, buffering, dropped connections
Subagents	AGUI601–606	duplicate/unmatched `SUBAGENT_STARTED`/`FINISHED`/`ERROR`, unterminated subagents, unknown `parentSubagentRunId`, continuation events that disagree with their entity's owner
Hygiene	AGUI901–903	`RAW`-wrapping typed events, missing timestamps, un-namespaced `CUSTOM` names

The event taxonomy itself (36 wire types, field schemas) is derived from @ag-ui/core's own schemas and drift-tested against the installed SDK on every run.

The fixture corpus

spec/fixtures/ is a language-neutral conformance corpus: 7 valid streams (one per canonical AG-UI feature — the false-positive guards) and 46 invalid fixtures (one per rule) with exact expected diagnostics. Any validator implementation that consumes the shared catalog can be tested against it; the replay protocol is documented in the corpus README.

Development

npm ci
npm run typecheck   # includes a src-only pass proving the core uses no Node APIs
npm run build       # dual ESM/CJS via tsdown
npm test            # full suite: unit + corpus + drift + purity + SDK alignment
npm run demo        # pretty-printed findings for a deliberately broken stream
npm run e2e         # live-transport checks against a real local HTTP server
npm run fuzz        # 50k hostile inputs against the never-throws invariant
npm run links:check # every specUrl resolves and every anchor exists

Component-by-component instructions live in docs/TESTING.md. Spec ambiguities found while grounding the rules are tracked in docs/spec-questions.md.

Adding a rule: add the catalog entry (with its specUrl), add the fixture stream + intended findings to js/scripts/build-fixtures.mjs, and run npm run fixtures:build — the meta-tests fail until both exist. Rule proposals belong upstream as issues on ag-ui-protocol/ag-ui first; this project does not invent rules the spec doesn't support.

Releasing: merge the pending changesets (npx changeset version) via a PR, then publish a GitHub release tagged vX.Y.Z (matching package.json) — the Publish workflow typechecks, builds, tests, and publishes to npm with provenance via trusted publishing. The workflow fails fast if the tag and package.json disagree.

License

MIT — maintained by Faraz.