raptor-test

August 17, 2026 · View on GitHub

Standalone live-stream conformance suite for RTSP servers. Built for the Raptor Streaming System; works against any RTSP source (checks self-skip when a feature is absent).

It exercises what unit and integration tests structurally cannot: real client sessions over TCP and UDP, several client implementations, concurrent viewers with join/leave churn, wire-level RTP inspection, timestamp/drift math, RFC conformance, NVR recording, and recorded-clip verification. Every check exists because an ad-hoc test once missed the bug it now catches.

Running the whole battery on a target

raptor-test checks one endpoint per invocation. To run the entire battery — every backend (rsd, rsd-555, SRT, RTMP, WebRTC, RHD), the H.265 sub and main passes (each flips one stream and keeps the other H.264, and the main pass carries SRT, the restreamers and an rsd-555 bringup so HEVC rides real frame sizes on every backend), signing, IPv6 and — with DRIFT_PASS=1 — the SR clock-truth pass (dead last: it stops ntpd and drives the kernel frequency ±300ppm with tools/drift-leg.sh), and every leg — the same way on every device, use run-battery with a target manifest so coverage can't drift between targets. The IPv6 pass dials the core suite at the literal in HOST6= (the policy is v6-first, so v6 is asserted, not assumed); targets without HOST6 report it as n/a rather than silently shrinking coverage. A final sr-wallclock pass maps SR NTP against this host's clock with rlatency (built on demand from the raptor sibling) and holds the p50 to 25ms — the anchor-class check that caught a steady -519ms bias every in-band leg was blind to:

./run-battery t23-cinnado      # one target, all its backends
./run-battery --all            # every targets/*.conf, then a matrix
./run-battery --list

Each targets/<name>.conf says where a device is and which backends and legs it exposes; run-battery runs raptor-test against each and prints a pass/fail matrix. Legs a target doesn't expose are marked n/a, never silently dropped. See targets/README.md. Bring-up (build, deploy, start the daemons) is the caller's job — the runner tests, it doesn't manage the lab.

Usage (single endpoint)

./raptor-test rtsp://CAM:9554/ch0 \
    --sub rtsp://CAM:9554/ch1 \          # second stream for the concurrency mix
    --ssh root@CAM \                     # device health checks (raptor bench)
    --go2rtc http://GO2RTC-HOST:1984 \   # restream leg
    --frigate \                          # dockerized Frigate record + clip verify
    --record-check /path/to/recordings \ # verify an external recorder's newest clip
    --user admin --pass secret \        # digest credentials (deployed cameras)
    --rhd https://CAM:8443 \            # RHD HTTP: snapshot, MJPEG, /audio
    --srt srt://CAM:9000 \               # SRT (rsr): MPEG-TS video+audio decode
    --tls rtsps://CAM:9554/ch0 \         # RTSPS/TLS leg (needs [rtsp] tls=true)
    --rtmp 1935 \                        # receive rsp's RTMP push on this port
    --webrtc https://CAM:8554/whip?stream=1  # WebRTC via rwd WHIP (H264 stream)
    --webrtc-main https://CAM:8554/whip \
                                         # second WebRTC pass on the H264 MAIN
                                         # stream, and the ride-through detector
                                         # for --restart-recovery
    --restart-recovery \                 # live set-resolution under load
                                         # (needs --ssh; URL must be the main)

Point the RTSP URL at any backend: raptor's compy server (rsd) or its live555 server (rsd-555) on its own port. The probes read each media's a=control from the SDP, so track naming differences between backends are handled transparently. Two known live555 differences, both upstream library behavior rather than rsd-555 defects: it answers a malformed Transport header with 200 instead of 461 (parseTransportHeader defaults to RTP/UDP and ignores tokens it doesn't recognize), so the garbage-transport robustness check reports a failure against rsd-555; and its PAUSE check can flake under heavy device load (one in-flight frame after PAUSE while the pre-window is starved) — it passes consistently on an idle device.

Requires ffmpeg/ffprobe and python3; uses mpv and docker when present. Run tools/fetch-tools.sh once — it pins an official ffmpeg 8 static build in tools/ (the suite prefers it over the system ffmpeg: client RTSP behavior shifts between majors — 7.1 streamcopy silently drops RTSP AAC audio, 8.x fixed it), builds openRTSP from live555 source for the live555-client leg, and provisions tools/webrtc-venv (aiortc) for the --webrtc leg. Building openRTSP needs make and a C++20-capable g++ (fetch-tools skips it with a warning otherwise; the live555-client leg then self-skips). An explicit FFMPEG/FFPROBE env override still wins, and every run logs the client version it used. Logs land in ./raptor-test-logs/<timestamp>/ and are kept whenever anything fails (--keep-logs keeps them always). Exit codes: 0 = every check passed (skips allowed), 1 = at least one check failed (failed names listed, logs kept), 2 = usage error.

What it checks

Area	Checks
Signaling	SDP structure, AAC `config=` internally consistent with the RTP clock (LC and backward-compatible HE-AAC v1 forms decoded bit-exactly)
Transports	Clean media over TCP-interleaved and UDP, main and sub stream
Wire level	RTP sequence continuity per track via a raw interleaved probe; RTP timestamp discipline: never backward, single-packet spikes triaged against wall-clock arrival (honest source gap vs timestamp anomaly), audio cadence modal share with nudge-direction split (one-sided = tracking a real source clock, two-sided = oscillating steering)
Timestamps	Real capture: per-track monotonicity in native timebases, A/V end alignment; audio cadence drift in ppm for any codec, from a dedicated 30s capture: least-squares slope of pts vs packet index against the capture's own modal packet interval, so AAC, opus, L16 and G.711 all measure without codec-specific frame-size knowledge (a device's audio clock can wander hundreds of ppm over ~10s, so a short window or an endpoint estimator flaps by thousands; `--drift-secs` / `--drift-ppm` tune it per target)
Content	Measured video fps vs nominal (within 15%), resolution sanity, decoded audio sample count vs declared rate (catches SBR/rate mislabels), video bitrate vs the SDP's own `b=AS` (overshoot-only failure: clients size buffers from the hint, so exceeding it harms; undershoot is legitimate on quiet scenes and stays visible, `--bitrate-tol` tunes it), audio channels declared vs decoded per codec: AAC from the ASC `channelConfiguration` (incl. the backward-compatible syncExtension form), opus from `sprop-stereo` (RFC 7587 fixes the rtpmap at `/2` whatever the content), L16 from the rtpmap channel count, G.711 from the RFC 3551 static payload-type table (PCMU/PCMA sections legally carry no rtpmap), with the sendonly ONVIF backchannel section excluded. Declared-mono AAC+SBR or opus showing 2ch passes only when L and R decode bit-identical (ffmpeg presumes Parametric Stereo on mono SBR, "Treating HE-AAC mono as stereo"; opus decoders fill a stereo layout by duplication); distinct content fails as real stereo leakage. GOP cadence from a raw bitstream census, per-frame SEI presence (raptor ST 0604; skipped when absent)
GOP adherence	Keyframes must be strictly periodic (census interval modal +/- 1 frame; drifting or elastic GOPs break recorder segmenting); with `--gop <n>` (`GOP=` in a manifest) the measured interval must equal the configured value
Backchannel	Auto-detected: DESCRIBE with the ONVIF Require header; if the server advertises an `a=sendonly` audio section, SETUP it and push real PCMU RTP at the server — the session must survive and, with `--ssh`, the speaker ring must exist on the device mid-session (found a real demand-accounting bug in rsd's jpeg path the day it was written)
MJPEG over RTSP	Auto-detected: the `/jpeg` endpoint (rsd default) must deliver decodable RFC 2435 MJPEG, not just answer signaling
RTSP behavior	PAUSE halts delivery and PLAY resumes; malformed requests (bogus path, garbage transport, unknown session) answered with 4xx and the server stays healthy
Sessions	A silently-vanished UDP client (no teardown, no keepalive, no RTCP) must stop receiving media within the advertised Session timeout and be reaped server-side; mid-session TCP-to-UDP re-SETUP either moves the media or is refused 455 with the session intact
Auth	(`--user/--pass`) unauthenticated DESCRIBE draws a Digest challenge, unauthenticated SETUP is refused, wrong password rejected, correct digest accepted; the whole suite then runs authenticated (probes speak digest, client legs carry URL credentials)
SRT (rsr)	(`--srt`) 30s window (`--srt-secs`): TS-layer census over rsr's RAW bytes via a concurrent second client (which also proves per-client mux isolation) — sync integrity, per-PID continuity counters (SRT transport is reliable, so any gap is a mux bug), PCR strict monotonicity and 100ms cadence, PAT/PMT 500ms repetition, PMT stream declarations incl. the Opus registration descriptor, PES alignment — plus fps declared-vs-measured over TS, A/V mux end-alignment and monotonicity, cross-transport bitrate agreement with the RTSP leg, audio decoded to PCM, STREAMID substream routing and unknown-id rejection, and with `--srt-passphrase` a wrong-key handshake rejection (AES)
MTU	Max RTP packet size at most 1472 bytes: anything larger IP-fragments on UDP paths, so one lost fragment costs the whole packet
IPv6	(`--ssh`) clean media over the target's global IPv6 address, the family production NVRs commonly attach over
Fault injection	Abruptly killed client (RST mid-stream) followed by clean media for the next client
RFC 2326	OPTIONS Public methods, DESCRIBE/SETUP/PLAY/TEARDOWN status codes, Session header, GET_PARAMETER keepalive, RTP-Info anchors matching the first actual packets
RFC 4566	`o=` origin sanity, media sections, `a=control`, sprop parameter sets for H.26x
RFC 3550	Sender Reports present, compound with SDES CNAME, plausible NTP, sane cadence, cross-track NTP↔RTP mapping consistency (A/V skew), §5.1 nonzero initial seq/timestamp
RFC 3640	AAC-hbr fmtp completeness
RFC 3551/7587	Per-codec audio conformance: G.711 static PT 0/8 at 8 kHz, Opus rtpmap `opus/48000/2`, L16 static PT only at 44.1 kHz (dynamic otherwise)
Concurrency	Ladder of 2, 3, then 4 simultaneous clients on the main stream and 2/4 on the sub stream (alternating transports, every client individually verified); sustained UDP client stays clean across three join/leave cycles
Deployed cameras	Established sessions are baselined before any suite traffic: clean refusals at `max_clients` with external viewers (an attached NVR) become capacity skips, the leak check compares against the baseline instead of zero, and single-digit UDP loss reports as wireless-grade rather than failing the regression guards
Regression	Repeated default-transport ffprobe sessions (UDP dual-SETUP) followed by clean UDP media — guards the re-SETUP fd-leak and cross-wiring bug classes
Players	mpv over TCP and UDP: error-free logs and A-V sync < 0.1s
live555 client	openRTSP (VLC's RTSP lineage, a different stack than libav) negotiates a session and its dumped elementary stream decodes clean — exercises server paths ffmpeg/mpv cannot
RTSPS/TLS	(`--tls rtsps://host:port/path`) TLS handshake, DESCRIBE 200 over rtsps, and clean media through the encrypted transport
RTMP push	(`--rtmp <listen-port>`) rsp pushes to a listener on this host; the received FLV carries video and decodes clean (catches HE-AAC ASC mislabels on the push path)
WebRTC	(`--webrtc <whip-url>`) a real WHIP client (aiortc: SDP/ICE/DTLS-SRTP) decodes rwd's stream and measures framerate, stall stability, PTS monotonicity/cadence, and that audio decodes to real samples (Opus or G.711). Point at an H264 stream (`?stream=N`) — rwd's WebRTC video is H264-only; signaling is HTTPS with Basic auth. Needs `tools/webrtc-venv` (fetch-tools.sh, which also raises aiortc's 128-packet video jitter buffer: stock aiortc cannot reassemble a frame past ~150KB and reads a healthy main stream as zero frames). With `--webrtc-main <url>` the same checks run a second pass against the main stream — the cell whose absence hid a main-stream-only collapse for months
Set-resolution recovery	(`--restart-recovery`, needs `--ssh`, main-stream URL) flips stream0 to half size and back while clients stream; the server must serve each new geometry with real frames inside a bounded wait. The up-flip raises the refmode stride mid-life — the seam the frame-buffer and region-swap bug class ships through. With `--webrtc-main`, a WebRTC session must additionally ride through both encoder restarts with zero stalls
go2rtc	Clean media through a restream; AAC `config=` passed through verbatim; with credentials + a distinct sub, an authenticated substream must flow (regression for digest nonce reuse across producer connections, raptor issue #6); RTSP output port discovered from the API
Frigate	Dockerized Frigate records via TCP and UDP inputs; recorded clips verified
RHD HTTP	(`--rhd`) `/snap` returns a decodable JPEG of sane size (and refuses unauthenticated access when credentials are set); `/mjpeg` is `multipart/x-mixed-replace` carrying multiple JPEG parts that decode frame-by-frame with no errors
RHD /audio	(`--rhd`) the HTTP audio stream (ADTS AAC, Ogg/Opus, or WAV/PCM, de-chunked) is framed correctly and decodes to PCM
Audio-only	Auto-detected from the SDP (m=audio, no m=video): video checks self-skip, clean-media asserts decoded time instead of frames, single-track probes SETUP the audio track, and the full transport/wire/session/RFC battery runs (validated against a live555 WAV source)
Soak	(`--soak <minutes>`) endurance mode replacing the battery: repeated bounded captures plus device RSS/FD trending over the run (fails on >25% RSS growth or >3 new FDs in rvd/rsd)
WebRTC talk-back	The WHIP offer carries a sendrecv audio track (aiortc silence); the answer direction must accept client audio and, with `--ssh`, the decoded audio must land in the device speaker ring mid-session
RTSP fuzzing	(`--fuzz`) throws ~200 malformed requests (bad request lines, oversized/truncated headers, absurd Content-Length/CSeq, garbage methods, partial interleaved frames) at the parser, sending a well-formed OPTIONS between batches; a crash, hang, or wedged accept loop is caught as a liveness failure
SRT	(`--srt`) connects as a caller to the SRT listener (rsr), confirms the MPEG-TS carries video, decodes the video clean, and decodes the audio all the way to PCM — the last step catches an ADTS sample-rate mislabel that a container probe would miss
Codec matrix	`tools/codec-matrix.sh` sweeps l16/pcmu/pcma/opus/aac against both RTSP backends per codec, asserting the audio RTP timestamp step equals the codec's real frame duration (a server timestamping AAC at its 20ms chunk rate decodes fine but runs the clock 3x fast)
Clips	Recorded files: streams present, durations aligned, full decode with zero errors (null-muxer dts nag and its repeat-tails excluded; file-level dts asserted to never move backward instead)
Device	(`--ssh`) rsd CPU sane after the suite, connection count back at the pre-suite baseline, daemon log lines emitted during the run free of ERROR-level entries and crashes, framesource "losted buffers" counters flat across the run

Day/night hardware cycle, ramps and light calibration

Targets with a scriptable room light unlock the --daynight hardware cycle: forced-mode rail assertions, real dusk/dawn transitions, an IR-lit dark hold, and — with a dimmable light (--lights-set) — log-spaced dusk/dawn ramps carrying a hysteresis assertion (the threshold region must be crossed exactly once under a monotonic dim; a step change can never test that). --light-cal characterizes the lamp against the camera's own AE and thresholds, --light-verify re-checks three rungs against the pinned table every session so bench drift is caught as a bench fact, and the daynight ramps derive their rungs from the stored table. BENCH.md is the full reproduction guide — minimum hardware, manifest wiring, methodology order, provenance, and how to quote numbers; examples/ holds sanitized real artifacts.

Self-tests

The suite tests itself, because a conformance suite that never fails is worthless. All three run on localhost with no camera:

tools/self-check.sh — starts probes/evil_server.py (a deliberately misbehaving RTSP/RTP server) in each break mode and asserts every probe PASSes on correct media and FAILs on the one broken property: silent-play (the vacuous-pass guard: a server that answers PLAY but sends nothing must fail "clean media", not pass it), sequence drops, backward timestamps, timestamp spikes, and a garbage-transport-200. 10/10, deterministic.
tools/host-battery.sh — runs the real --core entrypoint against a local live555MediaServer. The dry run of a CI job; the handful of documented live555 deviations are allowlisted so a genuine regression still turns it red.
tools/impair-check.sh — runs a server in a network namespace, applies tc netem to its loopback, and proves the capture path detects controlled loss (frames drop and decode errors appear under 15% loss) and recovers cleanly. Needs root; frame counts are asserted so "0 errors" can never silently mean "no media".

Run tools/lint.sh (shellcheck + ruff) before committing. Structured output for CI: --json <file> writes one JSONL record per check. Borderline thresholds are overridable per target: --fps-tol, --drift-ppm (and the matching manifest fields) so a marginal unit stops flapping an absolute limit.

Design notes

All log matching is case-insensitive (Error vs error once hid a day of HEVC join noise).
Pipelines are never truncated mid-evidence; failing checks print the exact matched lines and keep their logs.
Background clients are killed with their process tree. Killing only a timeout wrapper orphans its ffmpeg, which then streams forever and masquerades as server load.
The live-decode error pattern excludes the null muxer's "non monotonically increasing dts" nag: -f null forces a 1/1000 timebase where 30 fps frames legally collide on the same millisecond. Real monotonicity is asserted from captures in native timebases and on the wire.
ffprobe with no transport flag intentionally probes over UDP with a dual-track SETUP: that exact traffic shape found two real server bugs the day this suite was written.
Copy captures use -copyinkf: ffmpeg's streamcopy waits for a key-flagged packet per stream, but its RTP AAC depacketizer never emits one (AAC is not intra-only to ffmpeg), so a plain -c copy from ANY rtsp server -- go2rtc included -- silently records no audio. A dedicated check documents the plain-copy behavior.
Wire-level RTP timestamp checks (backward steps, single-packet spikes, audio cadence stability) exist because a steering loop that bang-bangs +-1ms passes every decode check while file muxers silently drop packets around the jitter.
Every external client call is wall-clock bounded. An SRT caller blocks forever against a listener that completes the handshake but never sends data (one wedged rsr hung a full battery for 40 minutes), and ffmpeg's -t only limits capture after connect. Bounded captures also require ffmpeg rc=0: a mid-capture stall killed by timeout leaves an empty error log, which a grep-only check would read as a vacuous "decodes clean" pass.
Target manifests are sourced bash; a quoting error aborts sourcing mid-file and silently drops every later field. run-battery hard-errors on a conf that fails to source instead of running a partial battery that still looks green.

Codec and rate matrices

The suite validates whatever the target is configured for. To sweep a raptor camera through its audio matrix (l16, pcmu, pcma, opus, aac at various rates), tools/codec-matrix.sh user@cam /path/to/conf edits the config over ssh, bounces rad+rsd from the NFS build, and runs the per-codec RFC probe, wire timestamp checks, and a decode against each combination. HW-validated matrix on T31: all five codecs conformant with exact wire cadence (G.711 modal 160, opus modal 960 at 100%).

Not covered (yet)

Multi-day soak (the --soak mode trends RSS/FDs over minutes to hours, not days) and content verification of talk-back audio at the far end (the suite proves arrival in the speaker ring, not what rad plays out). The raptor provenance suite separately covers snapshots, EXIF, signing, and rverify chains.

To self-test the audio-only mode without a camera: build live555MediaServer (same source fetch-tools uses for openRTSP), serve a plain WAV from its working directory (python's wave module output works; ffmpeg's WAV writer adds metadata chunks live555 rejects), and point the suite at rtsp://host:8554/<file>.wav.