Streaming latency engineering log

August 14, 2026 · View on GitHub

A running, append-only engineering log for tapflow's glass-to-glass latency work on tier1 (localhost + LAN). It records pipeline analysis, measurements, attempts, and decisions in chronological order — useful context for anyone touching the streaming render path. Unlike .work/ (local, throwaway) it is committed and kept.

Goal: on tier1 (localhost + LAN), reach localhost-JPEG-class responsiveness (glass-to-glass latency where operating the simulator/emulator in the browser feels nearly as direct as touching it locally). This is the start of tapflow's core value: no-install, low-friction, low-latency remote control.

Where it stands: MSE was dropped for a 2-tier decoder path — WebCodecs on secure contexts (HTTPS/localhost), WASM (tinyh264) on plain HTTP — reaching localhost-JPEG-class latency with no media-element buffer. relay drop-to-keyframe removes tearing under LAN backpressure, and H.264 is now the negotiated default. Same pipeline on iOS and Android.

Measurement tools: TAPFLOW_STREAM_METRICS=1 (agent throughput) · relay ws backpressure (drops) · the per-stage ?perf=1 panel (capture→display). Environment: LAN + HTTP / localhost, iPhone 16 Pro, macOS.

1. The problem — two demands at once

Feels-like-direct responsiveness needs both:

Low latency (responsiveness) — the screen follows the instant you swipe.
No drops (smoothness) — no tearing from backpressure drops.

So far no path has delivered both at once:

Path	Latency	Drops (bandwidth)
JPEG	✅ low (frame-independent, instant decode)	❌ large → LAN scroll relay drops 16–27/s (tearing)
H.264	❌ high (encoder + HW-decoder pipeline; on LAN, + MSE buffer)	✅ small → drops 1–11/s

→ Only localhost-JPEG meets the bar. LAN-JPEG tears; H.264 (local/LAN) misses the bar on latency.

2. Pipeline & bottleneck (glass-to-glass)

Stage	JPEG	H.264 (ours)	Cloud gaming
Touch send (browser→relay→agent→HID)	same	same	same
Simulator scroll render	same (simulator-inherent)	same	—
Capture (30fps polling)	0–33ms	0–33ms	—
Encoder pipe	0 (instant)	VT (minimized with MaxFrameDelayCount=0)	NVENC low-latency preset
Transport + buffer	instant	LAN = MSE media buffer ← the core bottleneck	UDP/WebRTC mini jitter buffer
Decoder	createImageBitmap, instant	WebCodecs (low-latency) / MSE (buffered)	HW low-latency decode

Key insights:

The bottleneck is not "H.264 vs JPEG" but the decode/transport path. Cloud gaming hitting glass-to-glass <50ms with H.264 is the existence proof — H.264 can deliver low bandwidth and low latency together.
The real problem is using MSE on LAN-HTTP. MSE is buffering by nature, so it is structural latency. Cloud gaming does not use MSE.
Why we were stuck with MSE = the secure-context wall: WebCodecs (zero-buffer, HW) only works on HTTPS/localhost → LAN-HTTP is non-secure, so it gets demoted to MSE.

3. Decoder tier model (decided — 2-tier, MSE removed)

pickDecoder selects automatically per environment. MSE is fully removed and simplified to 2 tiers — WebCodecs if secure, else WASM. Same pipeline on iOS and Android:

Environment	Decoder	Characteristics
HTTPS / localhost (secure)	WebCodecs	HW, lowest latency, all profiles
HTTP (non-secure)	WASM (tinyh264) ✅ measured PASS	CPU, low-latency, zero-buffer, baseline only
WebGL2/WASM unavailable	null (notice)	—

Why MSE was removed (2026-06-02): MSE has a structural ~235ms from the <video> buffer — goal unreachable. WebCodecs (secure) + WASM (HTTP) cover every environment, so an MSE fallback is needless complexity. The jmuxer dependency is dropped too. Android baseline premise: WASM (tinyh264) only decodes (constrained-)baseline → pin scrcpy to baseline (profile:int=1) so Android shares the HTTP→WASM path. (High profile only on the WebCodecs=secure path; on plain HTTP, baseline is guaranteed.) iOS uses VideoToolbox baseline.

Why WASM is attractive: no secure context needed + no media buffer = JPEG's immediacy + H.264's low bandwidth, both on LAN-HTTP. The cost is software (CPU) decode — at high resolution×fps it hits a CPU ceiling. Mitigation: downscale the encode resolution (display is small → a triple win on bandwidth, CPU, latency). Precedent: ws-scrcpy decodes phone-resolution baseline H.264 with tinyh264.

4. Measurement log (accumulated)

JPEG baseline (LAN, throughput sampler)

Scenario	Quality	avg/frame	Bandwidth	relay drops/s
Still	0.95	427 KB	~3.3 MB/s	0 (localhost illusion)
Still	0.8	235 KB	~1.97 MB/s	0
Scroll	0.8	~590 KB	12–16 MB/s	16–27

H.264 (VideoToolbox, steady 30fps)

Scenario	avg/frame	Bandwidth	relay drops/s
Still	~1.8 KB	~14 KB/s (~140x↓ vs JPEG)	0
Scroll	~90–110 KB	~2.6 MB/s (~5x↓)	1–11

→ H.264 decisively solves bandwidth/drops. But perceived latency is worse than JPEG (below).

Latency (perceived, qualitative)

Path	Feel
localhost JPEG	baseline — feels like direct manipulation
localhost H.264 (WebCodecs)	half-beat lag — misses the bar
LAN JPEG	responsive but scroll tears
LAN H.264 (MSE)	near-zero drops but the largest latency

⚠️ Precise per-stage latency (ms) not yet measured. In step 0, measure capture→display + encoder time to rank bottlenecks by number. (Especially why even localhost-WebCodecs-H.264 misses JPEG — encoder, decoder, or display path.)

per-stage latency (step-0 instrumentation — measured 2026-06-02, localhost)

Read from the ?perf=1 panel's p50/p95 ms folder + the [wc-diag] console (exact timestamp matching).

Path	decode→present p50/p95	glass→glass p50/p95	agent→relay	Note
JPEG still	12.4 / 15.4	(≈13)*	1	baseline
JPEG scroll	9.4 / 11.6	(≈11)*	1
H.264 still (before fix)	267 / 274	235 / 239	0–1	decoder buffers ~8 frames
H.264 WebCodecs still (after fix)	2.5 / 4	3.9 / 7.9	0–1	reorder=0 injected
H.264 WebCodecs scroll (after fix)	2.1 / 3.9	3.4 / 7.9	0–1
H.264 MSE still	239 / 254	240 / 255	0–1	`<video>` buffer remains even with reorder=0
H.264 MSE scroll	229 / 244	230 / 245	0–1	structural even with jmuxer `flushingTime:0`
H.264 WASM still	8.7 / 30.4	9.6 / 31.5	0–1	tinyh264, no media buffer — JPEG-class
H.264 WASM scroll	14.3 / 37.9	16 / 40.3	1–2	localhost-JPEG-class responsiveness

→ WebCodecs: the fix took decode→present 267→2.5ms (~100x↓), reaching the goal (localhost-JPEG-class). But MSE stays ~235ms even on the same reorder=0 stream — the media-element (<video>) buffer the SPS fix can't reach (already tuned to flushingTime:0, can't shave more). → Measurements confirm tier1 LAN-HTTP (non-secure, MSE) needs WASM (tinyh264). (Measured on localhost single-clock via the ?decoder=mse dev override — MSE buffers instead of dropping, so the FIFO tracker is accurate.)

→ Step-2 WASM measurement (2026-06-02, ?decoder=wasm): gate PASS. tinyh264 fully avoids MSE's <video> buffer → decode→present still 8.7ms / scroll 14.3ms (~27x·~16x↓ vs MSE 239/229), glass→glass 9.6/16ms. On par with localhost-JPEG (goal 12.4/9.4) — still is faster than JPEG. Color and feel are good, with CPU headroom (localhost = encoder + decoder on one Mac = worst case, so real LAN has more headroom). → Goal reached on the non-secure path; downscale (step 3) not needed for now. Next: flip the pickDecoder tier (non-secure → WASM first).

* JPEG glass→glass isn't computed by the panel (null) — only on the H.264 tracker path. ≈11–13ms from decode (~10ms) + agent→relay (~1ms). Clock validity: glass→glass = present(epoch) − capturedAt → only on a localhost single clock. decode→present (a delta) and agent→relay (both on the Mac) are valid in every environment. Root cause: the iOS encoder's SPS does not set bitstream_restriction (VUI check: bitstreamRestriction:false, Level 5.0). The decoder safely buffers up to the level's max DPB (110400/~13800MB ≈ 8 frames) → 267ms. It's baseline (66) so the actual reorder is 0 — only the signal is missing. → Fixed by injecting a max_num_reorder_frames=0 declaration into the SPS.

render/decode isolation (spike a — measured 2026-06-06, localhost `?perf=1&decoder=wasm`)

Isolated drawI420 (GPU plane upload + YUV→RGB shader) from the worker decode+transfer portion of the WASM decode→present, to locate the bottleneck before an openh264-wasm swap. [wasm-render] (WASMDecoder DEV diag) logs drawI420 duration; the panel's Log latency summary gives the full-session decodeMs. iOS path (only viewer with ?decoder=wasm).

Phase	decodeMs p50/p95/max (full)	drawI420 p50/p95 (render)	glass→glass p50/p95
still (n=1090)	8.6 / 18.8 / 64.8	~0.25 / ~0.5	9.7 / 23.0
scroll (n=996)	12.7 / 26.3 / 96.5	~0.20 / ~0.5	14.4 / 28.8

→ Render is ~2–3% of decode→present (drawI420 p50 ~0.25ms vs decodeMs p50 8.6–12.7) and content-independent — flat still↔scroll, while decodeMs rises with motion (p50 +4.1, p95 +7.5). So the scroll increase and the entire p95/max tail are decode+transfer, not render. Conclusion: the WASM bottleneck is the software decode (h264bsd, scalar) — renderer ruled out. Justifies the openh264-wasm spike; gate = shrink the decode p95 tail. The renderer is shared across decoders, so this isolation transfers (a swap's gate = decodeMs − ~0.5ms render).

FFmpeg-h264 (scalar wasm) vs tinyh264 — engine probe (c-0, measured 2026-06-07, localhost)

Probed whether a "better-engineered" decoder (FFmpeg h264, libavcodec) beats tinyh264 (h264bsd) on the non-secure WASM tier. Built FFmpeg n5.1.2 --disable-asm (scalar) to wasm, wired behind ?decoder=ffmpeg reusing WASMDecoder/YUVWebGLRenderer (packed I420 via av_image_copy_to_buffer). Both measured same session (machine state drifts, so the tinyh264 baseline was re-measured alongside, not reused from the spike-a numbers above).

metric	tinyh264 (fresh)	FFmpeg	delta
still p50/p95	10.2 / 22.5	8.7 / 13.4	p95 −40% ✓
scroll p50/p95	14.8 / 27.2	16.3 / 29.7	p95 +9% ✗

→ Mixed, not a clear win. FFmpeg crushes the still p95 tail (13.4 vs 22.5) but is slightly worse on scroll (the high-motion case that matters most) on both p50 and p95. Confirms the earlier thesis: FFmpeg's native lead is asm-derived; stripped to scalar C in wasm, it doesn't beat the leaner baseline-only h264bsd on heavy P-frame/motion-comp loops. By the success gate (shrink the scroll p95 tail), FFmpeg-scalar fails. Caveats before a verdict: (1) the FFmpeg core was built without explicit -msimd128 (only the glue + final LTO link had it) — autovectorizing the decode hot loops is an untested lever; (2) the packed repack adds one extra heap memcpy/frame vs tinyh264 (minor scroll handicap). Next: rebuild FFmpeg core with -msimd128 as the decisive test before accepting/rejecting the engine swap.

Follow-up — FFmpeg core rebuilt WITH -msimd128 (--extra-cflags, 2026-06-07): the SIMD autovectorization flipped the scroll result. FFmpeg now beats tinyh264 on all four:

metric	tinyh264	FFmpeg (no SIMD)	FFmpeg +SIMD	vs tinyh264
still p50/p95	10.2 / 22.5	8.7 / 13.4	8.5 / 12.9	p95 −43%
scroll p50/p95	14.8 / 27.2	16.3 / 29.7	11.7 / 25.5	p50 −21%, p95 −6%

→ -msimd128 cut scroll p50 16.3→11.7 (−28%) — the heavy motion-comp/IDCT loops do autovectorize. Gate passes (scroll p95 25.5 < 27.2). Validates the original "WASM SIMD" thesis, but narrowly: needs a decoder whose C loops autovectorize (FFmpeg) plus explicit -msimd128 on the core. Verdict so far: FFmpeg+SIMD is faster everywhere — strongest on still p95 (−43%) and scroll p50 (−21%); scroll p95 (the hardest tail) only −6%. Open cost question: 1.9MB worker vs tinyh264's 173KB (11×). Before committing, test whether tinyh264+-msimd128 (spike b) closes the gap at 1/11 the size — SIMD now demonstrably matters, so b is freshly motivated (was predicted marginal; worth an empirical check).

openh264 instead of b (2026-06-07): skipped b (h264bsd is not perf-engineered → low ceiling) and built openh264 v2.4.1 decoder-only to wasm (--disable asm = scalar C++, all sources -msimd128) as the "good engine + compact" candidate to dethrone FFmpeg. Compiled the decoder/common .cpp directly with em++ (skip the Makefile's platform/asm machinery) + a decode_to_i420 glue (ISVCDecoder → packed I420), reusing the same worker/renderer. Artifact 557KB — 3.4× smaller than FFmpeg's 1.9MB (compact claim ✓). But on speed:

metric	tinyh264 (173KB)	FFmpeg+SIMD (1.9MB)	openh264+SIMD (557KB)
still p50/p95	10.2 / 22.5	8.5 / 12.9	9.2 / 17.8
scroll p50/p95	14.8 / 27.2	11.7 / 25.5	18.4 / 32.9

→ openh264 is the worst of the three on scroll (18.4/32.9 — slower than even tinyh264), better than tinyh264 only on still. The "good engine + compact = best of both" thesis fails empirically: openh264's scalar-C fallback (+-msimd128) doesn't autovectorize/optimize like FFmpeg's, and on heavy motion it trails the simple h264bsd. openh264 eliminated (dominated: bigger than tinyh264 yet slower on scroll; smaller than FFmpeg yet slower everywhere).

Decision reduces to FFmpeg+SIMD vs keeping tinyh264 — a size/maintenance vs speed call: FFmpeg wins decode (still p95 −43%, scroll p50 −21%, scroll p95 −6%) at 11× the bundle (1.9MB, one-time load, non-secure tier only) plus a custom ffmpeg-wasm build to maintain; tinyh264 is 173KB and a plain npm dep. No clear dominant — a product judgment on whether the (mostly still/scroll-p50) latency win is worth the size + build-maintenance cost.

spike b finally measured — tinyh264+-msimd128 (2026-06-07): rebuilt h264bsd with -msimd128 (emsdk 3.1.51), vendored, ?decoder=tinysimd. Result: still 10.8/27.4, scroll 16.5/32.5 — no better than (slightly worse than) the plain npm tinyh264 (10.2/22.5, 14.8/27.2). h264bsd's C does not autovectorize — -msimd128 only added codegen overhead. Confirms the prediction; b eliminated. (Caveat: the npm baseline used a different emsdk, so part of the regression is version noise — but even best-case b ≈ plain tinyh264, nowhere near FFmpeg.) Full four-way (localhost, decode→present ms):

metric	tinyh264 (173KB)	tinyh264+SIMD (159KB)	openh264+SIMD (557KB)	FFmpeg+SIMD (1.9MB)
still p50/p95	10.2 / 22.5	10.8 / 27.4	9.2 / 17.8	8.5 / 12.9
scroll p50/p95	14.8 / 27.2	16.5 / 32.5	18.4 / 32.9	11.7 / 25.5

→ Exhaustive result: FFmpeg+SIMD is the only decoder that beats plain tinyh264 on every metric. Both "compact + fast" hopes failed (openh264 dominated; SIMD doesn't help h264bsd). So the final choice is genuinely binary — FFmpeg+SIMD (fastest, 1.9MB, custom build) vs keep tinyh264 (good-enough, 173KB, npm dep) — with no free lunch in between. Decode bottleneck remains the lever (render isolated at ~0.5ms throughout). Pending: product decision + real-LAN re-measure (these are localhost single-clock; remote LAN runs the decoder on a separate Mac with different CPU headroom, which may widen or narrow the FFmpeg gap).

real-LAN re-measure (W2 — measured 2026-06-07, host MacBook → remote Mac mini, `:3001` cross-machine)

Decoder runs on the Mac mini's CPU over a real LAN hop (glass→glass null = two clocks; decode→present valid). Same ?decoder=ffmpeg vs ?decoder=wasm:

metric	tinyh264	FFmpeg+SIMD	delta
still p50/p95	11.4 / 43.4	10.2 / 33.8	ffmpeg −11% / −22%
scroll p50/p95	15.3 / 53.0	17.6 / 53.3	ffmpeg +15% / ~tie

→ The localhost FFmpeg scroll win did NOT hold on real LAN — it reversed. ffmpeg still wins still (idle smoothness) but is slower on scroll p50 (the stress case) and tied on scroll p95. And both decoders' scroll p95 blew up to ~53ms (vs ~25–27 on localhost) — the real tier1 gap is largely decoder-independent. Read: FFmpeg's -msimd128 win is CPU-dependent (helped the MacBook, not the Mac mini), and remote viewers are uncontrolled heterogeneous hardware. Verdict: W2 does not justify adopting FFmpeg — an idle-only, hardware-dependent win is not worth 11× bundle + a custom build to maintain. Keep tinyh264; do not flip pickDecoder. The ~53ms scroll-p95 (shared by both) points to load-reduction levers (downscale, H.264 static-skip) over a decoder swap. (Caveat: single ~960-frame runs; scroll p50 gap is median-robust, p95 noisier.)

Confirmed — paired 4-round re-measure (2026-06-07, Mac mini, ~950 frames each): variance turned out tiny, so the trade-off is real, not noise. Means over 4 rounds:

metric	tinyh264	FFmpeg+SIMD	winner (4/4 rounds)
still p50/p95	11.3 / 43.9	9.8 / 33.0	ffmpeg (−13% / −25%)
scroll p50/p95	16.6 / 49.9	18.5 / 53.5	tinyh264 (ffmpeg +11% / +7%)

Every round split the same way: ffmpeg wins idle (still), tinyh264 wins motion (scroll) — per-round scroll deltas all favor tinyh264 (p50 −1.1…−2.9, p95 −0.8…−5.0). Per the pre-registered rule (adopt ffmpeg only on a consistent scroll win, given its 11× bundle + custom-build cost), ffmpeg is rejected: it loses the scroll case (the interaction responsiveness that defines "feels-direct") in all 4 rounds. DECISION: keep tinyh264; W1 ffmpeg package removed. (Likely cause of the scroll loss: ffmpeg's repack memcpy + heavier per-frame path on a bandwidth-bound machine vs h264bsd's lean baseline path.) Scroll p95 ~50ms on both (vs ~25 localhost) reconfirms the real gap is load/transport, not the decoder → next levers: H.264 static-skip (iOS) + downscale.

Android bottleneck localize (2026-06-07, localhost, Pixel_9 emulator 1080×2424)

First Android measurement on the shared perf harness (AndroidViewer now uses useDecoderStream → decode→present + recv fps). TAPFLOW_STREAM_METRICS=1 gives agent fpsSent.

phase	agent fpsSent	drop	relay bp	decode→present p50/p95 (WebCodecs)
still	6 → 0	0%	0	2.2 / 3.7 (n=30)
scroll	~22–29 (avg ~25, max 28.8)	0%	0	1.8 / 2.6

→ Android is source/encode-bound, not decode/transport. Idle = 0 fps (scrcpy is surface-driven — already a static-skip equivalent). Under scroll the agent only produces ~22–29 fps (can't sustain the 30 cap) while decode is trivial (1.8ms) and there are zero drops/backpressure — i.e. the emulator's software H.264 encoder + QEMU is the limit on frame production, not the pipeline downstream. (Plus QEMU's own input→render latency, unfixable downstream.) Android lever = reduce the emulator encoder's pixel load — preferably a lower AVD native resolution (no scaling step) over scrcpy max_size (whose earlier 30→4fps backfire was GPU-bound scaling, but under a possibly zombie-polluted session → re-measure clean). Decode/transport levers (a faster decoder, noDelay) would not help Android here.

Android host-encode feasibility spike (W7, 2026-06-07, emulator gRPC streamScreenshot)

Confirmed the emulator has no hardware H.264 encoder (list_encoders: only c2.android.avc.encoder (sw); OMX.google.h264.encoder is an alias of it; h265/av1 also sw). So scrcpy (guest MediaCodec) is capped at the software encoder. Probed the host-side raw-capture path — the emulator gRPC streamScreenshot (relaunched with -grpc 8554, unprotected; what Android Studio's embedded emulator uses) — with continuous scroll, native 1080×2424 RGBA8888:

recvFps 59.6 · producedFps(seq) 60.1 · droppedBeforeUs 5/602 (0.8%)
avgFrame 10.2MB · throughput 596 MB/s (loopback) · interFrame p50 17.4 / p95 19.6 ms

→ Key insight: the emulator renders at 60fps; the ~22–29fps we saw via scrcpy was purely the guest software H.264 encoder. Raw capture over gRPC delivers full 60fps native with ~0% drops (naive bytes transport, no shared-mem needed) on macOS. W7 (capture raw via gRPC → encode on the Mac with VideoToolbox, reusing the iOS encoder) is strongly viable — capture is not a bottleneck, and VT hardware-encodes 1080p@60 trivially. Would make the Android emulator iOS-class (or better, 60fps). Remaining: RGBA→CVPixelBuffer→VTCompressionSession→Annex B→envelope, and the input path (gRPC sendTouch/sendKey vs keep scrcpy control).

Android host-encode — implemented + payoff (W7, 2026-06-07, emulator gRPC → Mac VideoToolbox)

Built the backend: TS EmulatorGrpcClient (grpc-js) captures RGBA via streamScreenshot; pipes to a host-side Swift VT encoder (emulator-encoder, reusing the iOS VT config — baseline, B-off, MaxFrameDelay=0, BT.709, 8Mbps soft) → Annex B → same TFFE envelope as scrcpy. Input via gRPC sendTouch (display-resolution px, top-left origin). Continuous scroll, encoded fps out of the agent:

native 1080×2424   : encodedFps 34.3 · avgFrame 5.3KB · 1.5 Mbps
downscale 712×1600 : encodedFps 59.4 · avgFrame 2.7KB · 1.3 Mbps   (server-side resize, W3 free)

→ scrcpy scroll 22–29fps (guest SW encoder) → gRPC+VT 59.4fps downscaled (2×+, near 60). The native 34fps cap is pixel-volume bound (10MB/frame gRPC recv + pipe + R/B swizzle = the predicted B-architecture copy cost); downscale resolves it. The gRPC streamScreenshot is frame-driven (idle = 0 frames = free static-skip; iOS needs an explicit seed-skip). Orientation: the stream is delivered top-down (proto's "bottom up" note is pre-orientation-transform — verified visually), so no flip; only R↔B swizzle for the BGRA pixel buffer VT wants.

Default policy (2026-06-07): emulators auto-select the gRPC backend (pickAndroidBackend), capped at 30fps (iOS parity) — at 60fps the LAN-HTTP WASM decoder pays 2× decode/transport, the tier1 bottleneck; 30fps halves it. 60fps headroom stays available for localhost/LAN-HTTPS (WebCodecs hw decode) via TAPFLOW_ANDROID_FPS. Real devices keep scrcpy (HW encoder); gRPC failure → scrcpy fallback. Auth: a plain -grpc <port> endpoint is unsecured (localhost, = scrcpy's localhost adb trust); the agent launches emulators with it. The default (no flag) gRPC port requires a token → UNAUTHENTICATED, hence the agent owns the -grpc launch.

5. Work roadmap (gated by measurement)

Each step confirms its effect by measurement before the next. If it fails, roll back/redesign.

#	Step	Goal	Status
0	per-stage latency instrumentation (capture→display, encoder time)	rank bottlenecks by number; analyze localhost-WebCodecs-H.264	✅ done — cause = SPS reorder not declared
1	inject SPS reorder=0 to remove the decoder DPB buffer	bring WebCodecs to localhost-JPEG-class	✅ done — browser-verified (267→2.5ms) + moved into the encoder (ios-agent rewrites the SPS → every decoder benefits; localhost e2e confirms `bitstreamRestriction:true`)
2	WASM decoder + remove MSE → 2-tier (HTTPS=WebCodecs/HTTP=WASM, iOS·Android identical)	zero-buffer, no secure context → LAN low latency	✅ done — PR#1 merged, PR#2 smoke PASS (8.7/14.3ms, both platforms LAN-HTTP WASM). Residual tearing → PR-D below
2.5	relay drop-to-keyframe (PR-D)	on LAN drop, discard P up to the next IDR, removing tearing	✅ implemented + smoke (tearing → short freeze). PR #194
2.6	shorten the freeze = iOS bitrate cap (A) + IDR-on-drop (B)	shrink both ① congestion and ② IDR-wait of the freeze	✅ implemented + verified — A: VT `AverageBitRate` 8Mbps (env; DataRateLimits caused tearing → removed). B: relay→agent `stream:request-idr`→swift stdin `0x01` force IDR. LAN: no tearing, short freeze
2.7	codec negotiation + promote H.264 to default	browsers without a decoder auto-fall back to JPEG → safe to default to H.264	✅ implemented + tested — browser `canDecodeH264()` → `device:boot acceptH264`, agent priority `env=jpeg > capability > default`. Default `TAPFLOW_IOS_CODEC` jpeg→h264 (jpeg opt-out kept). Fallback target ~5% (old browsers without WebGL2). Unit tests ios +5, dashboard +5
3	downscale the encode resolution	a triple win on bandwidth, CPU, latency	✅ implemented + measured — both platforms via `TAPFLOW_MAX_SIZE` (iOS vImage scale; Android gRPC server-side resize). Tuning data + recommended `1280`: downscale-tuning.md. Finding: bandwidth floors at ~1280; below that only decode load drops (fidelity cost). decode→present column pending (viewer-side WASM, manual overlay)
4	event-driven/high-fps capture, optimize the touch path	shave the floor further	☐ #195 (priority: low)

Foundation (done/in progress)

H.264 encoder (VideoToolbox, baseline, MaxFrameDelayCount=0, steady cadence, BT.709) — ios-agent, now the default TAPFLOW_IOS_CODEC=h264 (jpeg opt-out, promoted in 2.7).
envelope codec/keyframe marker (byte5) — for relay keyframe-aware dropping (PR-D, unblock — next step).
decoder layer pickDecoder (WebCodecs/MSE) + IOSViewer shows video directly + WebCodecs multi-NAL decode.

6. How to measure (reproduce)

The full instrumentation reference — every emitter, exact output, the ?perf=1 panel, the ?decoder= override, and the localhost vs cross-machine procedures — lives in measurement.md. This campaign was driven by those tools: agent TAPFLOW_STREAM_METRICS=1 throughput, relay ws backpressure drops, and the ?perf=1 per-stage panel (glass→glass · decode→present · agent→relay) on a localhost single clock.

7. Decision log (accumulated chronologically)

2026-06-08 — LAN scroll tearing has TWO independent causes (the key lesson): While verifying the downscale tiers (localhost/LAN-HTTPS = native, LAN-HTTP = 1280, external = 1000; per-session from the viewer's isSecureContext + relay remote-IP classification), intermittent "breaks vertically then recovers" tearing remained on scroll. It turned out to be two separate bugs that look identical, needing separate fixes:
- ① Orphan P-frame (transport drop). A P-frame dropped under backpressure leaves the next P referencing a frame the decoder never got → the decoder (esp. WASM tinyh264, zero-buffer) shears until the next IDR. Fixed by keyframe-aware backpressure on every drop point: relay (createKeyframeAwareSender, already in place) and both agents' agent→relay pump (Android ddd9eb1, iOS 2c47798) — once dropping, drop the whole GOP until an IDR, and throttled request-idr for fast resync. Reproduced + confirmed with the real tinyh264 in a headless Chrome (not ffmpeg — different decoder, invalid as a proxy): orphan replay shears, drop-to-keyframe replay is clean.
- ② Source IOSurface tear (capture race). Even with ① fixed, iOS still tore — on native tier and on localhost (WebCodecs), which rules out drops, downscale, and the WASM decoder. Root cause: the simulator draws into a single IOSurface in place, asynchronously to our 30fps capture timer; reading it mid-draw bakes a horizontal tear into the H.264/JPEG frame (tier- and decoder-independent, recovers next frame). IOSurfaceLock(.readOnly) is cooperative → does not block the sim's GPU writes, so the existing lock didn't help. Fixed (d37262f) by copySurfaceStable: memcpy the surface to a private buffer bracketed by IOSurfaceGetSeed; if the seed moved during the copy, retry (budget 4). Applied to both encode paths. Evidence: during heavy scroll ~40% of frames raced a write (retries=24/60), all resolved (exhausted=0) — clean replay confirmed, 159 ios tests pass. See ios-agent/AGENTS.md → "Tear-free framebuffer snapshot".
- Methodology lesson: localhost masks LAN-HTTP bugs. localhost is a secure context → WebCodecs (hardware, buffered, forgiving); LAN-HTTP → WASM tinyh264 (zero-buffer, baseline, unforgiving). Decode/transport artifacts that are invisible on localhost surface only on LAN-HTTP. Always reproduce streaming bugs on plain-HTTP LAN with the WASM tier, and verify with the real decoder (headless Chrome on the captured .bin), never a stand-in codec.
2026-06-03 — codec negotiation + promote H.264 to default: WASM brought every path to the goal by measurement, but the encoder default was still JPEG. Simply defaulting to H.264 would give a black screen on browsers without a decoder (the agent sends H.264 one-way, no negotiation). → Negotiate the codec up front first: the browser decides via canDecodeH264() (= the same conditions as pickDecoder, evaluating capability only without creating an instance) and sends acceptH264 in the device:boot payload → the agent decides the codec by priority env=jpeg > acceptH264 > default (stored in DeviceState → same codec on reconnect). A missing field (version skew) = safe JPEG fallback. The relay forwards the payload as unknown (zero changes). Research (caniuse): H.264 possible ~95% (WebGL2 is the floor), fallback needed ~5% (iOS14↓, IE11/legacy Edge, Chrome56↓) — caught deterministically 100% by up-front feature-detect → a runtime decode-failure fallback is over-investment, so Out of Scope. Then promote the default: TAPFLOW_IOS_CODEC default jpeg→h264, =jpeg opt-out kept. The JPEG path (MjpegStreamer, createImageBitmap, relay's JPEG=keyframe assumption) is fully preserved. Unit tests ios +5 (codec-decision branches), dashboard +5 (canDecodeH264); ios 158, dashboard pickDecoder 13 pass; both packages tsc·lint pass. In the PR-D smoke, tearing was gone but a short freeze (felt as lag) remained after a drop. Diagnosis: freeze = ① persistent congestion (no bitrate cap on the iOS encoder → a ~20Mbps scroll burst exceeds the LAN → sustained backpressure) + ② waiting for the next IDR (2s period). A: VideoToolbox AverageBitRate + DataRateLimits cap (TAPFLOW_IOS_H264_BITRATE, default 8Mbps = same as Android) → reduces the congestion itself. B: when the relay's keyframe-aware sender recovers buffer but has no keyframe, onWantKeyframe → the relay sends the agent stream:request-idr (throttle 500ms) → ios-agent → swift stdin 0x01 → forces an IDR on the next frame (reusing the existing ForceKeyFrame) → once congestion clears, re-syncs ~immediately (removing the 2s wait). Android uses the scrcpy protocol so IDR requests aren't supported (the message is ignored, harmless). Unit tests +2 (onWantKeyframe); all packages tsc/lint·tests pass (agent-core 23, relay 168, ios 153). swift recompiled. Device verification (2026-06-03): freeze gone. But DataRateLimits (a hard cap) caused frame corruption (tearing) under high motion → removed it, keeping only AverageBitRate (a soft target). Re-verified: no tearing, short freeze. Lesson: no hard bitrate cap (DataRateLimits), only AverageBitRate.
2026-06-03 — PR-D relay drop-to-keyframe: Fixes the LAN scroll tearing exposed by the WASM (zero-buffer) switch. The relay's drop-to-latest (sendBinaryWithBackpressure) corrupts H.264 up to the next IDR if it drops a P-frame → introduced a per-session createKeyframeAwareSender (agent-core/stream.ts): once it drops under backpressure, it drops everything until the next keyframe (IDR) can be sent → the decoder never receives a P referencing a broken chain (re-syncs via IDR within 1–2s instead of tearing). Keyframe detection uses the envelope byte5 flag (zero NAL parsing). JPEG/no-envelope = always a keyframe → existing drop-to-latest is preserved identically. agent-core unit tests 7 (state machine), relay tsc/168 tests pass. Verification remaining: reproduce LAN scroll (tearing → short freeze). (Optional follow-up: shorten recovery by re-requesting an IDR from the agent on drop via stream:request-idr — currently relies on the periodic IDR.)
2026-06-03 — 2-tier decoder simplification (MSE fully removed, PR#2): Direction settled — HTTPS→WebCodecs / HTTP→WASM, iOS·Android identical pipeline. Deleted MSEDecoder, createJMuxer, jmuxer.d.ts, the jmuxer dependency; pickDecoder drops the createMuxer argument and is 2-tier (secure→WebCodecs, else wasm-capable→WASM, else null). Both viewers call pickDecoder() with no argument. Force Android baseline (ScrcpySession video_codec_options profile:int=1) — required once the MSE safety net is removed since WASM is baseline only. (The pinned OMX.google.h264.encoder is a software encoder so it's likely already baseline — pinning it explicitly crystallizes the decision. Emulator smoke needed to confirm the encoder accepts the profile option.) dashboard 169 tests·tsc·build pass. Smoke PASS (2026-06-03): ① localhost iOS [sps-vui] profileIdc:66 + decode→present 2.3ms held ② scrcpy accepts profile:int=1 and streams fine ③ on plaintext HTTP at :4000, Android (1080×2424) renders fine via WASM = indirect proof of baseline, iOS fine too. Finding: on iOS scroll, relay ws backpressure drops → intermittent screen tearing (WASM has no buffer so drops are more exposed than MSE). Not a regression from this PR = H.264 corruption from relay drop-to-latest. → PR-D (drop-to-keyframe) unblocked (MSE exit complete) = next step.
2026-06-02 — step-2 WASM done + measurement gate PASS (goal reached): PR#1 (R1–R3 + R5 override) implemented — tinyh264@0.0.7 (baseline, I/P only; wasm inlined as a data-URI = no separate asset) + WASMDecoder (worker-encapsulated, transfers the AU as-is) + a new YUVWebGLRenderer (I420 3-texture BT.709 limited) + IOSViewer ?decoder=wasm override. Unit tests 17, full suite 183 pass, vite build worker chunk 173KB verified. localhost ?perf=1&decoder=wasm measured: decode→present still 8.7/p95 30.4, scroll 14.3/p95 37.9; glass→glass still 9.6, scroll 16ms. ~27x·~16x↓ vs MSE (239/229), on par with localhost-JPEG (12.4/9.4) — goal achieved on the non-secure path. Color and feel good, CPU headroom (localhost = worst case). → Q4: downscale confirmed not needed. Next: pickDecoder flip (R4=PR#2) to make WASM the non-secure default.
2026-06-02 — H.264 migration (Phase 2): Confirmed the JPEG full-frame bandwidth problem (still 3.3MB/s, LAN scroll drops 16–27/s) → introduced VideoToolbox H.264. Bandwidth solved ~140x (still) / ~5x (scroll), drops nearly eliminated.
2026-06-02 — but latency discovered: H.264 is slower to respond than JPEG due to the codec pipeline + the LAN MSE buffer. Established localhost-JPEG as the "feels-like-direct" bar. → Redefined the real task as "a low-latency decode/transport path", not "switching to H.264".
2026-06-02 — work begins: Tackle bottlenecks one at a time, measurement-driven. The WASM decoder (tier1 LAN-HTTP) is the key lever. PR-D (drop-to-keyframe) held off given a possible MSE exit.
2026-06-02 — step-2 WASM plan locked (design decision): Wrote the implementation plan (.work/2026-06-02-wasm-tinyh264-decoder-plan.md, local). Key decisions: ① decoder-agnostic boundary — encapsulate the concrete decoder behind WASMDecoder (a Decoder impl) + the new YUVWebGLRenderer. tinyh264 is the first candidate but not a lock-in: since we own the encoder = VideoToolbox baseline, a baseline-only decoder constraint is moot, and at a full-res CPU ceiling we can swap only the decoder for a SIMD build (ffmpeg.wasm/OpenH264) with the renderer/pickDecoder unchanged. ② can't reuse the existing WebGLVideoRenderer — texImage2D(VideoFrame) is WebCodecs=secure-only, but the WASM tier exists precisely for non-secure, and tinyh264 outputs I420 YUV → a new Y/U/V 3-texture BT.709 shader renderer is needed (the latency win comes not from inside the decoder but from the absence of the <video> buffer → structural, decoder-independent). ③ PR = option A (2 PRs) — PR#1 = the whole implementation behind the ?decoder=wasm dev override only (pickDecoder unchanged = zero production impact, gate-measure here), PR#2 = the pickDecoder flip (non-secure → WASM first) only after the gate passes. If R1 (wiring Vite .wasm + worker assets) blocks, split it out first. ④ Q4 (full-res CPU sequencing: downscale/SIMD-swap/rollback) = decided after the R5 gate measurement. → Next: start R1.
2026-06-02 — MSE measured → WASM justified (entering step 2): After the reorder=0 fix applied to every decoder, measured the MSE path on localhost (single clock) via the ?decoder=mse dev override → decode→present p50 ~235ms (vs WebCodecs ~2.5ms). With jmuxer flushingTime:0 already applied, this is the <video> media-element buffer = structural and MSE-inherent (unrelated to SPS/reorder). MSE buffers rather than dropping, so the FIFO tracker is accurate (no drift). → Data confirms tier1 LAN-HTTP can't reach the goal with MSE → started WASM (no media element, no buffer, no secure context needed). dev decoder force: IOSViewer ?decoder=mse|webcodecs (when WASM is added, wasm at the same point).
2026-06-02 — step-1 encoder migration done: Moved the SPS reorder=0 rewrite into ios-agent (agent-core/src/utils/sps.ts canonical, only keyframe SPS via rewriteLowLatencySpsInFrame). The encoder declaring reorder=0 at the source benefits every path — WebCodecs, MSE, and future WASM. localhost e2e confirmed: the browser [sps-vui] shows bitstreamRestriction:true, maxNumReorderFrames:0 (receiving the agent's rewrite), decode→present p50 2.2ms held. The browser copy is a defensive line (no-op). Next (entering step 2): measure residual MSE latency — after pnpm --filter @tapflowio/dashboard dev --host, from the Mac open http://<mac-LAN-IP>:3001?perf=1 (non-secure→MSE, same device→single clock) to measure how much MSE media buffer remains → decide the WASM investment.
2026-06-02 — step-0 analysis done + step-1 verified (key progress): Pinned the cause of the ~267ms localhost-WebCodecs-H.264 latency — transport/relay (~1ms) and input backlog (queueSize=0) are clean, the decoder buffers max DPB (~8 frames). SPS VUI parsing confirmed bitstreamRestriction:false (Level 5.0) → the encoder doesn't tell the decoder "reorder 0", so it assumes the worst. It's baseline so the actual reorder is 0 — a pure missing signal. Fix: WebCodecsCore injects max_num_reorder_frames=0 / max_dec_frame_buffering=num_ref into the SPS just before configure (rewriteSpsLowLatency). Result: decode→present 267→2.5ms (~100x), glass→glass 235→3.5ms, feel reaches localhost-JPEG-class. (Only the WebCodecs path at this point — later widened to all paths via the encoder migration; see the entry above.)
2026-06-02 — step-0 instrumentation begins: Reused the existing perf system (?perf=1, FrameTiming, trace export) (discarding a new sampler). The key gap was the H.264 path logging decodeMs/paintMs=0 — fire-and-forget to the decoder surface left the analysis target in a blind spot. Fix: added onDecodedFrame to Decoder (WebCodecs = output callback, MSE = requestVideoFrameCallback) + a FrameLatencyTracker (submit↔present FIFO correlation, accurate because baseline·B-frame OFF means no reordering) to recover decode→present and glass→glass. glass→glass is epoch present−capturedAt, so it's only computed on a localhost single clock. Protocol/envelope unchanged; the interface change is internal to the dashboard (non-breaking). Measured numbers accumulate in §4.