SIP video

VoIP Stack can optionally turn the Home Assistant softphone card into a SIP video phone for standard SIP phones, softphones and door stations. SIP video is a supported 2026.8.0 capability. Standard ESPHome profiles remain audio-only. Qualified ESP32-P4 profiles can send and receive RTP/JPEG or H.264 video without changing the audio contract.

Video is disabled by default and does not alter an audio-only installation. Open the VoIP Stack integration, choose Reconfigure, then enable SIP video for the Home Assistant softphone. A second step offers two independent capabilities:

• Enable video transcoding lets VoIP Stack use the FFmpeg binary already available to Home Assistant when two negotiated video legs have no direct codec match. This covers legacy receive codecs for the HA softphone and H.264/JPEG conversion between HA-owned SIP legs.
• Allow browser camera transmission exposes a Send Camera control in the card. The logical Home Assistant phone stores that choice in its config subentry and exposes it as a native switch; each attached browser still asks for its own camera permission. Receiving video never requires camera access.

If video setup, decoding, camera permission or transcoding fails, the SIP dialog and browser audio remain active whenever audio negotiation succeeded.

Capability matrix

The browser path keeps H.264 and VP8 encoded media off the HA decoder. JPEG camera frames are encoded in a browser worker. If a browser emits Huffman tables that RFC 2435 cannot carry, HA lazily normalizes those JPEG frames with one bounded FFmpeg worker before RTP packetization.

Negotiated SIP codec	Receive in card	Send browser camera	Server transcode
H.264 Baseline, Main or High	Direct when supported by the browser	Direct for packetization mode 1	To H.264 or JPEG only when the other SIP leg requires it
VP8	Direct	Direct	To H.264 or JPEG only when the other SIP leg requires it
JPEG over RTP	Direct	Direct, with bounded normalization when required	To H.264 or JPEG only when the other SIP leg requires it
H.263	Optional	No	To VP8 for the browser, or H.264/JPEG for a SIP leg
H.263-1998 / H.263-2000	Optional	No	To VP8 for the browser, or H.264/JPEG for a SIP leg
H.265 / HEVC	Optional	No	To VP8 for the browser, or H.264/JPEG for a SIP leg
Audio-only or unsupported video	Audio continues	No	No

ESP32-P4 video endpoints

P4 video is compile-gated. A firmware includes exactly one SIP video codec:

• the VoIP-only JPEG profile uses the hardware JPEG camera path and the P4 JPEG decoder;
• the VoIP-only H.264 profile uses the P4 hardware encoder and the maintained receive decoder path;
• the canonical full-experience profile uses JPEG so video can coexist with AFE, Voice Assistant, Micro Wake Word and the landscape UI.

The full profile also publishes the same JPEG source as a standard ESPHome camera entity. Home Assistant can render that entity through its normal camera platform, while negotiated calls use RTP/JPEG in both directions without a second capture pipeline.

What the P4 sends and displays

The two video directions are independent and can run at the same time:

1. In the outgoing direction, esp_video owns camera capture. The JPEG profile forwards the camera's encoded JPEG frame into RTP without capturing or encoding a duplicate image. The H.264 profile converts the camera frame and uses the ESP32-P4 hardware encoder.
2. In the incoming direction, the VoIP component receives RTP packets, reorders the bounded packet window and reconstructs one complete JPEG or H.264 access unit. Damaged or incomplete frames are discarded instead of being shown later.
3. The codec-specific renderer uses the P4 hardware JPEG decoder for JPEG or Espressif's H.264 software decoder for H.264. PPA performs scaling, rotation and pixel-format conversion where required.
4. The resulting frame is presented on the physical MIPI DSI panel. This is a real receive display path, not the ESPHome camera entity and not evidence inferred from RTP counters.

Presentation follows the video media clock. Capture, decode and display use bounded queues, persistent workers and reusable buffers so that a slow frame cannot create an ever-growing delay. Video work is kept out of the ESPHome main loop, and JPEG and H.264 dependencies remain gated by the selected firmware profile.

The renderer also owns the transition between the normal LVGL interface and the call image. It enters the video view only for an accepted receive stream, can return to audio-only during the same dialog, and restores the idle page and navigation controls after hangup. Qualification therefore checks signaling, RTP/codec processing and physical-panel presentation separately.

Do not enable both JPEG and H.264 in one P4 firmware. The YAML codec choice gates the unused source, decoder, buffers and managed libraries at compile time.

H.264 receive uses the P4 direct display path. display_id is therefore required on p4_video_renderer when codec: h264; ESPHome rejects the YAML at validation time if it is missing. JPEG can present through the LVGL image path or an optional direct display. In either case, receiving RTP packets or decoding frames is not considered display success: the presentation counter or a framebuffer capture must also prove that frames reached the panel.

The profile supports:

• incoming and outgoing calls owned by the Home Assistant softphone;
• sendrecv, sendonly, recvonly and inactive SDP directions;
• one negotiated video stream, using the first accepted m=video section;
• H.264 single NAL units, STAP-A and FU-A;
• VP8 RTP payloads and RFC 2435 JPEG reassembly;
• sequence reordering with bounded queues and damaged-access-unit rejection;
• RTP/AVP by default and RTP/AVPF when offered by the remote endpoint;
• compound RTCP receiver reports, plus negotiated AVPF PLI or FIR key-frame requests;
• symmetric RTP source latching for ordinary SIP devices behind NAT;
• one authenticated browser media owner for the active HA call;
• exact-codec RTP and RTCP relay between two HA-owned standard SIP legs.

The direct bridge changes only the leg-local RTP payload type. Encoded payload, timestamp, sequence, marker, SSRC and RTP extensions remain intact. H.264 packetization mode, profile and RTP transport profile must match, and other codecs must have matching normalized format parameters.

When direct relay is impossible and transcoding is enabled, each active media direction is evaluated independently. HA converts only the incompatible direction to the format negotiated by that receiver. Direct media on the opposite direction stays encoded and unchanged. A known P4 target is offered only its compile-time H.264 or JPEG codec, so its SDP never claims support for the other path.

Direct, trunk and PBX calls

SIP video is not limited to LAN or VPN SIP URIs. The HA softphone offers and accepts the same video profile when a call uses a configured SIP trunk or a video-capable PBX. An authenticated 401 or 407 retry preserves the complete audio/video SDP offer, including media direction and codec parameters.

For inbound trunk video, Direct to default destination reaches the selected phone without digit-collection delay. DTMF extension selection can also preserve video when the initial trunk offer contains a compatible video stream: HA pre-binds and advertises the audio/video ports while collecting digits, then hands those exact sockets to the selected HA browser phone. Selecting an ESP, Assist or another audio-only target releases the unused video resources.

With RTP/AVP there may be no negotiated key-frame feedback. A browser that attaches after DTMF collection can therefore wait until the remote sender's next natural keyframe. VoIP Stack deliberately does not send unnegotiated PLI, FIR or proprietary SIP picture-update messages.

End-to-end success still depends on every provider or PBX in the route keeping the m=video section and forwarding the negotiated RTP/RTCP ports. A normal PSTN leg or an audio-only ITSP may remove video while leaving audio usable. When a PBX bridges two HA instances, exact-codec H.264, VP8 or JPEG can remain passthrough; this integration does not require both homes to share one HA instance.

Dahua door-station interoperability

Dahua UAC firmware commonly advertises PCM/16000 for audio. This is not the network-byte-order L16 payload defined by the RTP standards: captures and existing integrations show signed 16-bit little-endian, mono, 20 ms frames on a dynamic payload type. VoIP Stack recognizes the narrow Dahua UAC/... User-Agent profile and applies that contract only to that peer. It never treats generic PCM as an alias for standard L16.

The same profile is used in both directions:

• a Dahua INVITE can negotiate its exact PCM audio alongside compatible SIP video;
• an HA call to a Dahua endpoint registered over TCP reuses the live REGISTER connection and offers the vendor PCM format first;
• a repeated digest refresh with a stale nonce receives a fresh challenge with stale=true, without weakening replay protection;
• a compatible audio call may add video later with an in-dialog re-INVITE.

There is no public Dahua switch and no ESPHome firmware change. Detection, offer/answer and RTP conversion remain internal to the HA SIP leg. The profile has been qualified against public wire captures and an exact simulated peer, including bidirectional audio/video, re-INVITE and teardown; real-device firmware variants still need user feedback.

Card behavior and privacy

Received video becomes the background of the in-call card. The caller, duration and call state move into a full-width hang-up bar at the bottom so the picture remains the primary content. The layout follows Home Assistant Sections sizing from compact 6-column cards through wide and tall cards. Long caller names are truncated inside the bar rather than widening the page.

Detailed codec, packet and frame counters appear only when VoIP Stack debug mode is enabled. The normal card does not cover the picture with diagnostics.

The tune button beside Hangup selects the microphone, audio output and camera owned by the current browser or Companion App session. Camera replacement is atomic: the existing sender remains active until the replacement track and encoder are ready. The Hangup control keeps its responsive dimensions and pulses subtly from received audio only; it is not driven by the local microphone.

Camera transmission has two gates. The integration-level option must first be enabled, then Send Camera must be enabled on that logical HA phone. The preference survives cache clearing and is shared by every card bound to the same phone; camera permission remains local to the browser that owns media. Permission denial stops only the outgoing camera track. Incoming video, audio and call controls continue independently. Reloading the dashboard during ringing or an established call transfers media ownership to the new card and releases the old WebSocket deterministically.

Direct and transcoded media paths

Direct receive path:

SIP peer RTP -> bounded RTP reorder/depacketizer -> authenticated HA WebSocket
             -> browser WebCodecs or JPEG decoder -> card canvas

Direct camera path:

browser camera -> WebCodecs H.264/VP8 encoder -> authenticated HA WebSocket
               -> RTP packetizer -> SIP peer

browser camera -> browser JPEG worker -> authenticated HA WebSocket
               -> optional bounded JPEG normalizer -> RTP/JPEG packetizer
               -> SIP peer

Optional legacy-codec receive path:

SIP H.263/H.263-1998/H.265 RTP -> local FFmpeg subprocess -> VP8 RTP
                                -> authenticated HA WebSocket -> browser

Optional SIP bridge fallback:

SIP peer A RTP -> local FFmpeg subprocess -> receiver codec RTP -> SIP peer B
SIP peer B RTP -> local FFmpeg subprocess -> receiver codec RTP -> SIP peer A

The second process exists only when the reverse direction is also incompatible. If one direction already has an exact codec contract, that direction remains a direct RTP/RTCP relay.

Adding video to an established audio call

A standard SIP endpoint may begin with audio and later add video with an in-dialog re-INVITE. For a SIP-to-SIP bridge, HA acts as the B2BUA for both dialogs:

1. HA validates the source offer without changing the active audio relay.
2. HA sends a full audio/video re-INVITE to the destination dialog.
3. The destination answer selects its own video codec and direction.
4. HA prepares a direct relay when the codecs match, or the bounded FFmpeg fallback when transcoding is enabled and required.
5. HA answers the source and commits both dialog and relay generations.

If the destination rejects video or preparation fails, HA returns 488 Not Acceptable Here to the new offer and keeps the original audio call. A BYE during setup remains authoritative and releases the staged ports, sockets and transcoder slot. Immediate redial waits for an already stopping transcoder to release that single slot instead of failing the next video offer.

There is no intermediate recording or complete video file. FFmpeg receives and emits RTP continuously on loopback. The transcode path is intentionally bounded to one active call and at most one subprocess per incompatible active direction. Every subprocess uses one encoder thread. Browser VP8 output is limited to 1280x720, 15 frames per second, 700 kbit/s target and 900 kbit/s maximum. SIP H.264 output follows the receiver's level envelope and uses a bounded low-latency baseline profile. SIP JPEG output is limited to 800x800 at 10 frames per second. A second simultaneous transcode call remains audio-only instead of starting unbounded codec workers.

VoIP Stack first uses the FFmpeg binary configured by Home Assistant and then falls back to ffmpeg on the host path. Home Assistant OS and Home Assistant Container already include FFmpeg. Home Assistant Core installations must make it available separately, as described by the Home Assistant FFmpeg integration.

VoIP Stack does not bundle, modify or redistribute FFmpeg or codec libraries. It starts the installation's existing executable only when transcoding was explicitly enabled and is required by a call. FFmpeg builds can be LGPL or GPL depending on their enabled components; distributors of a custom binary remain responsible for that binary under the FFmpeg licensing guidance. This paragraph is informational, not legal advice.

Browser and network requirements

• Use Home Assistant through HTTPS or a browser-recognized secure local context. WebCodecs and camera capture are secure-context features.
• Use a current browser. Codec availability is checked at runtime because it can depend on the browser, operating system and installed media support.
• Permit camera access only when HA must transmit video. A remote sendonly door station needs no HA camera permission.
• Keep SIP and RTP on a trusted LAN or VPN. This profile uses unencrypted SIP/RTP and does not add SRTP, ICE, STUN or TURN.
• Allow the configured HA RTP range through local firewalls. A video call reserves an additional even RTP/odd RTCP pair alongside audio media.

Deliberate limits

This profile does not claim support for:

• video on ESPHome endpoints other than the explicitly qualified ESP32-P4 profiles;
• video through Assist, conference rooms or ring-group legs that traverse standard SIP/RTP endpoints (a local HA browser caller can retain direct browser video when another local browser phone wins the ring group);
• SIP bridge transcode output other than H.264 or JPEG;
• VP9, AV1, MxPEG or other proprietary payloads;
• more than one simultaneous server-side transcode;
• more than one browser owning the same active HA video stream;
• SRTP, DTLS, ICE, STUN or TURN;
• RTCP multiplexing, generic NACK retransmission or bandwidth adaptation;
• call recording or a call-snapshot service. Qualified P4 profiles may expose their camera separately through the normal ESPHome camera entity;
• arbitrary codec replacement on an established video stream. Compatible audio-to-video add/remove is negotiated transactionally through re-INVITE;
• IPv6 RTP media.

Unsupported video is rejected in the SDP answer with a zero media port while a compatible audio section remains active. HA-owned dialogs stage and commit a compatible video add/remove only after every affected dialog accepts it; rejected or stale updates preserve the previous media contract. SIP REFER call transfer is independent of the video codec and uses the normal established-call service.

Qualification

SIP video is covered by the maintained test matrix and has been exercised on real browser-to-browser, HA-to-HA and video-capable trunk calls, including bidirectional media, re-INVITE and cleanup. Qualified ESP32-P4 JPEG and H.264 profiles have also passed real incoming and outgoing bidirectional calls with clean teardown. The cross-codec bridge has passed simultaneous H.264-to-JPEG and JPEG-to-H.264 media with audio in both directions, zero RTP loss in the qualification capture and no retained FFmpeg or call resources after hangup. Compatibility with a particular third-party phone or door station still depends on its exact codec, SDP and RTP behavior.

The protocol work follows RFC 3264 offer/answer, RFC 3550 RTP/RTCP, RFC 4585 RTP/AVPF, RFC 6184 H.264 RTP, RFC 7741 VP8 RTP, RFC 2435 JPEG RTP, RFC 4629 H.263 RTP, RFC 7798 HEVC RTP and RFC 4961 symmetric RTP. Browser media uses the W3C WebCodecs API and the AVC Annex B registration.