Streaming a video diffusion model to the browser: WebRTC or WebSocket?

To stream a video diffusion model to a browser, generate in chunks, pace the chunks to the playout rate, and deliver them over WebRTC when the network path is clean or over a WebSocket carrying H.264 or JPEG frames when it is lossy or locked down. Both work; the choice is a property of your users' networks, not of the model.

This guide explains why the model's generation pattern drives the transport decision, compares the two options, and lists what to measure.

Generation arrives in chunks, playback does not

An autoregressive video model does not emit frames one at a time at a steady rate. It produces a chunk of frames, then the next chunk, each conditioned on the last. Models in the Windflow catalog generate chunks as short as 8 to 33 frames, and some generate windows of ten seconds or more.

That has three consequences for delivery:

1. Bursty supply. A whole chunk becomes available at once, then nothing until the next chunk is done. The viewer needs a steady 24 frames a second.
2. A playout buffer is mandatory. The client holds a short queue and drains it at the display rate. Too small and the stream stalls between chunks; too large and every control you send feels slow.
3. Time to first frame includes generation. For windowed models the first frame can only appear after the first window is complete, so it is much later than the frames that follow.

If generation is slower than playback, no transport can save the stream. The first requirement is that the model produces a second of video in less than a second, with margin; the transport decides whether that margin survives the network.

Option A: WebRTC

WebRTC carries media over UDP with congestion control, loss recovery and jitter handling built into the browser. Windflow's WebRTC path runs through LiveKit.

Strengths

• Hardware-accelerated decode and a mature jitter buffer in every browser.
• Congestion control that adapts bitrate to the path instead of queueing behind it.
• Audio and video share one clock, which matters for models that generate sound.
• Works well on clean, low-latency links.

Costs

• More moving parts: signalling, ICE, TURN relays for restrictive networks, and a media server.
• On a lossy path, recovery depends on keyframes. A lost reference frame forces a keyframe request and a burst, and the viewer sees a freeze until it lands.
• Harder to debug than a socket.

Option B: WebSocket carrying H.264 or JPEG

A single TCP connection carries encoded frames in one direction and control messages in the other. Windflow exposes this as the ws-h264 transport.

Strengths

• One connection, one port, easy to pass through proxies and firewalls.
• Controls and video share a connection, so ordering is simple.
• In-order delivery makes the client trivial to reason about, and frames can be decoded with the browser's VideoDecoder or drawn from JPEG.
• With intra-only JPEG there are no reference frames to lose, so there is no keyframe recovery to stall on.

Costs

• TCP retransmits lost packets in order, so under loss one late packet delays everything behind it. You need your own pacing and a drop policy.
• No built-in congestion adaptation for media; bitrate has to be managed by the sender.
• JPEG frames use far more bandwidth than H.264 at the same quality.

Which one to choose

Windflow negotiates per session: each model lists the transports it supports in order of preference, and the client takes the first it can use. Seven of the ten catalog models offer both. ABot-World is WebSocket-only, and the two models that generate audio, LTX-2.5 and MiniMax FastH3, are WebRTC-only.

A field result: when WebSocket won

On a lossy China-to-US path, we found that sending JPEG frames over one WebSocket removed stalls that WebRTC's keyframe recovery caused, as long as generation was rate-matched to playout. WebRTC stays the default wherever the link is clean. The write-up is in the field note JPEG over WebSocket beat WebRTC on a lossy trans-Pacific link; it is a decision about one bad path, not a verdict on the transport.

Pace the stream, do not throw frames away

Whatever the transport, two rules hold:

• Rate-match generation to playout. If the model runs ahead, hold the surplus in the buffer instead of dropping generated frames. If it runs behind, slow the clock rather than skip.
• Keep the buffer shallow but not empty. A few hundred milliseconds to a couple of seconds is typical. Measure the stall rate before shrinking it.

Controls have three moments

A prompt change or key press passes through three points, and a good client tells the user about each:

1. Acknowledged: the server received it.
2. Applied: the model started using it at a chunk boundary.
3. Visible: the first frame that reflects it reached the screen.

Acknowledged is not visible. Showing an immediate acknowledgement and then updating when the change is actually on screen keeps a laggy-feeling stream from feeling broken.

What to measure

• Time to first frame after starting a session.
• Delivered frame rate against the target, per session.
• Stalls: count and total duration per minute of viewing.
• Control-to-visible delay: from a control being sent to the frame that shows it.
• Bytes per second, to size the link you ask your users to have.

Measure from the client. Server logs show the frames you sent, not the frames the viewer saw.

FAQ

Should I use WebRTC or WebSocket to stream video from a diffusion model?

Use WebRTC on clean networks and when audio must stay in sync. Use a WebSocket carrying H.264 or JPEG on lossy, long-haul or restricted networks, or when you want controls and frames on one connection. The decision depends on your users' network paths more than on the model.

Why does my generated video stutter even though the model is fast enough?

Chunked generation delivers frames in bursts. Without a playout buffer the stream stalls between chunks, and on a lossy path WebRTC keyframe recovery can freeze the picture. Rate-match generation to playout and keep a shallow buffer.

Can I send controls and receive video on the same connection?

Yes. With the ws-h264 transport, controls travel up the same WebSocket that carries frames down. With WebRTC, media uses the peer connection and controls use the session's control plane.

Which Windflow models support which transports?

Seven of the ten catalog models support both WebSocket (H.264) and WebRTC. ABot-World supports WebSocket only; LTX-2.5 and MiniMax FastH3 support WebRTC only. Each model's page lists its transports; see the model catalog.