How realtime video models work: autoregressive diffusion, chunks and the latency budget
Why standard video diffusion cannot run live, what autoregressive chunked generation and few-step distillation change, where the latency goes, and how to read a realtime claim.
A realtime video model is a diffusion model reorganised into a loop. Instead of denoising a whole clip over dozens of steps, it produces a small chunk of frames in a few denoising steps, conditioned on what it already generated, hands the chunk to the viewer, accepts new input, and repeats. It is realtime when producing a chunk takes less time than playing it.
This guide explains what changes from standard video diffusion, where the delay between a control and a visible result comes from, and how to judge a realtime claim.
Why standard video diffusion cannot run live
A conventional text-to-video model generates all frames of a clip together. Every frame can attend to every other frame, and the model refines the whole clip over dozens of denoising steps. That design produces coherent clips, but it has two properties a live stream cannot tolerate:
- Nothing is ready until everything is. The first frame cannot be shown until the whole clip is denoised.
- The cost is high per second of video. Dozens of passes over every frame make generation slower than playback on a single GPU.
What changes in a realtime model
Four ideas turn that design into a stream.
1. Autoregressive, chunked generation
The model generates a few frames at a time, each chunk conditioned on the ones before it. Attention is causal: a chunk looks at the past, not the future. That is what lets a stream continue indefinitely and accept new input between chunks. Catalog models use chunks from 8 to 33 frames, and some produce windows of ten seconds or more.
2. A key-value cache for the past
To avoid recomputing the history for every chunk, the model keeps the attention keys and values of earlier frames in a cache. A rolling cache drops the oldest frames to bound memory, which is why very old content can be forgotten.
3. Few-step distillation
A distilled model learns to reach in a handful of steps what the original reaches in many. Techniques named in the realtime video literature include Self-Forcing, which trains the model on its own autoregressive output so that it behaves well when it runs open-loop, and distribution-matching distillation such as DMD2. For example, Matrix-Game 4.0 redraws each four-frame block of a running game in a single denoising step, and MiniMax FastH3 is distilled to four diffusion-transformer calls.
4. A serving stack that keeps up
Algorithms get a model close; engineering gets it over the line. Fast attention kernels, compiled graphs, sparse attention, quantisation, sequence parallelism across GPUs and a fast decoder are all part of staying ahead of playback. They are also why the same model can be realtime on one platform and not on another.
The latency budget
Two delays matter, and they are different.
- Time to first frame: from starting a session to the first frame on screen. For windowed models it includes generating a whole window.
- Control latency: from sending a control to seeing its effect.
Control latency is a sum:
- Waiting for the next boundary. A control lands at the next chunk or window boundary. On average that is half a chunk; at worst a full one.
- Generating the next chunk.
- Decoding and encoding the frames.
- Network transit.
- The client's playout buffer.
As an illustration, a 12-frame chunk at 24 frames per second is half a second of video. A control waits on average a quarter of a second for the boundary, then the chunk must be generated, encoded, sent and buffered. Shrinking the chunk shortens the first term; it also gives the model less to work with per step, so chunk size is a design trade-off, not a free lever.
What gets harder in a long run
- Drift. Each chunk inherits the previous chunk's small errors, so identity, colour and geometry wander over minutes. Models counter it with anchoring, memory and periodic re-conditioning.
- Memory. A rolling cache forgets; a world can change when you turn around.
- Controls take effect at boundaries, so movement is responsive but not frame-exact.
- Resolution versus speed. Higher resolution costs more per chunk. Learned super-resolution can generate at a lower resolution and deliver more detail.
How to read a "realtime" claim
Ask for the numbers behind the word:
| Question | Why it matters |
|---|---|
| What resolution, and at what frame rate, sustained? | A burst is not a stream. |
| On which hardware, and how many GPUs? | Cost and availability depend on it. |
| How many denoising steps? | It is the main speed lever. |
| Is the frame rate native or interpolated? | Interpolation can double a figure without doing more generation. |
| What is the time to first frame? | Windowed models can be slow to start. |
| What is the control latency, and how was it measured? | It decides how interactive it feels. |
| How long was the run? | Short tests hide drift. |
Windflow's measurement report states hardware, software versions, resolution and method next to each number for this reason.
FAQ
How does realtime video generation work?
A realtime model generates video in small chunks with causal attention and a cache of past frames, uses a distilled model that needs only a few denoising steps, and accepts new input between chunks. It is realtime when each chunk is produced faster than it plays back.
What is autoregressive video diffusion?
Autoregressive video diffusion generates a video chunk by chunk, with each chunk conditioned on the earlier ones, instead of generating every frame at once. It allows a stream of unbounded length and input between chunks.
Why is realtime video harder than image generation?
A viewer needs a steady stream, 24 or more frames per second, that stays coherent from one frame to the next, so the model has to be fast and consistent over time, not just good on a single image.
What limits how interactive a model feels?
The wait for the next chunk boundary, the generation time of the chunk, the delivery path to the viewer and the playout buffer. The glossary defines each term.
Which realtime video models can I try?
Windflow hosts ten, from continuous long video to navigable worlds and live editing. See the model catalog and how to choose one.