How to recover a realtime video stream after a dropped connection

When a realtime video stream stops, decide first whether the connection dropped or the session ended. Retry a dropped connection on the same session inside a bounded grace, without calling anything that ends the session, then restore playback from the worker's state snapshot. An ended session is over: say why, and let the user choose whether to start a new one.

The examples use Windflow's WebSocket transport (ws-h264) and the call helper from windflow-session.js in the Helios, JoyAI-Video-Edit, LingBot and ABot-World guides. The same rules apply to WebRTC.

Dropped connection or ended session?

Read the last messages and the close code before you reconnect.

When in doubt, ask the control plane. GET /v1/sessions/{id} returns status, where ended, failed and canceled are final, and error says why.

// `lastEnd` is the last session_ended or generation_ended message,
// cleared again when a generation_started arrives.
function classifyClose(event, lastEnd) {
  if (lastEnd || event.code === 4408) return "ended";
  if (event.code === 4409) return "taken";
  if (event.code === 4401) return "refresh";
  return "retry";
}

Retry inside a bounded grace

Two clocks bound the retries.

• The client grace is transport.fallback_policy.reconnect_grace_ms in the session response, 9 seconds by default. Inside it, reopen the same offer: its launch token is still valid and the worker still holds the session. Back off 100, 200, 400 and 800 ms, then every 1.5 seconds. Also stop after six sockets in a row that close without a single frame, because a worker that accepts and then drops would otherwise reset your timer forever.
• The worker grace is how long the worker keeps a detached session before releasing it, 30 seconds by default. Past the client grace, ask for fresh offers on the same session and connection: POST /v1/sessions with session_id and connection_id, using the session jwt. Reattaching is not a new session, so it is not rate-limited and does not use up a token's max_sessions.

Time the socket spends detached inside the worker grace is not billed.

import { call } from "./windflow-session.js";

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
const WORKER_GRACE_MS = 30_000; // the worker's default; the session response does not carry it
let inFlight = null;

// Wire ws.onclose (when classifyClose says "retry" or "refresh"), the window
// "online" event and your retry button to this. A second call joins the first.
export function recover(stream) {
  // Sockets in a row that closed before any frame arrived; reset it when a frame paints.
  if (stream.framelessCloses >= 6) return Promise.resolve("lost");
  inFlight ??= reconnect(stream).finally(() => (inFlight = null));
  return inFlight;
}

async function reconnect(stream) {
  const startedAt = performance.now();
  const graceMs = stream.session.transport.fallback_policy?.reconnect_grace_ms ?? 9_000;

  // 1. The same offer and launch token, with a short backoff.
  for (let attempt = 0; performance.now() - startedAt < graceMs; attempt++) {
    await sleep(Math.min(1_500, 100 * 2 ** Math.min(attempt, 4)));
    const outcome = await stream.reopen(); // "ready", or classifyClose() of a socket that closed first
    if (outcome === "ready") return "resumed";
    if (outcome === "ended" || outcome === "taken") return outcome;
    if (outcome === "refresh") break;
  }

  // 2. Fresh offers for the same session and connection, until the worker's grace is spent.
  while (performance.now() - startedAt < WORKER_GRACE_MS) {
    try {
      const fresh = await call("/v1/sessions", stream.session.jwt, {
        model: stream.session.model,
        session_id: stream.session.session_id,
        connection_id: stream.session.connection_id,
        transport: "ws-h264",
      });
      if (["ended", "failed", "canceled"].includes(fresh.status)) return "ended";
      stream.session = fresh; // a new jwt and launch token, the same session_id
      if (fresh.status === "queued") return "queued"; // wait for placement as at startup
      const outcome = await stream.reopen();
      if (outcome === "ready") return "resumed";
      if (outcome !== "retry") return outcome === "refresh" ? "lost" : outcome; // refused twice, ended or taken
    } catch (error) {
      if (error.code === "token_expired") {
        stream.session.jwt = await sessionToken(stream.session.session_id);
        continue;
      }
    }
    await sleep(1_500); // most likely still offline
  }
  return "lost";
}

Here stream is your own object: stream.session holds the latest POST /v1/sessions response, stream.socket the open socket and stream.command() sends commands as in the model guides. stream.reopen() repeats the socket setup from startup on the current ws-h264 offer, authenticate with the launch_token and then client_hello, and resolves "ready" on worker_ready or with the classification of a socket that closed first. sessionToken() asks your server for a token bound to this session, described below.

resumed continues with the restore steps below and queued waits for placement as at startup. ended, taken and lost mean this session is not coming back: say so, and offer a new session.

Do not end the session while you retry

• Do not send { action: "detach", release: true } or DELETE /v1/sessions/{id}?connection_id=…&release=true before or during a retry. Release ends the session for good.
• Keep the player mounted while it recovers. A component that unmounts on a "disconnected" state and releases in its cleanup ends the very session you are trying to reach.
• Do not send stop or reset to clean up: stop ends the run, and reset throws away the session's staged state.
• Do not open a new session as the retry. It is a second billed session, new sessions are rate-limited (429 rate_limited with Retry-After), and a token minted with max_sessions may have none left.
• Keep session_id, connection_id and the latest session jwt. If that JWT has expired (401 token_expired), have your server call POST /v1/tokens with { "session_id": "sess_…" }. That token is bound to this one session and cannot create another.

Restore playback after reconnecting

On a reattach inside the worker grace, the worker sends worker_ready and then a state message whose data is a snapshot of the session, and generation carries on with the model state it had. Rebuild the client around it:

1. Put the loading state back and keep it until the next frame is painted, exactly as at startup.
2. Send the current controls, not their history. A key released during the drop never reached the worker, so send one set_input_state with the keys held now, often none. The snapshot's input_seq is the last one the worker holds; number yours above it.
3. Read the snapshot. If running is false although the stream was running, nothing is generating, usually because the session came back on a fresh worker: re-send the latest of each setup command, such as set_mode, set_video or set_conditioning, then start, and pause if the user had paused. Do not replay one-shot commands such as trigger_event, schedule_prompt or scene_cut, and do not replay held keys: a fresh worker starts with neutral movement.
4. If the picture is in a <video> element, call play() again and handle NotAllowedError with a play button.

const setup = new Map(); // command type -> latest fields, e.g. "set_conditioning" -> { prompt, image }
let streaming = false;   // true once frames were showing, false after the user stops
let userPaused = false;

// Call with `data` from the first `state` message after worker_ready on a reopened socket.
async function restore(snapshot) {
  // LingBot and ABot-World: the keys held now, usually none.
  inputSeq = Math.max(inputSeq, snapshot.input_seq ?? 0);
  stream.socket.send(JSON.stringify({ type: "set_input_state", input_seq: ++inputSeq, pressed_keys: [...held] }));
  if (snapshot.running || !streaming) return; // same worker, same state: nothing to re-send
  for (const [type, fields] of setup) await stream.command(type, fields);
  await stream.command("start");
  if (userPaused) await stream.command("pause");
}

Give each command a command_id and reuse it when you retry that command. The worker remembers recent ids and drops the duplicate; a rejection for ENGINE_ERROR or INTERNAL is not remembered, so the same id gets a real second attempt.

If you use WebRTC

On the session's livekit offer, the WebRTC client reports that it is reconnecting before it gives up; treat its final disconnect with the same rules. To move a live session from one transport to the other instead, call POST /v1/sessions/{id}/transport with to, the from_rev you last saw in transport.rev, and an optional reason. A stale from_rev answers 409, and repeating a switch that already happened is safe.

Test checklist

Run each test on the kind of network your users have, with the browser's network panel open.

1. Short drop. Turn the network off for about 3 seconds. Expect the same session_id, no new POST /v1/sessions call, worker_ready and a state message on the new socket, and frames that resume.
2. Long drop. Turn it off for about 15 seconds, then for about 45. At 15 seconds, expect a POST /v1/sessions refresh with the same session_id and connection_id once the network is back, and a new launch token. At 45 seconds, past the worker's default grace, the session has normally ended: expect a final status and an interface that says so without opening a new session on its own.
3. Expired token. Make the refresh fail with 401 token_expired, by waiting past the session JWT's expires_at or by stubbing the response. Expect your server to mint a token with session_id and the same session to resume.
4. Double-clicked retry. While disconnected, click retry twice quickly. Expect one socket, one refresh, one connection when you POST /v1/sessions/{id} with { "action": "list" }, and a retried command applied once.

FAQ

Why does the stream reconnect but stay black?

The socket is back but no frame has been painted. Check the state snapshot: a paused session sends no new frames, and a session on a fresh worker needs its setup commands and start again.

Is the time spent reconnecting billed?

Time the socket spends detached inside the worker's reconnect grace is not billed. Once the client reattaches, the meter runs as before, and paused time counts.

Should the app reconnect without asking the user?

Inside the graces, yes: it is the same session. Once the session has ended, ask before starting a new one, because that opens a new billed session.