# How to use the SANA-Streaming API

> Edit a live webcam or an uploaded clip with the SANA-Streaming API: one WebSocket for camera and video, an edit prompt, periodic re-anchoring and release.

Source: https://windflow.online/guides/sana-streaming-api · Updated 2026-11-09 · Windflow Labs

The **SANA-Streaming API** on Windflow runs [SANA-Streaming](https://windflow.online/models/sana-streaming) as a realtime session that edits a video you bring, instead of generating one from scratch. Point it at a live webcam or an uploaded clip, describe the change, and it restyles what the prompt names while the rest of the picture, and its motion, stays as it was. A new prompt lands at the next 24-frame chunk boundary, so you can change the edit while the stream runs.

## Start from something runnable

The [public browser examples](https://github.com/windflow-labs/realtime-examples) provide a local Node server and a plain JavaScript client for LingBot, SANA-Streaming, Helios and LongLive. They keep the account key on the server, request a single-session token capped at 180 seconds, and include an explicit release action. The repository states its verification limits; approved beta access and billable model capacity are required for a live run.

You need an approved Windflow account and an API key that begins with `wf_live_` ([request access](https://windflow.online/auth?mode=request)). Requests go to `https://api.windflow.online`.

**1. Check the key (server).** The response lists this model's commands, messages and limitations.

```bash
curl https://api.windflow.online/v1/models/sana-streaming \
  -H "Authorization: Bearer $WINDFLOW_API_KEY"
```

**2. Mint a browser token (server).** The account key never reaches the browser.

```ts
// For example app/api/windflow-token/route.ts. Authenticate your own user first.
export async function POST() {
  const res = await fetch("https://api.windflow.online/v1/tokens", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.WINDFLOW_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ model: "sana-streaming", max_sessions: 1 }),
  });
  return Response.json(await res.json(), { status: res.status });
}
```

The token is bound to this model (another model gets `403 model_mismatch`) and can create one session. Read `expires_at` from the response instead of assuming a lifetime.

**3. Open the session and draw frames (browser).** Save this as `windflow-session.js`. It is the same file the [Helios guide](https://windflow.online/guides/helios-api) walks through line by line.

```js
// windflow-session.js: plain fetch and WebSocket, no dependencies
const API = "https://api.windflow.online";

export async function call(path, jwt, body) {
  const res = await fetch(API + path, {
    method: body ? "POST" : "GET",
    headers: { Authorization: `Bearer ${jwt}`, "Content-Type": "application/json" },
    body: body && JSON.stringify(body),
  });
  const data = await res.json();
  if (!res.ok) throw Object.assign(new Error(data.error.message), data.error, { status: res.status });
  return data;
}

export function openSession(model, canvas, onMessage = () => {}) {
  const ctx = canvas.getContext("2d");
  let session, ws, closed = false, released = false;
  let painting = Promise.resolve();

  const paint = async (bytes) => {
    const frame = await createImageBitmap(new Blob([bytes], { type: "image/jpeg" }));
    if (canvas.width !== frame.width) Object.assign(canvas, { width: frame.width, height: frame.height });
    ctx.drawImage(frame, 0, 0);
    frame.close();
  };

  const release = () => {
    closed = true;
    ws?.close();
    if (!session || released) return;
    released = true;
    fetch(`${API}/v1/sessions/${session.session_id}`, {
      method: "POST",
      keepalive: true, // still sent while the page unloads
      headers: { Authorization: `Bearer ${session.jwt}`, "Content-Type": "application/json" },
      body: JSON.stringify({ action: "detach", connection_id: session.connection_id, release: true }),
    });
  };

  const ready = (async () => {
    const { jwt } = await (await fetch("/api/windflow-token", { method: "POST" })).json();
    session = await call("/v1/sessions", jwt, { model, transport: "ws-h264" });
    while (session.status === "queued" && !closed) {
      await new Promise((resolve) => setTimeout(resolve, 2000));
      const poll = await call(`/v1/sessions/${session.session_id}`, session.jwt);
      onMessage({ type: "queued", position: poll.queue_position });
      if (poll.status !== "queued") {
        session = await call("/v1/sessions", session.jwt, { model, session_id: session.session_id, transport: "ws-h264" });
      }
    }
    const offer = session.transport.offers.find((o) => o.kind === "ws-h264");
    if (closed || !offer) throw new Error(closed ? "released" : `no stream offer (status ${session.status})`);
    const url = new URL(offer.relay_path, API);
    url.protocol = "wss:";
    ws = new WebSocket(url);
    ws.binaryType = "arraybuffer";
    return new Promise((resolve, reject) => {
      ws.onopen = () => {
        ws.send(JSON.stringify({ type: "authenticate", token: offer.launch_token }));
        ws.send(JSON.stringify({ type: "client_hello", codecs: ["jpeg"] }));
      };
      ws.onmessage = ({ data }) => {
        if (typeof data !== "string") {
          painting = painting.then(() => paint(data)).catch(console.warn);
          return;
        }
        const msg = JSON.parse(data);
        if (msg.type === "worker_ready") resolve(msg);
        if (msg.type === "failed") reject(new Error(msg.message ?? msg.error));
        onMessage(msg);
      };
      ws.onclose = (event) => {
        onMessage({ type: "socket_closed", code: event.code });
        reject(new Error(`socket closed (${event.code})`));
      };
    });
  })();
  ready.catch(release); // a failed start must not leave a session running

  return {
    ready,
    release,
    get session() { return session; },
    get socket() { return ws; },
    command: (type, fields = {}) =>
      call(`/v1/sessions/${session.session_id}/commands`, session.jwt, { command: { type, ...fields } }),
    upload: async (file) => {
      const intent = await call("/v1/uploads", session.jwt, {
        filename: file.name, mime: file.type, size: file.size, session_id: session.session_id,
      });
      const put = await fetch(intent.upload_url, { method: "PUT", headers: intent.headers, body: file });
      if (!put.ok) throw new Error(`upload failed (${put.status})`);
      return call(`/v1/uploads/${intent.file_id}/complete`, session.jwt, {});
    },
  };
}
```

`queued` and `socket_closed` are messages this file generates; every other message comes from the session.

**4. Send the webcam and start editing.** In live mode, camera frames go up the same socket as JPEG binary messages. The worker's `worker_ready` says how fast (`cameraFps`) and how wide (`cameraMaxWidth`) it wants them; sending more only adds backlog.

```js
import { openSession } from "./windflow-session.js";

let markSourceReady;
const sourceReady = new Promise((resolve) => (markSourceReady = resolve));
const sana = openSession("sana-streaming", document.querySelector("canvas"), (msg) => {
  if (msg.type === "source_ready") markSourceReady(); // a camera frame reached the worker
  console.log(msg.type);
});

const hello = await sana.ready;
await sana.command("set_mode", { mode: "live" });
const stopCamera = await sendCamera(sana.socket, hello, document.querySelector("#preview"));
await sana.command("set_prompt", { prompt: "Turn the background into a hand-painted watercolor; keep the person unchanged" });
await sourceReady;
await sana.command("start");

// `video` is a visible <video id="preview"> element: the user's own camera preview.
async function sendCamera(socket, hello, video) {
  const fps = hello.cameraFps || 16;
  const maxWidth = hello.cameraMaxWidth || 1280;
  const media = await navigator.mediaDevices.getUserMedia({ video: true });
  Object.assign(video, { muted: true, playsInline: true, srcObject: media });
  await video.play();
  const canvas = document.createElement("canvas");
  let busy = false;
  const timer = setInterval(() => {
    // Skip a tick rather than queue: a stale camera frame is edited late.
    if (busy || !video.videoWidth || socket.bufferedAmount > 1 << 20) return;
    const scale = Math.min(1, maxWidth / video.videoWidth);
    canvas.width = Math.round(video.videoWidth * scale);
    canvas.height = Math.round(video.videoHeight * scale);
    canvas.getContext("2d").drawImage(video, 0, 0, canvas.width, canvas.height);
    busy = true;
    canvas.toBlob(async (blob) => {
      if (blob && socket.readyState === WebSocket.OPEN) socket.send(await blob.arrayBuffer());
      busy = false;
    }, "image/jpeg", 0.85);
  }, 1000 / fps);
  return () => {
    clearInterval(timer);
    media.getTracks().forEach((track) => track.stop());
  };
}
```

Keep the capture size steady for the whole session: do not switch cameras or resize the capture while the stream runs. If you send the camera as a WebRTC track over the session's `livekit` offer instead, publish it as `camera` and set `track.contentHint = "detail"`.

## Wait until the session is ready

Capacity is brought up on demand, so `POST /v1/sessions` can answer `status: "queued"` with a `queue_position` and no transport offers. Keep polling `GET /v1/sessions/{id}`: a queue ticket nobody polls expires with `error: "scheduler_queue_expired"`. When the status leaves `queued`, `POST /v1/sessions` with the same `session_id` returns the WebSocket offer without creating a second session.

On the socket, `authenticate` with the offer's `launch_token` must be the first message; `client_hello` with `codecs: ["jpeg"]` asks for one JPEG per binary message. Commands are accepted after `worker_ready`. Before it they fail with `409 queued` or `409 no_active_session`.

## Edit an uploaded clip instead

File mode edits a clip you upload. The clip needs at least 41 frames, it can only be set before `start`, and the session ends when the clip does.

```js
await sana.command("set_mode", { mode: "file" });
const clip = await sana.upload(document.querySelector("#clip").files[0]); // video/*, up to 512 MiB
await sana.command("set_video", { video: { id: clip.id } });              // answered by video_accepted
await sana.command("set_prompt", { prompt: "Make it a snowy winter street; keep the cars and people moving as they are" });
await sana.command("start");
```

`video_accepted` reports the clip's `width`, `height` and `num_frames`. A file over 100 MiB comes back from `POST /v1/uploads` with `upload_strategy: "multipart"` and a list of `parts`; PUT each part, then send their `part_number` and `etag` to `POST /v1/uploads/{id}/complete`.

| Command | What it does |
|---|---|
| `set_mode` | `live` for the webcam, `file` for an uploaded clip |
| `set_video` | Sets the uploaded clip in file mode, before `start` |
| `set_prompt` | Sets or changes the edit prompt; a change lands at the next chunk boundary |
| `set_anchor_interval` | `intervalChunks` ≥ 0: re-ground generation on the live source every N chunks; 0 turns it off |
| `start`, `stop`, `reset`, `set_seed` | Run control; `reset` clears source, prompt and progress |

## Keep long live streams on the source

`set_anchor_interval` keeps a long live stream close to the camera by re-grounding the model on the live source every N chunks. Each refresh emits `anchored` with its `chunk_index` and `frame_seq`, so a product can add a short cross-fade at that moment for a seamless picture.

```js
await sana.command("set_anchor_interval", { intervalChunks: 8 });
```

Edits are scoped to what the prompt names: parts of the source the prompt does not mention are preserved. For how to word an edit so it changes only what you mean, see [writing edit instructions for realtime video editing](https://windflow.online/guides/realtime-video-edit-instructions).

## Track arrived is not first frame visible

| Milestone | Signal | What the user should see |
|---|---|---|
| Ready | `worker_ready` | Their own camera preview |
| Source ready | `source_ready` | Still the preview |
| Generating | `generation_started`, after `start` | The preview with a working indicator |
| Stream arrived | `video_config`, then the first binary message | Same |
| First edited frame visible | The first `drawImage` returns | The edited stream |

Keep the local camera preview on screen until the first edited frame is drawn, so the user never faces an empty box. `transport.fallback_policy.first_frame_timeout_ms` in the session response (10 seconds by default) is how long the platform's own client waits for media after `generation_started`.

## Release the session on unmount

Call `stopCamera()` and `sana.release()` when the component unmounts and on `pagehide`. Release sends `{ action: "detach", connection_id, release: true }` with `keepalive`; the control plane stops the run and hands the capacity back, and the response reports `session_released: true`. Paused time is billed, so release is what stops the meter. If a tab dies without releasing, the worker keeps the session for a reconnect grace (30 seconds by default). For drops, see [recovering a dropped stream](https://windflow.online/guides/recover-a-dropped-realtime-video-stream).

## Troubleshooting by the stage where startup stopped

| Stage | Symptom | Cause and fix |
|---|---|---|
| Token | `401 invalid_api_key` | Send the `wf_live_` key from your server only |
| Create | `403 model_mismatch` or `token_sessions_exhausted` | Token minted for another model, or its `max_sessions` used up |
| Create | `402 insufficient_balance` or `429 rate_limited` | Balance below the shortest useful session, or too many new sessions; honour `Retry-After` |
| Queue | `failed` with `scheduler_queue_expired` | Polling stopped while queued |
| Socket | Closes with `4401` or `4408` | `authenticate` not sent first; or the session already hit its time limit |
| Camera | `getUserMedia` rejects, or `navigator.mediaDevices` is undefined | Camera permission was denied, or the page is not served over HTTPS |
| Ready | `start` rejected: "Webcam is still buffering" | Not enough camera frames have reached the worker yet; wait for `source_ready` and keep sending |
| Ready | `start` rejected: "No source video set" | File mode needs `set_video` first |
| Ready | `start` rejected for too few source frames | The clip is shorter than the edit window; use a clip of at least 41 frames |
| Live | A refresh every few chunks | That is the re-anchor; cross-fade on `anchored`, or raise `intervalChunks` |
| Live | The picture changes after switching cameras | Keep one capture size for the whole session |

Rejected commands return `422` with a `command_…` code over HTTP and a `command_error` message on the socket. `conditions_ready` reports `has_video` and `has_prompt` so you can see what is staged.

## FAQ

### Does SANA-Streaming generate video from a prompt alone?

No. It is an editing model: it needs a source, either a live camera or an uploaded clip, and it changes what the prompt names. For text-to-video, use [Helios](https://windflow.online/guides/helios-api) or [LongLive 2](https://windflow.online/guides/longlive-2-api).

### How fast does a new edit prompt take effect?

At the next chunk boundary. SANA-Streaming edits in 24-frame chunks, so the running stream switches to the new edit without restarting.

### What is the difference between SANA-Streaming and JoyAI-Video-Edit?

Both edit a live camera or a clip and keep what the instruction does not name. SANA-Streaming applies a new prompt at the next chunk boundary and offers periodic re-anchoring for long live streams. [JoyAI-Video-Edit](https://windflow.online/guides/joyai-video-edit-api) is a larger model that also takes a visual reference, and a new instruction briefly holds the picture while its model session reopens.
