How to use the SANA-Streaming API

The SANA-Streaming API on Windflow runs SANA-Streaming as a realtime session that edits a video you bring, instead of generating one from scratch. Point it at a live webcam or an uploaded clip, describe the change, and it restyles what the prompt names while the rest of the picture, and its motion, stays as it was. A new prompt lands at the next 24-frame chunk boundary, so you can change the edit while the stream runs.

Start from something runnable

The public browser examples provide a local Node server and a plain JavaScript client for LingBot, SANA-Streaming, Helios and LongLive. They keep the account key on the server, request a single-session token capped at 180 seconds, and include an explicit release action. The repository states its verification limits; approved beta access and billable model capacity are required for a live run.

You need an approved Windflow account and an API key that begins with wf_live_ (request access). Requests go to https://api.windflow.online.

1. Check the key (server). The response lists this model's commands, messages and limitations.

curl https://api.windflow.online/v1/models/sana-streaming \
  -H "Authorization: Bearer $WINDFLOW_API_KEY"

2. Mint a browser token (server). The account key never reaches the browser.

// For example app/api/windflow-token/route.ts. Authenticate your own user first.
export async function POST() {
  const res = await fetch("https://api.windflow.online/v1/tokens", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.WINDFLOW_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ model: "sana-streaming", max_sessions: 1 }),
  });
  return Response.json(await res.json(), { status: res.status });
}

The token is bound to this model (another model gets 403 model_mismatch) and can create one session. Read expires_at from the response instead of assuming a lifetime.

3. Open the session and draw frames (browser). Save this as windflow-session.js. It is the same file the Helios guide walks through line by line.

// windflow-session.js: plain fetch and WebSocket, no dependencies
const API = "https://api.windflow.online";

export async function call(path, jwt, body) {
  const res = await fetch(API + path, {
    method: body ? "POST" : "GET",
    headers: { Authorization: `Bearer ${jwt}`, "Content-Type": "application/json" },
    body: body && JSON.stringify(body),
  });
  const data = await res.json();
  if (!res.ok) throw Object.assign(new Error(data.error.message), data.error, { status: res.status });
  return data;
}

export function openSession(model, canvas, onMessage = () => {}) {
  const ctx = canvas.getContext("2d");
  let session, ws, closed = false, released = false;
  let painting = Promise.resolve();

  const paint = async (bytes) => {
    const frame = await createImageBitmap(new Blob([bytes], { type: "image/jpeg" }));
    if (canvas.width !== frame.width) Object.assign(canvas, { width: frame.width, height: frame.height });
    ctx.drawImage(frame, 0, 0);
    frame.close();
  };

  const release = () => {
    closed = true;
    ws?.close();
    if (!session || released) return;
    released = true;
    fetch(`${API}/v1/sessions/${session.session_id}`, {
      method: "POST",
      keepalive: true, // still sent while the page unloads
      headers: { Authorization: `Bearer ${session.jwt}`, "Content-Type": "application/json" },
      body: JSON.stringify({ action: "detach", connection_id: session.connection_id, release: true }),
    });
  };

  const ready = (async () => {
    const { jwt } = await (await fetch("/api/windflow-token", { method: "POST" })).json();
    session = await call("/v1/sessions", jwt, { model, transport: "ws-h264" });
    while (session.status === "queued" && !closed) {
      await new Promise((resolve) => setTimeout(resolve, 2000));
      const poll = await call(`/v1/sessions/${session.session_id}`, session.jwt);
      onMessage({ type: "queued", position: poll.queue_position });
      if (poll.status !== "queued") {
        session = await call("/v1/sessions", session.jwt, { model, session_id: session.session_id, transport: "ws-h264" });
      }
    }
    const offer = session.transport.offers.find((o) => o.kind === "ws-h264");
    if (closed || !offer) throw new Error(closed ? "released" : `no stream offer (status ${session.status})`);
    const url = new URL(offer.relay_path, API);
    url.protocol = "wss:";
    ws = new WebSocket(url);
    ws.binaryType = "arraybuffer";
    return new Promise((resolve, reject) => {
      ws.onopen = () => {
        ws.send(JSON.stringify({ type: "authenticate", token: offer.launch_token }));
        ws.send(JSON.stringify({ type: "client_hello", codecs: ["jpeg"] }));
      };
      ws.onmessage = ({ data }) => {
        if (typeof data !== "string") {
          painting = painting.then(() => paint(data)).catch(console.warn);
          return;
        }
        const msg = JSON.parse(data);
        if (msg.type === "worker_ready") resolve(msg);
        if (msg.type === "failed") reject(new Error(msg.message ?? msg.error));
        onMessage(msg);
      };
      ws.onclose = (event) => {
        onMessage({ type: "socket_closed", code: event.code });
        reject(new Error(`socket closed (${event.code})`));
      };
    });
  })();
  ready.catch(release); // a failed start must not leave a session running

  return {
    ready,
    release,
    get session() { return session; },
    get socket() { return ws; },
    command: (type, fields = {}) =>
      call(`/v1/sessions/${session.session_id}/commands`, session.jwt, { command: { type, ...fields } }),
    upload: async (file) => {
      const intent = await call("/v1/uploads", session.jwt, {
        filename: file.name, mime: file.type, size: file.size, session_id: session.session_id,
      });
      const put = await fetch(intent.upload_url, { method: "PUT", headers: intent.headers, body: file });
      if (!put.ok) throw new Error(`upload failed (${put.status})`);
      return call(`/v1/uploads/${intent.file_id}/complete`, session.jwt, {});
    },
  };
}

queued and socket_closed are messages this file generates; every other message comes from the session.

4. Send the webcam and start editing. In live mode, camera frames go up the same socket as JPEG binary messages. The worker's worker_ready says how fast (cameraFps) and how wide (cameraMaxWidth) it wants them; sending more only adds backlog.

import { openSession } from "./windflow-session.js";

let markSourceReady;
const sourceReady = new Promise((resolve) => (markSourceReady = resolve));
const sana = openSession("sana-streaming", document.querySelector("canvas"), (msg) => {
  if (msg.type === "source_ready") markSourceReady(); // a camera frame reached the worker
  console.log(msg.type);
});

const hello = await sana.ready;
await sana.command("set_mode", { mode: "live" });
const stopCamera = await sendCamera(sana.socket, hello, document.querySelector("#preview"));
await sana.command("set_prompt", { prompt: "Turn the background into a hand-painted watercolor; keep the person unchanged" });
await sourceReady;
await sana.command("start");

// `video` is a visible <video id="preview"> element: the user's own camera preview.
async function sendCamera(socket, hello, video) {
  const fps = hello.cameraFps || 16;
  const maxWidth = hello.cameraMaxWidth || 1280;
  const media = await navigator.mediaDevices.getUserMedia({ video: true });
  Object.assign(video, { muted: true, playsInline: true, srcObject: media });
  await video.play();
  const canvas = document.createElement("canvas");
  let busy = false;
  const timer = setInterval(() => {
    // Skip a tick rather than queue: a stale camera frame is edited late.
    if (busy || !video.videoWidth || socket.bufferedAmount > 1 << 20) return;
    const scale = Math.min(1, maxWidth / video.videoWidth);
    canvas.width = Math.round(video.videoWidth * scale);
    canvas.height = Math.round(video.videoHeight * scale);
    canvas.getContext("2d").drawImage(video, 0, 0, canvas.width, canvas.height);
    busy = true;
    canvas.toBlob(async (blob) => {
      if (blob && socket.readyState === WebSocket.OPEN) socket.send(await blob.arrayBuffer());
      busy = false;
    }, "image/jpeg", 0.85);
  }, 1000 / fps);
  return () => {
    clearInterval(timer);
    media.getTracks().forEach((track) => track.stop());
  };
}

Keep the capture size steady for the whole session: do not switch cameras or resize the capture while the stream runs. If you send the camera as a WebRTC track over the session's livekit offer instead, publish it as camera and set track.contentHint = "detail".

Wait until the session is ready

Capacity is brought up on demand, so POST /v1/sessions can answer status: "queued" with a queue_position and no transport offers. Keep polling GET /v1/sessions/{id}: a queue ticket nobody polls expires with error: "scheduler_queue_expired". When the status leaves queued, POST /v1/sessions with the same session_id returns the WebSocket offer without creating a second session.

On the socket, authenticate with the offer's launch_token must be the first message; client_hello with codecs: ["jpeg"] asks for one JPEG per binary message. Commands are accepted after worker_ready. Before it they fail with 409 queued or 409 no_active_session.

Edit an uploaded clip instead

File mode edits a clip you upload. The clip needs at least 41 frames, it can only be set before start, and the session ends when the clip does.

await sana.command("set_mode", { mode: "file" });
const clip = await sana.upload(document.querySelector("#clip").files[0]); // video/*, up to 512 MiB
await sana.command("set_video", { video: { id: clip.id } });              // answered by video_accepted
await sana.command("set_prompt", { prompt: "Make it a snowy winter street; keep the cars and people moving as they are" });
await sana.command("start");

video_accepted reports the clip's width, height and num_frames. A file over 100 MiB comes back from POST /v1/uploads with upload_strategy: "multipart" and a list of parts; PUT each part, then send their part_number and etag to POST /v1/uploads/{id}/complete.

Keep long live streams on the source

set_anchor_interval keeps a long live stream close to the camera by re-grounding the model on the live source every N chunks. Each refresh emits anchored with its chunk_index and frame_seq, so a product can add a short cross-fade at that moment for a seamless picture.

await sana.command("set_anchor_interval", { intervalChunks: 8 });

Edits are scoped to what the prompt names: parts of the source the prompt does not mention are preserved. For how to word an edit so it changes only what you mean, see writing edit instructions for realtime video editing.

Track arrived is not first frame visible

Keep the local camera preview on screen until the first edited frame is drawn, so the user never faces an empty box. transport.fallback_policy.first_frame_timeout_ms in the session response (10 seconds by default) is how long the platform's own client waits for media after generation_started.

Release the session on unmount

Call stopCamera() and sana.release() when the component unmounts and on pagehide. Release sends { action: "detach", connection_id, release: true } with keepalive; the control plane stops the run and hands the capacity back, and the response reports session_released: true. Paused time is billed, so release is what stops the meter. If a tab dies without releasing, the worker keeps the session for a reconnect grace (30 seconds by default). For drops, see recovering a dropped stream.

Troubleshooting by the stage where startup stopped

Rejected commands return 422 with a command_… code over HTTP and a command_error message on the socket. conditions_ready reports has_video and has_prompt so you can see what is staged.

FAQ

Does SANA-Streaming generate video from a prompt alone?

No. It is an editing model: it needs a source, either a live camera or an uploaded clip, and it changes what the prompt names. For text-to-video, use Helios or LongLive 2.

How fast does a new edit prompt take effect?

At the next chunk boundary. SANA-Streaming edits in 24-frame chunks, so the running stream switches to the new edit without restarting.

What is the difference between SANA-Streaming and JoyAI-Video-Edit?

Both edit a live camera or a clip and keep what the instruction does not name. SANA-Streaming applies a new prompt at the next chunk boundary and offers periodic re-anchoring for long live streams. JoyAI-Video-Edit is a larger model that also takes a visual reference, and a new instruction briefly holds the picture while its model session reopens.