How to use the Helios API

The Helios API on Windflow runs Helios as a realtime session. Your server mints a short-lived token, the browser opens a helios session, stages the opening prompt, sends start and draws frames from the session's WebSocket while you keep steering one continuous clip. Everything below is plain fetch and WebSocket, so it works in any framework.

Start from something runnable

The public browser examples provide a local Node server and a plain JavaScript client for LingBot, SANA-Streaming, Helios and LongLive. They keep the account key on the server, request a single-session token capped at 180 seconds, and include an explicit release action. The repository states its verification limits; approved beta access and billable model capacity are required for a live run.

You need an approved Windflow account and an API key that begins with wf_live_ (request access). Requests go to https://api.windflow.online.

1. Check the key (server). The response lists the commands, messages and limitations this page relies on; 401 invalid_api_key means the key is wrong, expired or revoked.

curl https://api.windflow.online/v1/models/helios \
  -H "Authorization: Bearer $WINDFLOW_API_KEY"

2. Mint a browser token (server). The account key never reaches the browser.

// For example app/api/windflow-token/route.ts. Authenticate your own user first.
export async function POST() {
  const res = await fetch("https://api.windflow.online/v1/tokens", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.WINDFLOW_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ model: "helios", max_sessions: 1 }),
  });
  return Response.json(await res.json(), { status: res.status });
}

model binds the token, so using it for another model returns 403 model_mismatch. max_sessions: 1 lets it create one session in total, which gives a leaked token a known worst case. Read expires_at from the response instead of assuming a lifetime.

3. Open the session and draw frames (browser). Save this as windflow-session.js; the JoyAI-Video-Edit, LingBot and ABot-World guides use the same file.

// windflow-session.js: plain fetch and WebSocket, no dependencies
const API = "https://api.windflow.online";

export async function call(path, jwt, body) {
  const res = await fetch(API + path, {
    method: body ? "POST" : "GET",
    headers: { Authorization: `Bearer ${jwt}`, "Content-Type": "application/json" },
    body: body && JSON.stringify(body),
  });
  const data = await res.json();
  if (!res.ok) throw Object.assign(new Error(data.error.message), data.error, { status: res.status });
  return data;
}

export function openSession(model, canvas, onMessage = () => {}) {
  const ctx = canvas.getContext("2d");
  let session, ws, closed = false, released = false;
  let painting = Promise.resolve();

  const paint = async (bytes) => {
    const frame = await createImageBitmap(new Blob([bytes], { type: "image/jpeg" }));
    if (canvas.width !== frame.width) Object.assign(canvas, { width: frame.width, height: frame.height });
    ctx.drawImage(frame, 0, 0);
    frame.close();
  };

  const release = () => {
    closed = true;
    ws?.close();
    if (!session || released) return;
    released = true;
    fetch(`${API}/v1/sessions/${session.session_id}`, {
      method: "POST",
      keepalive: true, // still sent while the page unloads
      headers: { Authorization: `Bearer ${session.jwt}`, "Content-Type": "application/json" },
      body: JSON.stringify({ action: "detach", connection_id: session.connection_id, release: true }),
    });
  };

  const ready = (async () => {
    const { jwt } = await (await fetch("/api/windflow-token", { method: "POST" })).json();
    session = await call("/v1/sessions", jwt, { model, transport: "ws-h264" });
    while (session.status === "queued" && !closed) {
      await new Promise((resolve) => setTimeout(resolve, 2000));
      const poll = await call(`/v1/sessions/${session.session_id}`, session.jwt);
      onMessage({ type: "queued", position: poll.queue_position });
      if (poll.status !== "queued") {
        session = await call("/v1/sessions", session.jwt, { model, session_id: session.session_id, transport: "ws-h264" });
      }
    }
    const offer = session.transport.offers.find((o) => o.kind === "ws-h264");
    if (closed || !offer) throw new Error(closed ? "released" : `no stream offer (status ${session.status})`);
    const url = new URL(offer.relay_path, API);
    url.protocol = "wss:";
    ws = new WebSocket(url);
    ws.binaryType = "arraybuffer";
    return new Promise((resolve, reject) => {
      ws.onopen = () => {
        ws.send(JSON.stringify({ type: "authenticate", token: offer.launch_token }));
        ws.send(JSON.stringify({ type: "client_hello", codecs: ["jpeg"] }));
      };
      ws.onmessage = ({ data }) => {
        if (typeof data !== "string") {
          painting = painting.then(() => paint(data)).catch(console.warn);
          return;
        }
        const msg = JSON.parse(data);
        if (msg.type === "worker_ready") resolve(msg);
        if (msg.type === "failed") reject(new Error(msg.message ?? msg.error));
        onMessage(msg);
      };
      ws.onclose = (event) => {
        onMessage({ type: "socket_closed", code: event.code });
        reject(new Error(`socket closed (${event.code})`));
      };
    });
  })();
  ready.catch(release); // a failed start must not leave a session running

  return {
    ready,
    release,
    get session() { return session; },
    get socket() { return ws; },
    command: (type, fields = {}) =>
      call(`/v1/sessions/${session.session_id}/commands`, session.jwt, { command: { type, ...fields } }),
    upload: async (file) => {
      const intent = await call("/v1/uploads", session.jwt, {
        filename: file.name, mime: file.type, size: file.size, session_id: session.session_id,
      });
      const put = await fetch(intent.upload_url, { method: "PUT", headers: intent.headers, body: file });
      if (!put.ok) throw new Error(`upload failed (${put.status})`);
      return call(`/v1/uploads/${intent.file_id}/complete`, session.jwt, {});
    },
  };
}

queued and socket_closed are messages this file generates; every other message comes from the session.

4. Start Helios.

import { openSession } from "./windflow-session.js";

const helios = openSession("helios", document.querySelector("canvas"), (msg) => console.log(msg.type));
await helios.ready; // worker_ready: commands are accepted from here
await helios.command("set_prompt", { prompt: "A lighthouse on a sea cliff in a storm, slow push-in" });
await helios.command("start");

Wait until the session is ready

Capacity is brought up on demand, so POST /v1/sessions can answer status: "queued" with a queue_position and no transport offers. Poll GET /v1/sessions/{id} every couple of seconds and show the position: polling is what keeps the place, and a queue ticket nobody polls expires with error: "scheduler_queue_expired". Once the status leaves queued, call POST /v1/sessions again with the same session_id to get the WebSocket offer. That refreshes your connection; it does not create a second session.

On the socket, the first message must be authenticate with the offer's launch_token. client_hello with codecs: ["jpeg"] then asks for one JPEG per binary message, and the worker answers worker_ready. A command sent earlier fails with 409 queued or 409 no_active_session. Use the jwt from the latest POST /v1/sessions for every later call; it is scoped to this session and connection.

Send the opening prompt and reference together

Helios needs a prompt before start, because the opening prompt anchors the first chunk; start without one is rejected with "prompt is required". For image-to-video, upload the reference once the session is ready and stage both in one set_conditioning call:

const file = document.querySelector("input[type=file]").files[0];
const image = await helios.upload(file); // POST /v1/uploads, PUT the bytes, POST /v1/uploads/{id}/complete
await helios.command("set_conditioning", { prompt: "The same lighthouse at dawn", image: { id: image.id } });
await helios.command("start");

Uploads accept image/* and video/* files up to 512 MiB. While the stream runs:

prompt_accepted only acknowledges a revision. prompt_applied arrives when the model conditions on it, which is the moment to update your interface.

Track arrived is not first frame visible

Helios delivers its first chunk after roughly one chunk of generation, so a loader that disappears at worker_ready exposes an empty canvas. Start your timer at generation_started: the session's transport.fallback_policy.first_frame_timeout_ms (10 seconds by default) is how long the platform's own client waits for media before treating the stream as failed.

A canvas paints without play(), so autoplay rules do not apply here. If you move the picture into a <video> element, through canvas.captureStream() or the session's livekit offer, set muted and playsInline and handle the promise from play(). NotAllowedError means the browser wants a user gesture: keep the stream, show a play button and call play() from its click handler. AbortError only means a newer play() replaced the old one.

Release the session on unmount

The client that creates a session owns it. Releasing ends it at the worker and hands the capacity back; the response reports session_released: true. Pausing is no substitute, because paused time is billed.

useEffect(() => {
  const helios = openSession("helios", canvasRef.current, onMessage);
  helios.ready
    .then(() => helios.command("set_prompt", { prompt: OPENING_PROMPT }))
    .then(() => helios.command("start"))
    .catch(onError);
  addEventListener("pagehide", helios.release);
  return () => {
    removeEventListener("pagehide", helios.release);
    helios.release();
  };
}, []);

release() sends { action: "detach", connection_id, release: true } with keepalive, so it still goes out while the page unloads. If a tab dies without releasing, the worker keeps the session for a reconnect grace (30 seconds by default) before letting it go. To survive a drop instead, see recovering a dropped stream.

Troubleshooting by the stage where startup stopped

A rejected command returns 422 with a command_… code over HTTP and a command_error message on the socket, both naming the reason. conditions_ready reports has_prompt and has_image before you send start.

FAQ

Can I change the prompt while Helios is streaming?

Yes. set_prompt keeps subjects, layout and camera motion continuous; scene_cut starts independent content. Wait for prompt_applied to know the model has used the change.

Can I use WebRTC instead of the WebSocket?

Yes. Request transport: "livekit" and the session carries a livekit offer with a url and participant_token; the generated track is named main_video. The commands are the same on both transports.

When does billing start, and does pausing stop it?

The session response states that billing starts at generation_started and that paused time is billed, and it gives the ceiling as session_max_seconds and billing.max_usd. Release the session to stop the meter.