# How to use the Helios API

> Use the Helios API from a browser: mint a token on your server, wait for ready, stage the opening prompt, draw the live stream and release the session.

Source: https://windflow.online/guides/helios-api · Updated 2026-11-09 · Windflow Labs

The **Helios API** on Windflow runs [Helios](https://windflow.online/models/helios) as a realtime session. Your server mints a short-lived token, the browser opens a `helios` session, stages the opening prompt, sends `start` and draws frames from the session's WebSocket while you keep steering one continuous clip. Everything below is plain `fetch` and `WebSocket`, so it works in any framework.

## Start from something runnable

The [public browser examples](https://github.com/windflow-labs/realtime-examples) provide a local Node server and a plain JavaScript client for LingBot, SANA-Streaming, Helios and LongLive. They keep the account key on the server, request a single-session token capped at 180 seconds, and include an explicit release action. The repository states its verification limits; approved beta access and billable model capacity are required for a live run.

You need an approved Windflow account and an API key that begins with `wf_live_` ([request access](https://windflow.online/auth?mode=request)). Requests go to `https://api.windflow.online`.

**1. Check the key (server).** The response lists the commands, messages and limitations this page relies on; `401 invalid_api_key` means the key is wrong, expired or revoked.

```bash
curl https://api.windflow.online/v1/models/helios \
  -H "Authorization: Bearer $WINDFLOW_API_KEY"
```

**2. Mint a browser token (server).** The account key never reaches the browser.

```ts
// For example app/api/windflow-token/route.ts. Authenticate your own user first.
export async function POST() {
  const res = await fetch("https://api.windflow.online/v1/tokens", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.WINDFLOW_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ model: "helios", max_sessions: 1 }),
  });
  return Response.json(await res.json(), { status: res.status });
}
```

`model` binds the token, so using it for another model returns `403 model_mismatch`. `max_sessions: 1` lets it create one session in total, which gives a leaked token a known worst case. Read `expires_at` from the response instead of assuming a lifetime.

**3. Open the session and draw frames (browser).** Save this as `windflow-session.js`; the [JoyAI-Video-Edit](https://windflow.online/guides/joyai-video-edit-api), [LingBot](https://windflow.online/guides/lingbot-api) and [ABot-World](https://windflow.online/guides/abot-world-api) guides use the same file.

```js
// windflow-session.js: plain fetch and WebSocket, no dependencies
const API = "https://api.windflow.online";

export async function call(path, jwt, body) {
  const res = await fetch(API + path, {
    method: body ? "POST" : "GET",
    headers: { Authorization: `Bearer ${jwt}`, "Content-Type": "application/json" },
    body: body && JSON.stringify(body),
  });
  const data = await res.json();
  if (!res.ok) throw Object.assign(new Error(data.error.message), data.error, { status: res.status });
  return data;
}

export function openSession(model, canvas, onMessage = () => {}) {
  const ctx = canvas.getContext("2d");
  let session, ws, closed = false, released = false;
  let painting = Promise.resolve();

  const paint = async (bytes) => {
    const frame = await createImageBitmap(new Blob([bytes], { type: "image/jpeg" }));
    if (canvas.width !== frame.width) Object.assign(canvas, { width: frame.width, height: frame.height });
    ctx.drawImage(frame, 0, 0);
    frame.close();
  };

  const release = () => {
    closed = true;
    ws?.close();
    if (!session || released) return;
    released = true;
    fetch(`${API}/v1/sessions/${session.session_id}`, {
      method: "POST",
      keepalive: true, // still sent while the page unloads
      headers: { Authorization: `Bearer ${session.jwt}`, "Content-Type": "application/json" },
      body: JSON.stringify({ action: "detach", connection_id: session.connection_id, release: true }),
    });
  };

  const ready = (async () => {
    const { jwt } = await (await fetch("/api/windflow-token", { method: "POST" })).json();
    session = await call("/v1/sessions", jwt, { model, transport: "ws-h264" });
    while (session.status === "queued" && !closed) {
      await new Promise((resolve) => setTimeout(resolve, 2000));
      const poll = await call(`/v1/sessions/${session.session_id}`, session.jwt);
      onMessage({ type: "queued", position: poll.queue_position });
      if (poll.status !== "queued") {
        session = await call("/v1/sessions", session.jwt, { model, session_id: session.session_id, transport: "ws-h264" });
      }
    }
    const offer = session.transport.offers.find((o) => o.kind === "ws-h264");
    if (closed || !offer) throw new Error(closed ? "released" : `no stream offer (status ${session.status})`);
    const url = new URL(offer.relay_path, API);
    url.protocol = "wss:";
    ws = new WebSocket(url);
    ws.binaryType = "arraybuffer";
    return new Promise((resolve, reject) => {
      ws.onopen = () => {
        ws.send(JSON.stringify({ type: "authenticate", token: offer.launch_token }));
        ws.send(JSON.stringify({ type: "client_hello", codecs: ["jpeg"] }));
      };
      ws.onmessage = ({ data }) => {
        if (typeof data !== "string") {
          painting = painting.then(() => paint(data)).catch(console.warn);
          return;
        }
        const msg = JSON.parse(data);
        if (msg.type === "worker_ready") resolve(msg);
        if (msg.type === "failed") reject(new Error(msg.message ?? msg.error));
        onMessage(msg);
      };
      ws.onclose = (event) => {
        onMessage({ type: "socket_closed", code: event.code });
        reject(new Error(`socket closed (${event.code})`));
      };
    });
  })();
  ready.catch(release); // a failed start must not leave a session running

  return {
    ready,
    release,
    get session() { return session; },
    get socket() { return ws; },
    command: (type, fields = {}) =>
      call(`/v1/sessions/${session.session_id}/commands`, session.jwt, { command: { type, ...fields } }),
    upload: async (file) => {
      const intent = await call("/v1/uploads", session.jwt, {
        filename: file.name, mime: file.type, size: file.size, session_id: session.session_id,
      });
      const put = await fetch(intent.upload_url, { method: "PUT", headers: intent.headers, body: file });
      if (!put.ok) throw new Error(`upload failed (${put.status})`);
      return call(`/v1/uploads/${intent.file_id}/complete`, session.jwt, {});
    },
  };
}
```

`queued` and `socket_closed` are messages this file generates; every other message comes from the session.

**4. Start Helios.**

```js
import { openSession } from "./windflow-session.js";

const helios = openSession("helios", document.querySelector("canvas"), (msg) => console.log(msg.type));
await helios.ready; // worker_ready: commands are accepted from here
await helios.command("set_prompt", { prompt: "A lighthouse on a sea cliff in a storm, slow push-in" });
await helios.command("start");
```

## Wait until the session is ready

Capacity is brought up on demand, so `POST /v1/sessions` can answer `status: "queued"` with a `queue_position` and no transport offers. Poll `GET /v1/sessions/{id}` every couple of seconds and show the position: polling is what keeps the place, and a queue ticket nobody polls expires with `error: "scheduler_queue_expired"`. Once the status leaves `queued`, call `POST /v1/sessions` again with the same `session_id` to get the WebSocket offer. That refreshes your connection; it does not create a second session.

On the socket, the first message must be `authenticate` with the offer's `launch_token`. `client_hello` with `codecs: ["jpeg"]` then asks for one JPEG per binary message, and the worker answers `worker_ready`. A command sent earlier fails with `409 queued` or `409 no_active_session`. Use the `jwt` from the latest `POST /v1/sessions` for every later call; it is scoped to this session and connection.

## Send the opening prompt and reference together

Helios needs a prompt before `start`, because the opening prompt anchors the first chunk; `start` without one is rejected with "prompt is required". For image-to-video, upload the reference once the session is ready and stage both in one `set_conditioning` call:

```js
const file = document.querySelector("input[type=file]").files[0];
const image = await helios.upload(file); // POST /v1/uploads, PUT the bytes, POST /v1/uploads/{id}/complete
await helios.command("set_conditioning", { prompt: "The same lighthouse at dawn", image: { id: image.id } });
await helios.command("start");
```

Uploads accept `image/*` and `video/*` files up to 512 MiB. While the stream runs:

| Command | What it does |
|---|---|
| `set_prompt` | Steers the running scene and keeps its memory; a legacy `transition: "hard"` is treated as soft |
| `scene_cut` | Clears scene memory and starts independent content from a new prompt |
| `schedule_prompt` | Applies a prompt at a future `chunk` index |
| `set_image`, `set_conditioning` | Replace the reference image, alone or with the prompt |
| `set_image_strength` | `image_strength` from 0.0 to 1.0; lower values allow more motion |
| `set_sr_scale` | Delivery super-resolution: `off`, `2x` or `4x`, from the next chunk boundary |
| `pause`, `resume`, `reset`, `set_seed` | Run control |

`prompt_accepted` only acknowledges a revision. `prompt_applied` arrives when the model conditions on it, which is the moment to update your interface.

## Track arrived is not first frame visible

| Milestone | Signal | What the viewer should see |
|---|---|---|
| Ready | `worker_ready` | Preparing, controls enabled |
| Generating | `generation_started`, after `start` | Generating, no picture yet |
| Stream arrived | `video_config`, then the first binary message | Still the loader |
| First frame visible | The first `drawImage` returns | The picture |

Helios delivers its first chunk after roughly one chunk of generation, so a loader that disappears at `worker_ready` exposes an empty canvas. Start your timer at `generation_started`: the session's `transport.fallback_policy.first_frame_timeout_ms` (10 seconds by default) is how long the platform's own client waits for media before treating the stream as failed.

A canvas paints without `play()`, so autoplay rules do not apply here. If you move the picture into a `<video>` element, through `canvas.captureStream()` or the session's `livekit` offer, set `muted` and `playsInline` and handle the promise from `play()`. `NotAllowedError` means the browser wants a user gesture: keep the stream, show a play button and call `play()` from its click handler. `AbortError` only means a newer `play()` replaced the old one.

## Release the session on unmount

The client that creates a session owns it. Releasing ends it at the worker and hands the capacity back; the response reports `session_released: true`. Pausing is no substitute, because paused time is billed.

```jsx
useEffect(() => {
  const helios = openSession("helios", canvasRef.current, onMessage);
  helios.ready
    .then(() => helios.command("set_prompt", { prompt: OPENING_PROMPT }))
    .then(() => helios.command("start"))
    .catch(onError);
  addEventListener("pagehide", helios.release);
  return () => {
    removeEventListener("pagehide", helios.release);
    helios.release();
  };
}, []);
```

`release()` sends `{ action: "detach", connection_id, release: true }` with `keepalive`, so it still goes out while the page unloads. If a tab dies without releasing, the worker keeps the session for a reconnect grace (30 seconds by default) before letting it go. To survive a drop instead, see [recovering a dropped stream](https://windflow.online/guides/recover-a-dropped-realtime-video-stream).

## Troubleshooting by the stage where startup stopped

| Stage | Symptom | Cause and fix |
|---|---|---|
| Token | `401 missing_api_key` or `invalid_api_key` | Send the key from your server as `Authorization: Bearer wf_live_…` or `Windflow-API-Key` |
| Create | `403 model_mismatch` | The token was minted for another model |
| Create | `403 token_sessions_exhausted` | The token's `max_sessions` is used up; mint one token per start |
| Create | `402 insufficient_balance` | The balance cannot cover the model's shortest useful session |
| Create | `429 rate_limited` | New sessions are rate-limited per account; wait for `Retry-After` |
| Queue | `queued`, then `failed` with `scheduler_queue_expired` | Polling stopped, often in a background tab |
| Socket | Closes with `4401` | `authenticate` was not the first message, or the launch token is for another session |
| Socket | Closes with `4408` | The session already reached its time limit; open a new one |
| Ready | `start` rejected: "prompt is required" | Stage the prompt with `set_prompt` or `set_conditioning` first |
| Generating | No binary messages after `generation_started` | Allow one chunk; past the timeout, read `status` and `error` from `GET /v1/sessions/{id}` |
| Visible | Binary messages arrive, nothing shows | The canvas has no size in the layout, or `play()` was refused on a `<video>` |

A rejected command returns `422` with a `command_…` code over HTTP and a `command_error` message on the socket, both naming the reason. `conditions_ready` reports `has_prompt` and `has_image` before you send `start`.

## FAQ

### Can I change the prompt while Helios is streaming?

Yes. `set_prompt` keeps subjects, layout and camera motion continuous; `scene_cut` starts independent content. Wait for `prompt_applied` to know the model has used the change.

### Can I use WebRTC instead of the WebSocket?

Yes. Request `transport: "livekit"` and the session carries a `livekit` offer with a `url` and `participant_token`; the generated track is named `main_video`. The commands are the same on both transports.

### When does billing start, and does pausing stop it?

The session response states that billing starts at `generation_started` and that paused time is billed, and it gives the ceiling as `session_max_seconds` and `billing.max_usd`. Release the session to stop the meter.