# How to use the EdenLive API

> Render a game or live video with the EdenLive API: send greybox frames up one WebSocket, set a scene prompt and visual anchor, and draw the generated stream.

Source: https://windflow.online/guides/edenlive-api · Updated 2026-11-03 · Windflow Labs

The **EdenLive API** on Windflow runs [EdenLive](https://windflow.online/models/edenlive) as a realtime session that renders a video you supply. Your program runs the game, its rules and its physics, and draws a plain proxy picture, often called greybox footage. EdenLive takes those frames, plus a scene prompt and an optional visual anchor, and streams back a generated picture at 1280 × 704 that follows the source's motion. A webcam or an uploaded clip works as the source too.

## Start from something runnable

You need an approved Windflow account and an API key that begins with `wf_live_` ([request access](https://windflow.online/auth?mode=request)). Requests go to `https://api.windflow.online`.

**1. Check the key (server).** The response lists this model's commands, messages and limitations.

```bash
curl https://api.windflow.online/v1/models/edenlive \
  -H "Authorization: Bearer $WINDFLOW_API_KEY"
```

**2. Mint a browser token (server).** The account key never reaches the browser.

```ts
// For example app/api/windflow-token/route.ts. Authenticate your own user first.
export async function POST() {
  const res = await fetch("https://api.windflow.online/v1/tokens", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.WINDFLOW_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ model: "edenlive", max_sessions: 1 }),
  });
  return Response.json(await res.json(), { status: res.status });
}
```

The token is bound to this model (another model gets `403 model_mismatch`) and can create one session. Read `expires_at` from the response instead of assuming a lifetime.

**3. Open the session and draw frames (browser).** Save this as `windflow-session.js`. It is the same file the [Helios guide](https://windflow.online/guides/helios-api) walks through line by line.

```js
// windflow-session.js: plain fetch and WebSocket, no dependencies
const API = "https://api.windflow.online";

export async function call(path, jwt, body) {
  const res = await fetch(API + path, {
    method: body ? "POST" : "GET",
    headers: { Authorization: `Bearer ${jwt}`, "Content-Type": "application/json" },
    body: body && JSON.stringify(body),
  });
  const data = await res.json();
  if (!res.ok) throw Object.assign(new Error(data.error.message), data.error, { status: res.status });
  return data;
}

export function openSession(model, canvas, onMessage = () => {}) {
  const ctx = canvas.getContext("2d");
  let session, ws, closed = false, released = false;
  let painting = Promise.resolve();

  const paint = async (bytes) => {
    const frame = await createImageBitmap(new Blob([bytes], { type: "image/jpeg" }));
    if (canvas.width !== frame.width) Object.assign(canvas, { width: frame.width, height: frame.height });
    ctx.drawImage(frame, 0, 0);
    frame.close();
  };

  const release = () => {
    closed = true;
    ws?.close();
    if (!session || released) return;
    released = true;
    fetch(`${API}/v1/sessions/${session.session_id}`, {
      method: "POST",
      keepalive: true, // still sent while the page unloads
      headers: { Authorization: `Bearer ${session.jwt}`, "Content-Type": "application/json" },
      body: JSON.stringify({ action: "detach", connection_id: session.connection_id, release: true }),
    });
  };

  const ready = (async () => {
    const { jwt } = await (await fetch("/api/windflow-token", { method: "POST" })).json();
    session = await call("/v1/sessions", jwt, { model, transport: "ws-h264" });
    while (session.status === "queued" && !closed) {
      await new Promise((resolve) => setTimeout(resolve, 2000));
      const poll = await call(`/v1/sessions/${session.session_id}`, session.jwt);
      onMessage({ type: "queued", position: poll.queue_position });
      if (poll.status !== "queued") {
        session = await call("/v1/sessions", session.jwt, { model, session_id: session.session_id, transport: "ws-h264" });
      }
    }
    const offer = session.transport.offers.find((o) => o.kind === "ws-h264");
    if (closed || !offer) throw new Error(closed ? "released" : `no stream offer (status ${session.status})`);
    const url = new URL(offer.relay_path, API);
    url.protocol = "wss:";
    ws = new WebSocket(url);
    ws.binaryType = "arraybuffer";
    return new Promise((resolve, reject) => {
      ws.onopen = () => {
        ws.send(JSON.stringify({ type: "authenticate", token: offer.launch_token }));
        ws.send(JSON.stringify({ type: "client_hello", codecs: ["jpeg"] }));
      };
      ws.onmessage = ({ data }) => {
        if (typeof data !== "string") {
          painting = painting.then(() => paint(data)).catch(console.warn);
          return;
        }
        const msg = JSON.parse(data);
        if (msg.type === "worker_ready") resolve(msg);
        if (msg.type === "failed") reject(new Error(msg.message ?? msg.error));
        onMessage(msg);
      };
      ws.onclose = (event) => {
        onMessage({ type: "socket_closed", code: event.code });
        reject(new Error(`socket closed (${event.code})`));
      };
    });
  })();
  ready.catch(release); // a failed start must not leave a session running

  return {
    ready,
    release,
    get session() { return session; },
    get socket() { return ws; },
    command: (type, fields = {}) =>
      call(`/v1/sessions/${session.session_id}/commands`, session.jwt, { command: { type, ...fields } }),
    upload: async (file) => {
      const intent = await call("/v1/uploads", session.jwt, {
        filename: file.name, mime: file.type, size: file.size, session_id: session.session_id,
      });
      const put = await fetch(intent.upload_url, { method: "PUT", headers: intent.headers, body: file });
      if (!put.ok) throw new Error(`upload failed (${put.status})`);
      return call(`/v1/uploads/${intent.file_id}/complete`, session.jwt, {});
    },
  };
}
```

`queued` and `socket_closed` are messages this file generates; every other message comes from the session.

**4. Send your game's frames and start rendering.** In live mode, source frames go up the same socket as JPEG binary messages. Here the source is the `<canvas>` your game draws into. The worker's `worker_ready` says how fast (`cameraFps`) and how wide (`cameraMaxWidth`) it wants frames; sending more only adds backlog.

```js
import { openSession } from "./windflow-session.js";

const eden = openSession("edenlive", document.querySelector("#output"), (msg) => console.log(msg.type));
const hello = await eden.ready;
await eden.command("set_mode", { mode: "live" });
const stopSource = sendCanvas(eden.socket, hello, document.querySelector("#game")); // your greybox game
await eden.command("set_prompt", {
  prompt: "Cinematic photorealism. A volcanic cavern with a glowing lava lake, dark basalt and drifting embers, seen first-person",
});
await eden.command("start"); // retry if it reports the source is still buffering

// Sends a canvas the page already draws, for example a WebGL game.
function sendCanvas(socket, hello, source) {
  const fps = hello.cameraFps || 24;
  const maxWidth = hello.cameraMaxWidth || 1280;
  const frame = document.createElement("canvas");
  let busy = false;
  const timer = setInterval(() => {
    // Skip a tick rather than queue: a stale source frame is rendered late.
    if (busy || socket.bufferedAmount > 1 << 20) return;
    const scale = Math.min(1, maxWidth / source.width);
    frame.width = Math.round(source.width * scale);
    frame.height = Math.round(source.height * scale);
    frame.getContext("2d").drawImage(source, 0, 0, frame.width, frame.height);
    busy = true;
    frame.toBlob(async (blob) => {
      if (blob && socket.readyState === WebSocket.OPEN) socket.send(await blob.arrayBuffer());
      busy = false;
    }, "image/jpeg", 0.85);
  }, 1000 / fps);
  return () => clearInterval(timer);
}
```

A WebGL canvas must be created with `preserveDrawingBuffer: true`, or `drawImage` reads it back blank between frames. To use a webcam as the source instead, capture it with `getUserMedia` the way the [JoyAI-Video-Edit guide](https://windflow.online/guides/joyai-video-edit-api) does.

## Wait until the session is ready

Capacity is brought up on demand, so `POST /v1/sessions` can answer `status: "queued"` with a `queue_position` and no transport offers. Keep polling `GET /v1/sessions/{id}`: a queue ticket nobody polls expires with `error: "scheduler_queue_expired"`. When the status leaves `queued`, `POST /v1/sessions` with the same `session_id` returns the WebSocket offer without creating a second session.

On the socket, `authenticate` with the offer's `launch_token` must be the first message; `client_hello` with `codecs: ["jpeg"]` asks for one JPEG per binary message. Commands are accepted after `worker_ready`. Before it they fail with `409 queued` or `409 no_active_session`.

## Set the scene prompt and the visual anchor

`start` needs two things: a scene prompt, and a source. In live mode that means at least 9 frames buffered from your stream; in file mode, an uploaded clip set with `set_video`. EdenLive generates a 9-frame opening chunk and then 12 frames per chunk.

The opening chunk is generated from a visual anchor. Without one, the first source frame is the anchor, which means the model starts from your greybox look. To open on a finished look instead, upload a reference image, for example a polished render of your game's first frame, and stage it with the prompt:

```js
const look = await eden.upload(document.querySelector("#reference").files[0]); // image/*
await eden.command("set_conditioning", { prompt: SCENE_PROMPT, image: { id: look.id } });
```

`set_conditioning` needs both the prompt and the image. To change only the prompt, use `set_prompt`.

| Command | What it does |
|---|---|
| `set_mode` | `live` for frames on the socket (a game or a webcam), `file` for an uploaded clip; before `start` |
| `set_video` | Sets the uploaded clip in file mode; only before `start`, and the session ends with the clip |
| `set_prompt` | Sets the scene prompt; during a run, a change reopens generation at the next chunk boundary |
| `set_image` | Sets or replaces the visual anchor; during a run, a change reopens generation |
| `set_conditioning` | Prompt and anchor together; both required |
| `start`, `stop`, `reset`, `set_seed` | Run control; a seed change during a run also reopens generation |

### Use changes as beats

A new prompt, anchor or seed during a run takes effect from the next chunk boundary, starting fresh from the current source. Use a change for a deliberate beat, such as entering a new level or a change of weather, rather than something to send every few seconds.

### Movement comes from your source

There are no movement commands. Whatever moves in the source frames moves in the output, so camera motion, character motion and physics belong to your game. Anything drawn into the source passes through the model, text and HUD included, so keep the source to the scene itself and draw your HUD on top of the output canvas.

For how to write the scene prompt, see the [game rendering use case](https://windflow.online/use-cases/restyle-a-game-live-with-ai), which walks through the prompt structure EdenLive's own samples use.

## Track arrived is not first frame visible

| Milestone | Signal | What the player should see |
|---|---|---|
| Ready | `worker_ready` | The greybox game, with a loading state |
| Source accepted | `source_accepted` | Same |
| Generating | `generation_started`, after `start` | Same |
| Stream arrived | `video_config`, then the first binary message | Same |
| First frame visible | The first `drawImage` returns | The rendered game |

Keep the greybox view on screen until the first rendered frame is drawn. `transport.fallback_policy.first_frame_timeout_ms` in the session response (10 seconds by default) is how long the platform's own client waits for media after `generation_started`. To set gameplay timing, measure input-to-display latency on your own setup with the method in [measuring realtime video latency](https://windflow.online/guides/measure-realtime-video-latency).

## Release the session on unmount

Call `stopSource()` and `eden.release()` when the component unmounts and on `pagehide`. Release sends `{ action: "detach", connection_id, release: true }` with `keepalive`; the control plane stops the run and hands the capacity back, and the response reports `session_released: true`. Paused time is billed, so release is what stops the meter. If a tab dies without releasing, the worker keeps the session for a reconnect grace (30 seconds by default).

## Troubleshooting by the stage where startup stopped

| Stage | Symptom | Cause and fix |
|---|---|---|
| Token | `401 invalid_api_key` | Send the `wf_live_` key from your server only |
| Create | `403 model_mismatch` or `token_sessions_exhausted` | Token minted for another model, or its `max_sessions` used up |
| Create | `402 insufficient_balance` or `429 rate_limited` | Balance below the shortest useful session, or too many new sessions; honour `Retry-After` |
| Queue | `failed` with `scheduler_queue_expired` | Polling stopped while queued |
| Socket | Closes with `4401` or `4408` | `authenticate` not sent first; or the session already hit its time limit |
| Ready | `start` rejected: "prompt is required" | Send `set_prompt` or `set_conditioning` first |
| Ready | `start` rejected: the source is still buffering | Fewer than 9 frames have arrived; keep sending and retry |
| Ready | `start` rejected: "No source video set" | File mode needs `set_video` first |
| Ready | Frames arrive blank | The WebGL canvas was created without `preserveDrawingBuffer: true` |
| Running | The output opens looking like greybox | No visual anchor was set; stage a reference image with `set_conditioning` |

Rejected commands return `422` with a `command_…` code over HTTP and a `command_error` message on the socket. `conditions_ready` reports `has_video`, `has_prompt` and `has_image` so you can see what is staged.

## FAQ

### Can I control EdenLive with WASD keys?

Not directly. EdenLive renders the motion in its source. Map the keys in your own game, and the rendered stream follows what the game draws.

### Does EdenLive need a game engine?

It needs a source of frames. That can be a game your page draws, a webcam, or an uploaded clip. A simple greybox scene with clear shapes and motion is enough; the scene prompt and anchor supply the look.

### Is EdenLive the same as Matrix-Game 4.0?

No. Matrix-Game 4.0 renders one of three bundled games with a model trained per scene, and it takes no prompt. EdenLive renders the frames you send it, guided by a prompt you write. The [game rendering use case](https://windflow.online/use-cases/restyle-a-game-live-with-ai) compares the two.