How to use the EdenLive API

The EdenLive API on Windflow runs EdenLive as a realtime session that renders a video you supply. Your program runs the game, its rules and its physics, and draws a plain proxy picture, often called greybox footage. EdenLive takes those frames, plus a scene prompt and an optional visual anchor, and streams back a generated picture at 1280 × 704 that follows the source's motion. A webcam or an uploaded clip works as the source too.

Start from something runnable

You need an approved Windflow account and an API key that begins with wf_live_ (request access). Requests go to https://api.windflow.online.

1. Check the key (server). The response lists this model's commands, messages and limitations.

curl https://api.windflow.online/v1/models/edenlive \
  -H "Authorization: Bearer $WINDFLOW_API_KEY"

2. Mint a browser token (server). The account key never reaches the browser.

// For example app/api/windflow-token/route.ts. Authenticate your own user first.
export async function POST() {
  const res = await fetch("https://api.windflow.online/v1/tokens", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.WINDFLOW_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ model: "edenlive", max_sessions: 1 }),
  });
  return Response.json(await res.json(), { status: res.status });
}

The token is bound to this model (another model gets 403 model_mismatch) and can create one session. Read expires_at from the response instead of assuming a lifetime.

3. Open the session and draw frames (browser). Save this as windflow-session.js. It is the same file the Helios guide walks through line by line.

// windflow-session.js: plain fetch and WebSocket, no dependencies
const API = "https://api.windflow.online";

export async function call(path, jwt, body) {
  const res = await fetch(API + path, {
    method: body ? "POST" : "GET",
    headers: { Authorization: `Bearer ${jwt}`, "Content-Type": "application/json" },
    body: body && JSON.stringify(body),
  });
  const data = await res.json();
  if (!res.ok) throw Object.assign(new Error(data.error.message), data.error, { status: res.status });
  return data;
}

export function openSession(model, canvas, onMessage = () => {}) {
  const ctx = canvas.getContext("2d");
  let session, ws, closed = false, released = false;
  let painting = Promise.resolve();

  const paint = async (bytes) => {
    const frame = await createImageBitmap(new Blob([bytes], { type: "image/jpeg" }));
    if (canvas.width !== frame.width) Object.assign(canvas, { width: frame.width, height: frame.height });
    ctx.drawImage(frame, 0, 0);
    frame.close();
  };

  const release = () => {
    closed = true;
    ws?.close();
    if (!session || released) return;
    released = true;
    fetch(`${API}/v1/sessions/${session.session_id}`, {
      method: "POST",
      keepalive: true, // still sent while the page unloads
      headers: { Authorization: `Bearer ${session.jwt}`, "Content-Type": "application/json" },
      body: JSON.stringify({ action: "detach", connection_id: session.connection_id, release: true }),
    });
  };

  const ready = (async () => {
    const { jwt } = await (await fetch("/api/windflow-token", { method: "POST" })).json();
    session = await call("/v1/sessions", jwt, { model, transport: "ws-h264" });
    while (session.status === "queued" && !closed) {
      await new Promise((resolve) => setTimeout(resolve, 2000));
      const poll = await call(`/v1/sessions/${session.session_id}`, session.jwt);
      onMessage({ type: "queued", position: poll.queue_position });
      if (poll.status !== "queued") {
        session = await call("/v1/sessions", session.jwt, { model, session_id: session.session_id, transport: "ws-h264" });
      }
    }
    const offer = session.transport.offers.find((o) => o.kind === "ws-h264");
    if (closed || !offer) throw new Error(closed ? "released" : `no stream offer (status ${session.status})`);
    const url = new URL(offer.relay_path, API);
    url.protocol = "wss:";
    ws = new WebSocket(url);
    ws.binaryType = "arraybuffer";
    return new Promise((resolve, reject) => {
      ws.onopen = () => {
        ws.send(JSON.stringify({ type: "authenticate", token: offer.launch_token }));
        ws.send(JSON.stringify({ type: "client_hello", codecs: ["jpeg"] }));
      };
      ws.onmessage = ({ data }) => {
        if (typeof data !== "string") {
          painting = painting.then(() => paint(data)).catch(console.warn);
          return;
        }
        const msg = JSON.parse(data);
        if (msg.type === "worker_ready") resolve(msg);
        if (msg.type === "failed") reject(new Error(msg.message ?? msg.error));
        onMessage(msg);
      };
      ws.onclose = (event) => {
        onMessage({ type: "socket_closed", code: event.code });
        reject(new Error(`socket closed (${event.code})`));
      };
    });
  })();
  ready.catch(release); // a failed start must not leave a session running

  return {
    ready,
    release,
    get session() { return session; },
    get socket() { return ws; },
    command: (type, fields = {}) =>
      call(`/v1/sessions/${session.session_id}/commands`, session.jwt, { command: { type, ...fields } }),
    upload: async (file) => {
      const intent = await call("/v1/uploads", session.jwt, {
        filename: file.name, mime: file.type, size: file.size, session_id: session.session_id,
      });
      const put = await fetch(intent.upload_url, { method: "PUT", headers: intent.headers, body: file });
      if (!put.ok) throw new Error(`upload failed (${put.status})`);
      return call(`/v1/uploads/${intent.file_id}/complete`, session.jwt, {});
    },
  };
}

queued and socket_closed are messages this file generates; every other message comes from the session.

4. Send your game's frames and start rendering. In live mode, source frames go up the same socket as JPEG binary messages. Here the source is the <canvas> your game draws into. The worker's worker_ready says how fast (cameraFps) and how wide (cameraMaxWidth) it wants frames; sending more only adds backlog.

import { openSession } from "./windflow-session.js";

const eden = openSession("edenlive", document.querySelector("#output"), (msg) => console.log(msg.type));
const hello = await eden.ready;
await eden.command("set_mode", { mode: "live" });
const stopSource = sendCanvas(eden.socket, hello, document.querySelector("#game")); // your greybox game
await eden.command("set_prompt", {
  prompt: "Cinematic photorealism. A volcanic cavern with a glowing lava lake, dark basalt and drifting embers, seen first-person",
});
await eden.command("start"); // retry if it reports the source is still buffering

// Sends a canvas the page already draws, for example a WebGL game.
function sendCanvas(socket, hello, source) {
  const fps = hello.cameraFps || 24;
  const maxWidth = hello.cameraMaxWidth || 1280;
  const frame = document.createElement("canvas");
  let busy = false;
  const timer = setInterval(() => {
    // Skip a tick rather than queue: a stale source frame is rendered late.
    if (busy || socket.bufferedAmount > 1 << 20) return;
    const scale = Math.min(1, maxWidth / source.width);
    frame.width = Math.round(source.width * scale);
    frame.height = Math.round(source.height * scale);
    frame.getContext("2d").drawImage(source, 0, 0, frame.width, frame.height);
    busy = true;
    frame.toBlob(async (blob) => {
      if (blob && socket.readyState === WebSocket.OPEN) socket.send(await blob.arrayBuffer());
      busy = false;
    }, "image/jpeg", 0.85);
  }, 1000 / fps);
  return () => clearInterval(timer);
}

A WebGL canvas must be created with preserveDrawingBuffer: true, or drawImage reads it back blank between frames. To use a webcam as the source instead, capture it with getUserMedia the way the JoyAI-Video-Edit guide does.

Wait until the session is ready

Capacity is brought up on demand, so POST /v1/sessions can answer status: "queued" with a queue_position and no transport offers. Keep polling GET /v1/sessions/{id}: a queue ticket nobody polls expires with error: "scheduler_queue_expired". When the status leaves queued, POST /v1/sessions with the same session_id returns the WebSocket offer without creating a second session.

On the socket, authenticate with the offer's launch_token must be the first message; client_hello with codecs: ["jpeg"] asks for one JPEG per binary message. Commands are accepted after worker_ready. Before it they fail with 409 queued or 409 no_active_session.

Set the scene prompt and the visual anchor

start needs two things: a scene prompt, and a source. In live mode that means at least 9 frames buffered from your stream; in file mode, an uploaded clip set with set_video. EdenLive generates a 9-frame opening chunk and then 12 frames per chunk.

The opening chunk is generated from a visual anchor. Without one, the first source frame is the anchor, which means the model starts from your greybox look. To open on a finished look instead, upload a reference image, for example a polished render of your game's first frame, and stage it with the prompt:

const look = await eden.upload(document.querySelector("#reference").files[0]); // image/*
await eden.command("set_conditioning", { prompt: SCENE_PROMPT, image: { id: look.id } });

set_conditioning needs both the prompt and the image. To change only the prompt, use set_prompt.

Use changes as beats

A new prompt, anchor or seed during a run takes effect from the next chunk boundary, starting fresh from the current source. Use a change for a deliberate beat, such as entering a new level or a change of weather, rather than something to send every few seconds.

Movement comes from your source

There are no movement commands. Whatever moves in the source frames moves in the output, so camera motion, character motion and physics belong to your game. Anything drawn into the source passes through the model, text and HUD included, so keep the source to the scene itself and draw your HUD on top of the output canvas.

For how to write the scene prompt, see the game rendering use case, which walks through the prompt structure EdenLive's own samples use.

Track arrived is not first frame visible

Keep the greybox view on screen until the first rendered frame is drawn. transport.fallback_policy.first_frame_timeout_ms in the session response (10 seconds by default) is how long the platform's own client waits for media after generation_started. To set gameplay timing, measure input-to-display latency on your own setup with the method in measuring realtime video latency.

Release the session on unmount

Call stopSource() and eden.release() when the component unmounts and on pagehide. Release sends { action: "detach", connection_id, release: true } with keepalive; the control plane stops the run and hands the capacity back, and the response reports session_released: true. Paused time is billed, so release is what stops the meter. If a tab dies without releasing, the worker keeps the session for a reconnect grace (30 seconds by default).

Troubleshooting by the stage where startup stopped

Rejected commands return 422 with a command_… code over HTTP and a command_error message on the socket. conditions_ready reports has_video, has_prompt and has_image so you can see what is staged.

FAQ

Can I control EdenLive with WASD keys?

Not directly. EdenLive renders the motion in its source. Map the keys in your own game, and the rendered stream follows what the game draws.

Does EdenLive need a game engine?

It needs a source of frames. That can be a game your page draws, a webcam, or an uploaded clip. A simple greybox scene with clear shapes and motion is enough; the scene prompt and anchor supply the look.

Is EdenLive the same as Matrix-Game 4.0?

No. Matrix-Game 4.0 renders one of three bundled games with a model trained per scene, and it takes no prompt. EdenLive renders the frames you send it, guided by a prompt you write. The game rendering use case compares the two.