# How to use the JoyAI Video Edit API

> Edit a live webcam or an uploaded clip with the JoyAI-Video-Edit API: server token, one WebSocket for camera and video, the edit instruction and release.

Source: https://windflow.online/guides/joyai-video-edit-api · Updated 2026-11-03 · Windflow Labs

The **JoyAI Video Edit API** on Windflow runs [JoyAI-Video-Edit](https://windflow.online/models/joyai-video-edit) as a realtime session that edits a source you provide: a live webcam or an uploaded clip. You give an edit instruction, optionally with a reference image, and the edited stream comes back with the source's motion and identity preserved. Your server mints the token; the browser sends camera frames up and receives edited frames down on one WebSocket.

## Start from something runnable

You need an approved Windflow account and an API key that begins with `wf_live_` ([request access](https://windflow.online/auth?mode=request)). Requests go to `https://api.windflow.online`.

**1. Check the key (server).** The response lists this model's commands, messages and limitations.

```bash
curl https://api.windflow.online/v1/models/joyai-video-edit \
  -H "Authorization: Bearer $WINDFLOW_API_KEY"
```

**2. Mint a browser token (server).** The account key never reaches the browser.

```ts
// For example app/api/windflow-token/route.ts. Authenticate your own user first.
export async function POST() {
  const res = await fetch("https://api.windflow.online/v1/tokens", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.WINDFLOW_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ model: "joyai-video-edit", max_sessions: 1 }),
  });
  return Response.json(await res.json(), { status: res.status });
}
```

The token is bound to this model (another model gets `403 model_mismatch`) and can create one session. Read `expires_at` from the response instead of assuming a lifetime.

**3. Open the session and draw frames (browser).** Save this as `windflow-session.js`; the [Helios](https://windflow.online/guides/helios-api), [LingBot](https://windflow.online/guides/lingbot-api) and [ABot-World](https://windflow.online/guides/abot-world-api) guides use the same file.

```js
// windflow-session.js: plain fetch and WebSocket, no dependencies
const API = "https://api.windflow.online";

export async function call(path, jwt, body) {
  const res = await fetch(API + path, {
    method: body ? "POST" : "GET",
    headers: { Authorization: `Bearer ${jwt}`, "Content-Type": "application/json" },
    body: body && JSON.stringify(body),
  });
  const data = await res.json();
  if (!res.ok) throw Object.assign(new Error(data.error.message), data.error, { status: res.status });
  return data;
}

export function openSession(model, canvas, onMessage = () => {}) {
  const ctx = canvas.getContext("2d");
  let session, ws, closed = false, released = false;
  let painting = Promise.resolve();

  const paint = async (bytes) => {
    const frame = await createImageBitmap(new Blob([bytes], { type: "image/jpeg" }));
    if (canvas.width !== frame.width) Object.assign(canvas, { width: frame.width, height: frame.height });
    ctx.drawImage(frame, 0, 0);
    frame.close();
  };

  const release = () => {
    closed = true;
    ws?.close();
    if (!session || released) return;
    released = true;
    fetch(`${API}/v1/sessions/${session.session_id}`, {
      method: "POST",
      keepalive: true, // still sent while the page unloads
      headers: { Authorization: `Bearer ${session.jwt}`, "Content-Type": "application/json" },
      body: JSON.stringify({ action: "detach", connection_id: session.connection_id, release: true }),
    });
  };

  const ready = (async () => {
    const { jwt } = await (await fetch("/api/windflow-token", { method: "POST" })).json();
    session = await call("/v1/sessions", jwt, { model, transport: "ws-h264" });
    while (session.status === "queued" && !closed) {
      await new Promise((resolve) => setTimeout(resolve, 2000));
      const poll = await call(`/v1/sessions/${session.session_id}`, session.jwt);
      onMessage({ type: "queued", position: poll.queue_position });
      if (poll.status !== "queued") {
        session = await call("/v1/sessions", session.jwt, { model, session_id: session.session_id, transport: "ws-h264" });
      }
    }
    const offer = session.transport.offers.find((o) => o.kind === "ws-h264");
    if (closed || !offer) throw new Error(closed ? "released" : `no stream offer (status ${session.status})`);
    const url = new URL(offer.relay_path, API);
    url.protocol = "wss:";
    ws = new WebSocket(url);
    ws.binaryType = "arraybuffer";
    return new Promise((resolve, reject) => {
      ws.onopen = () => {
        ws.send(JSON.stringify({ type: "authenticate", token: offer.launch_token }));
        ws.send(JSON.stringify({ type: "client_hello", codecs: ["jpeg"] }));
      };
      ws.onmessage = ({ data }) => {
        if (typeof data !== "string") {
          painting = painting.then(() => paint(data)).catch(console.warn);
          return;
        }
        const msg = JSON.parse(data);
        if (msg.type === "worker_ready") resolve(msg);
        if (msg.type === "failed") reject(new Error(msg.message ?? msg.error));
        onMessage(msg);
      };
      ws.onclose = (event) => {
        onMessage({ type: "socket_closed", code: event.code });
        reject(new Error(`socket closed (${event.code})`));
      };
    });
  })();
  ready.catch(release); // a failed start must not leave a session running

  return {
    ready,
    release,
    get session() { return session; },
    get socket() { return ws; },
    command: (type, fields = {}) =>
      call(`/v1/sessions/${session.session_id}/commands`, session.jwt, { command: { type, ...fields } }),
    upload: async (file) => {
      const intent = await call("/v1/uploads", session.jwt, {
        filename: file.name, mime: file.type, size: file.size, session_id: session.session_id,
      });
      const put = await fetch(intent.upload_url, { method: "PUT", headers: intent.headers, body: file });
      if (!put.ok) throw new Error(`upload failed (${put.status})`);
      return call(`/v1/uploads/${intent.file_id}/complete`, session.jwt, {});
    },
  };
}
```

`queued` and `socket_closed` are messages this file generates; every other message comes from the session.

**4. Send the webcam and start editing.** In live mode, camera frames go up the same socket as JPEG binary messages. The worker's `worker_ready` says how fast (`cameraFps`) and how wide (`cameraMaxWidth`) it wants them; sending more only adds backlog.

```js
import { openSession } from "./windflow-session.js";

let markSourceReady;
const sourceReady = new Promise((resolve) => (markSourceReady = resolve));
const joy = openSession("joyai-video-edit", document.querySelector("canvas"), (msg) => {
  if (msg.type === "source_ready") markSourceReady(); // a camera frame reached the worker
  console.log(msg.type);
});

const hello = await joy.ready;
await joy.command("set_mode", { mode: "live" });
const stopCamera = await sendCamera(joy.socket, hello, document.querySelector("#preview"));
await joy.command("set_prompt", { prompt: "Change the jacket to red leather" });
await sourceReady;
await joy.command("start");

// `video` is a visible <video id="preview"> element: the user's own camera preview.
async function sendCamera(socket, hello, video) {
  const fps = hello.cameraFps || 16;
  const maxWidth = hello.cameraMaxWidth || 1280;
  const media = await navigator.mediaDevices.getUserMedia({ video: true });
  Object.assign(video, { muted: true, playsInline: true, srcObject: media });
  await video.play();
  const canvas = document.createElement("canvas");
  let busy = false;
  const timer = setInterval(() => {
    // Skip a tick rather than queue: a stale camera frame is edited late.
    if (busy || !video.videoWidth || socket.bufferedAmount > 1 << 20) return;
    const scale = Math.min(1, maxWidth / video.videoWidth);
    canvas.width = Math.round(video.videoWidth * scale);
    canvas.height = Math.round(video.videoHeight * scale);
    canvas.getContext("2d").drawImage(video, 0, 0, canvas.width, canvas.height);
    busy = true;
    canvas.toBlob(async (blob) => {
      if (blob && socket.readyState === WebSocket.OPEN) socket.send(await blob.arrayBuffer());
      busy = false;
    }, "image/jpeg", 0.85);
  }, 1000 / fps);
  return () => {
    clearInterval(timer);
    media.getTracks().forEach((track) => track.stop());
  };
}
```

## Wait until the session is ready

Capacity is brought up on demand, so `POST /v1/sessions` can answer `status: "queued"` with a `queue_position` and no transport offers. Keep polling `GET /v1/sessions/{id}`: a queue ticket nobody polls expires with `error: "scheduler_queue_expired"`. When the status leaves `queued`, `POST /v1/sessions` with the same `session_id` returns the WebSocket offer without creating a second session.

On the socket, `authenticate` with the offer's `launch_token` must be the first message; `client_hello` with `codecs: ["jpeg"]` asks for one JPEG per binary message. Commands are accepted after `worker_ready`. Before it they fail with `409 queued` or `409 no_active_session`.

If several clients can attach to one session, claim the camera before sending it: `POST /v1/sessions/{id}` with `{ action: "publish", connection_id, track: "camera" }`. A second publisher is refused with `409`, because two camera feeds into one model interleave instead of blending.

## Send the instruction and reference together

`set_conditioning` stages the edit instruction and a reference image in one call, for the next model session boundary; it needs both, so send the instruction alone with `set_prompt`. Before `start` the reference anchors the opening edit:

```js
const look = await joy.upload(document.querySelector("#reference").files[0]); // image/*
await joy.command("set_conditioning", { prompt: "Dress them in this jacket", image: { id: look.id } });
```

To edit an uploaded clip instead of the webcam, switch to file mode and set the clip before `start`:

```js
await joy.command("set_mode", { mode: "file" });
const clip = await joy.upload(document.querySelector("#clip").files[0]); // video/*, up to 512 MiB
await joy.command("set_video", { video: { id: clip.id } });              // answered by video_accepted
await joy.command("set_prompt", { prompt: "Make it a rainy night street" });
await joy.command("start");
```

The `upload` helper covers single uploads. A file over 100 MiB comes back with `upload_strategy: "multipart"` and a list of `parts`; PUT each part, then send their `part_number` and `etag` to `POST /v1/uploads/{id}/complete`. In file mode the session ends when the clip runs out, with `session_ended` and `reason: "source_complete"`.

| Command | What it does |
|---|---|
| `set_mode` | `live` for the webcam, `file` for an uploaded clip |
| `set_video` | Sets the uploaded clip in file mode; only before `start` |
| `set_prompt` | Changes the instruction; the model session reopens and the picture holds for about a second |
| `set_image` | Sets or replaces the reference at the next boundary, with the same brief hold |
| `set_conditioning` | Instruction plus reference image, applied together; both required |
| `set_sr_scale` | Delivery upscale, `scale` of `off` or `2x`, over the 840 × 480 interactive tier |
| `start`, `stop`, `reset`, `set_seed` | Run control; `reset` clears source, prompt and progress |

## Track arrived is not first frame visible

| Milestone | Signal | What the user should see |
|---|---|---|
| Ready | `worker_ready` | Their own camera preview |
| Source ready | `source_ready` | Still the preview |
| Generating | `generation_started`, after `start` | The preview with a working indicator |
| Stream arrived | `video_config`, then the first binary message | Same |
| First edited frame visible | The first `drawImage` returns | The edited stream |

Keep the local camera preview on screen until the first edited frame is drawn, so the user never faces an empty box. `transport.fallback_policy.first_frame_timeout_ms` in the session response (10 seconds by default) is how long the platform's own client waits for media after `generation_started`.

The output canvas paints without `play()`, and the camera preview is muted and `playsInline`, so autoplay policies let it start. If you show the edited stream in a `<video>` (via `canvas.captureStream()` or the session's `livekit` offer), handle `play()`: `NotAllowedError` means the browser wants a user gesture, so show a play button; `AbortError` only means a newer `play()` replaced the old one.

## Release the session on unmount

Call `stopCamera()` and `joy.release()` when the component unmounts and on `pagehide`. Release sends `{ action: "detach", connection_id, release: true }` with `keepalive`; for this model the control plane also sends `stop` to the worker before handing the capacity back, and the response reports `session_released: true`. Releasing is what ends billing, because paused time is billed. If a tab dies without it, the worker keeps the session for a reconnect grace (30 seconds by default). For drops, see [recovering a dropped stream](https://windflow.online/guides/recover-a-dropped-realtime-video-stream).

## Troubleshooting by the stage where startup stopped

| Stage | Symptom | Cause and fix |
|---|---|---|
| Token | `401 invalid_api_key` | Send the `wf_live_` key from your server only |
| Create | `403 model_mismatch` or `token_sessions_exhausted` | Token minted for another model, or its `max_sessions` used up |
| Create | `402 insufficient_balance` or `429 rate_limited` | Balance below the shortest useful session, or too many new sessions; honour `Retry-After` |
| Queue | `failed` with `scheduler_queue_expired` | Polling stopped while queued |
| Socket | Closes with `4401` or `4408` | `authenticate` not sent first; or the session already hit its time limit |
| Camera | `getUserMedia` rejects with `NotAllowedError`, or `navigator.mediaDevices` is undefined | Camera permission was denied, or the page is not served over HTTPS |
| Ready | `start` rejected: "Webcam is still buffering" | No camera frame has reached the worker; wait for `source_ready` |
| Ready | `start` rejected: "No source video set" | File mode needs `set_video` first |
| Ready | `set_video` rejected after `start` | The clip can only change before `start`; send `reset` first |
| Live | The picture freezes for about a second after a new instruction | Expected: the model session reopens |
| Live | The session ends at the end of a clip | `source_complete` in file mode; start a new session to edit again |

Rejected commands return `422` with a `command_…` code over HTTP and a `command_error` message on the socket. `conditions_ready` reports `has_video`, `has_prompt` and `has_image` so you can see what is staged.

## FAQ

### Can JoyAI-Video-Edit edit a live webcam stream?

Yes. Select `live` with `set_mode`, send camera frames as JPEG binary messages on the session's WebSocket at the advertised `cameraFps`, and send `start` after `source_ready`.

### Does an edit change the whole picture?

No. Edits are scoped to what the instruction names. Unmentioned parts of the source are preserved, including its motion and identity.

### Why does the picture hold when I change the instruction?

The model has no mid-stream conditioning swap, so a new instruction or reference reopens the model session against the same source. Expect about a second of held picture, not a re-buffered camera.