How to use the JoyAI Video Edit API

The JoyAI Video Edit API on Windflow runs JoyAI-Video-Edit as a realtime session that edits a source you provide: a live webcam or an uploaded clip. You give an edit instruction, optionally with a reference image, and the edited stream comes back with the source's motion and identity preserved. Your server mints the token; the browser sends camera frames up and receives edited frames down on one WebSocket.

Start from something runnable

You need an approved Windflow account and an API key that begins with wf_live_ (request access). Requests go to https://api.windflow.online.

1. Check the key (server). The response lists this model's commands, messages and limitations.

curl https://api.windflow.online/v1/models/joyai-video-edit \
  -H "Authorization: Bearer $WINDFLOW_API_KEY"

2. Mint a browser token (server). The account key never reaches the browser.

// For example app/api/windflow-token/route.ts. Authenticate your own user first.
export async function POST() {
  const res = await fetch("https://api.windflow.online/v1/tokens", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.WINDFLOW_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ model: "joyai-video-edit", max_sessions: 1 }),
  });
  return Response.json(await res.json(), { status: res.status });
}

The token is bound to this model (another model gets 403 model_mismatch) and can create one session. Read expires_at from the response instead of assuming a lifetime.

3. Open the session and draw frames (browser). Save this as windflow-session.js; the Helios, LingBot and ABot-World guides use the same file.

// windflow-session.js: plain fetch and WebSocket, no dependencies
const API = "https://api.windflow.online";

export async function call(path, jwt, body) {
  const res = await fetch(API + path, {
    method: body ? "POST" : "GET",
    headers: { Authorization: `Bearer ${jwt}`, "Content-Type": "application/json" },
    body: body && JSON.stringify(body),
  });
  const data = await res.json();
  if (!res.ok) throw Object.assign(new Error(data.error.message), data.error, { status: res.status });
  return data;
}

export function openSession(model, canvas, onMessage = () => {}) {
  const ctx = canvas.getContext("2d");
  let session, ws, closed = false, released = false;
  let painting = Promise.resolve();

  const paint = async (bytes) => {
    const frame = await createImageBitmap(new Blob([bytes], { type: "image/jpeg" }));
    if (canvas.width !== frame.width) Object.assign(canvas, { width: frame.width, height: frame.height });
    ctx.drawImage(frame, 0, 0);
    frame.close();
  };

  const release = () => {
    closed = true;
    ws?.close();
    if (!session || released) return;
    released = true;
    fetch(`${API}/v1/sessions/${session.session_id}`, {
      method: "POST",
      keepalive: true, // still sent while the page unloads
      headers: { Authorization: `Bearer ${session.jwt}`, "Content-Type": "application/json" },
      body: JSON.stringify({ action: "detach", connection_id: session.connection_id, release: true }),
    });
  };

  const ready = (async () => {
    const { jwt } = await (await fetch("/api/windflow-token", { method: "POST" })).json();
    session = await call("/v1/sessions", jwt, { model, transport: "ws-h264" });
    while (session.status === "queued" && !closed) {
      await new Promise((resolve) => setTimeout(resolve, 2000));
      const poll = await call(`/v1/sessions/${session.session_id}`, session.jwt);
      onMessage({ type: "queued", position: poll.queue_position });
      if (poll.status !== "queued") {
        session = await call("/v1/sessions", session.jwt, { model, session_id: session.session_id, transport: "ws-h264" });
      }
    }
    const offer = session.transport.offers.find((o) => o.kind === "ws-h264");
    if (closed || !offer) throw new Error(closed ? "released" : `no stream offer (status ${session.status})`);
    const url = new URL(offer.relay_path, API);
    url.protocol = "wss:";
    ws = new WebSocket(url);
    ws.binaryType = "arraybuffer";
    return new Promise((resolve, reject) => {
      ws.onopen = () => {
        ws.send(JSON.stringify({ type: "authenticate", token: offer.launch_token }));
        ws.send(JSON.stringify({ type: "client_hello", codecs: ["jpeg"] }));
      };
      ws.onmessage = ({ data }) => {
        if (typeof data !== "string") {
          painting = painting.then(() => paint(data)).catch(console.warn);
          return;
        }
        const msg = JSON.parse(data);
        if (msg.type === "worker_ready") resolve(msg);
        if (msg.type === "failed") reject(new Error(msg.message ?? msg.error));
        onMessage(msg);
      };
      ws.onclose = (event) => {
        onMessage({ type: "socket_closed", code: event.code });
        reject(new Error(`socket closed (${event.code})`));
      };
    });
  })();
  ready.catch(release); // a failed start must not leave a session running

  return {
    ready,
    release,
    get session() { return session; },
    get socket() { return ws; },
    command: (type, fields = {}) =>
      call(`/v1/sessions/${session.session_id}/commands`, session.jwt, { command: { type, ...fields } }),
    upload: async (file) => {
      const intent = await call("/v1/uploads", session.jwt, {
        filename: file.name, mime: file.type, size: file.size, session_id: session.session_id,
      });
      const put = await fetch(intent.upload_url, { method: "PUT", headers: intent.headers, body: file });
      if (!put.ok) throw new Error(`upload failed (${put.status})`);
      return call(`/v1/uploads/${intent.file_id}/complete`, session.jwt, {});
    },
  };
}

queued and socket_closed are messages this file generates; every other message comes from the session.

4. Send the webcam and start editing. In live mode, camera frames go up the same socket as JPEG binary messages. The worker's worker_ready says how fast (cameraFps) and how wide (cameraMaxWidth) it wants them; sending more only adds backlog.

import { openSession } from "./windflow-session.js";

let markSourceReady;
const sourceReady = new Promise((resolve) => (markSourceReady = resolve));
const joy = openSession("joyai-video-edit", document.querySelector("canvas"), (msg) => {
  if (msg.type === "source_ready") markSourceReady(); // a camera frame reached the worker
  console.log(msg.type);
});

const hello = await joy.ready;
await joy.command("set_mode", { mode: "live" });
const stopCamera = await sendCamera(joy.socket, hello, document.querySelector("#preview"));
await joy.command("set_prompt", { prompt: "Change the jacket to red leather" });
await sourceReady;
await joy.command("start");

// `video` is a visible <video id="preview"> element: the user's own camera preview.
async function sendCamera(socket, hello, video) {
  const fps = hello.cameraFps || 16;
  const maxWidth = hello.cameraMaxWidth || 1280;
  const media = await navigator.mediaDevices.getUserMedia({ video: true });
  Object.assign(video, { muted: true, playsInline: true, srcObject: media });
  await video.play();
  const canvas = document.createElement("canvas");
  let busy = false;
  const timer = setInterval(() => {
    // Skip a tick rather than queue: a stale camera frame is edited late.
    if (busy || !video.videoWidth || socket.bufferedAmount > 1 << 20) return;
    const scale = Math.min(1, maxWidth / video.videoWidth);
    canvas.width = Math.round(video.videoWidth * scale);
    canvas.height = Math.round(video.videoHeight * scale);
    canvas.getContext("2d").drawImage(video, 0, 0, canvas.width, canvas.height);
    busy = true;
    canvas.toBlob(async (blob) => {
      if (blob && socket.readyState === WebSocket.OPEN) socket.send(await blob.arrayBuffer());
      busy = false;
    }, "image/jpeg", 0.85);
  }, 1000 / fps);
  return () => {
    clearInterval(timer);
    media.getTracks().forEach((track) => track.stop());
  };
}

Wait until the session is ready

Capacity is brought up on demand, so POST /v1/sessions can answer status: "queued" with a queue_position and no transport offers. Keep polling GET /v1/sessions/{id}: a queue ticket nobody polls expires with error: "scheduler_queue_expired". When the status leaves queued, POST /v1/sessions with the same session_id returns the WebSocket offer without creating a second session.

On the socket, authenticate with the offer's launch_token must be the first message; client_hello with codecs: ["jpeg"] asks for one JPEG per binary message. Commands are accepted after worker_ready. Before it they fail with 409 queued or 409 no_active_session.

If several clients can attach to one session, claim the camera before sending it: POST /v1/sessions/{id} with { action: "publish", connection_id, track: "camera" }. A second publisher is refused with 409, because two camera feeds into one model interleave instead of blending.

Send the instruction and reference together

set_conditioning stages the edit instruction and a reference image in one call, for the next model session boundary; it needs both, so send the instruction alone with set_prompt. Before start the reference anchors the opening edit:

const look = await joy.upload(document.querySelector("#reference").files[0]); // image/*
await joy.command("set_conditioning", { prompt: "Dress them in this jacket", image: { id: look.id } });

To edit an uploaded clip instead of the webcam, switch to file mode and set the clip before start:

await joy.command("set_mode", { mode: "file" });
const clip = await joy.upload(document.querySelector("#clip").files[0]); // video/*, up to 512 MiB
await joy.command("set_video", { video: { id: clip.id } });              // answered by video_accepted
await joy.command("set_prompt", { prompt: "Make it a rainy night street" });
await joy.command("start");

The upload helper covers single uploads. A file over 100 MiB comes back with upload_strategy: "multipart" and a list of parts; PUT each part, then send their part_number and etag to POST /v1/uploads/{id}/complete. In file mode the session ends when the clip runs out, with session_ended and reason: "source_complete".

Track arrived is not first frame visible

Keep the local camera preview on screen until the first edited frame is drawn, so the user never faces an empty box. transport.fallback_policy.first_frame_timeout_ms in the session response (10 seconds by default) is how long the platform's own client waits for media after generation_started.

The output canvas paints without play(), and the camera preview is muted and playsInline, so autoplay policies let it start. If you show the edited stream in a <video> (via canvas.captureStream() or the session's livekit offer), handle play(): NotAllowedError means the browser wants a user gesture, so show a play button; AbortError only means a newer play() replaced the old one.

Release the session on unmount

Call stopCamera() and joy.release() when the component unmounts and on pagehide. Release sends { action: "detach", connection_id, release: true } with keepalive; for this model the control plane also sends stop to the worker before handing the capacity back, and the response reports session_released: true. Releasing is what ends billing, because paused time is billed. If a tab dies without it, the worker keeps the session for a reconnect grace (30 seconds by default). For drops, see recovering a dropped stream.

Troubleshooting by the stage where startup stopped

Rejected commands return 422 with a command_… code over HTTP and a command_error message on the socket. conditions_ready reports has_video, has_prompt and has_image so you can see what is staged.

FAQ

Can JoyAI-Video-Edit edit a live webcam stream?

Yes. Select live with set_mode, send camera frames as JPEG binary messages on the session's WebSocket at the advertised cameraFps, and send start after source_ready.

Does an edit change the whole picture?

No. Edits are scoped to what the instruction names. Unmentioned parts of the source are preserved, including its motion and identity.

Why does the picture hold when I change the instruction?

The model has no mid-stream conditioning swap, so a new instruction or reference reopens the model session against the same source. Expect about a second of held picture, not a re-buffered camera.