How to use the Matrix-Game 4.0 API

The Matrix-Game 4.0 API on Windflow runs Matrix-Game 4.0, Skywork's neural game renderer, as a realtime session. A real game runs on the server and owns movement, physics and combat; a scene-specific model draws every frame of it at 864 × 480 and 24 fps. You choose one of three games, send the player's held keys, and the rendered game streams back. There is no prompt and no seed image.

Matrix-Game 4.0 is delivered over the WebSocket transport.

Start from something runnable

You need an approved Windflow account and an API key that begins with wf_live_ (request access). Requests go to https://api.windflow.online.

1. Check the key (server). The response lists this model's commands, messages and limitations.

curl https://api.windflow.online/v1/models/matrix-game-4 \
  -H "Authorization: Bearer $WINDFLOW_API_KEY"

2. Mint a browser token (server). The account key never reaches the browser.

// For example app/api/windflow-token/route.ts. Authenticate your own user first.
export async function POST() {
  const res = await fetch("https://api.windflow.online/v1/tokens", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.WINDFLOW_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ model: "matrix-game-4", max_sessions: 1 }),
  });
  return Response.json(await res.json(), { status: res.status });
}

The token is bound to this model (another model gets 403 model_mismatch) and can create one session. Read expires_at from the response instead of assuming a lifetime.

3. Open the session and draw frames (browser). Save this as windflow-session.js. It is the same file the Helios guide walks through line by line.

// windflow-session.js: plain fetch and WebSocket, no dependencies
const API = "https://api.windflow.online";

export async function call(path, jwt, body) {
  const res = await fetch(API + path, {
    method: body ? "POST" : "GET",
    headers: { Authorization: `Bearer ${jwt}`, "Content-Type": "application/json" },
    body: body && JSON.stringify(body),
  });
  const data = await res.json();
  if (!res.ok) throw Object.assign(new Error(data.error.message), data.error, { status: res.status });
  return data;
}

export function openSession(model, canvas, onMessage = () => {}) {
  const ctx = canvas.getContext("2d");
  let session, ws, closed = false, released = false;
  let painting = Promise.resolve();

  const paint = async (bytes) => {
    const frame = await createImageBitmap(new Blob([bytes], { type: "image/jpeg" }));
    if (canvas.width !== frame.width) Object.assign(canvas, { width: frame.width, height: frame.height });
    ctx.drawImage(frame, 0, 0);
    frame.close();
  };

  const release = () => {
    closed = true;
    ws?.close();
    if (!session || released) return;
    released = true;
    fetch(`${API}/v1/sessions/${session.session_id}`, {
      method: "POST",
      keepalive: true, // still sent while the page unloads
      headers: { Authorization: `Bearer ${session.jwt}`, "Content-Type": "application/json" },
      body: JSON.stringify({ action: "detach", connection_id: session.connection_id, release: true }),
    });
  };

  const ready = (async () => {
    const { jwt } = await (await fetch("/api/windflow-token", { method: "POST" })).json();
    session = await call("/v1/sessions", jwt, { model, transport: "ws-h264" });
    while (session.status === "queued" && !closed) {
      await new Promise((resolve) => setTimeout(resolve, 2000));
      const poll = await call(`/v1/sessions/${session.session_id}`, session.jwt);
      onMessage({ type: "queued", position: poll.queue_position });
      if (poll.status !== "queued") {
        session = await call("/v1/sessions", session.jwt, { model, session_id: session.session_id, transport: "ws-h264" });
      }
    }
    const offer = session.transport.offers.find((o) => o.kind === "ws-h264");
    if (closed || !offer) throw new Error(closed ? "released" : `no stream offer (status ${session.status})`);
    const url = new URL(offer.relay_path, API);
    url.protocol = "wss:";
    ws = new WebSocket(url);
    ws.binaryType = "arraybuffer";
    return new Promise((resolve, reject) => {
      ws.onopen = () => {
        ws.send(JSON.stringify({ type: "authenticate", token: offer.launch_token }));
        ws.send(JSON.stringify({ type: "client_hello", codecs: ["jpeg"] }));
      };
      ws.onmessage = ({ data }) => {
        if (typeof data !== "string") {
          painting = painting.then(() => paint(data)).catch(console.warn);
          return;
        }
        const msg = JSON.parse(data);
        if (msg.type === "worker_ready") resolve(msg);
        if (msg.type === "failed") reject(new Error(msg.message ?? msg.error));
        onMessage(msg);
      };
      ws.onclose = (event) => {
        onMessage({ type: "socket_closed", code: event.code });
        reject(new Error(`socket closed (${event.code})`));
      };
    });
  })();
  ready.catch(release); // a failed start must not leave a session running

  return {
    ready,
    release,
    get session() { return session; },
    get socket() { return ws; },
    command: (type, fields = {}) =>
      call(`/v1/sessions/${session.session_id}/commands`, session.jwt, { command: { type, ...fields } }),
    upload: async (file) => {
      const intent = await call("/v1/uploads", session.jwt, {
        filename: file.name, mime: file.type, size: file.size, session_id: session.session_id,
      });
      const put = await fetch(intent.upload_url, { method: "PUT", headers: intent.headers, body: file });
      if (!put.ok) throw new Error(`upload failed (${put.status})`);
      return call(`/v1/uploads/${intent.file_id}/complete`, session.jwt, {});
    },
  };
}

queued and socket_closed are messages this file generates; every other message comes from the session.

4. Pick the scene, then start. The scene is fixed once the game starts, so choose it first and wait for scene_accepted:

import { openSession } from "./windflow-session.js";

let markScene;
const sceneAccepted = new Promise((resolve) => (markScene = resolve));
const game = openSession("matrix-game-4", document.querySelector("canvas"), (msg) => {
  if (msg.type === "scene_accepted") markScene(msg.scene);
  console.log(msg.type);
});

await game.ready; // worker_ready: commands are accepted from here
await game.command("set_scene", { scene: "ancient-egypt" }); // "castle", "ancient-egypt" or "racing"
await sceneAccepted;
await game.command("start");

A session that never sends set_scene plays castle.

5. Send held keys as one snapshot. The game reads controls on its next frame. Send the complete held set with a rising input_seq on every press and release:

const SHOOTER_KEYS = {
  KeyW: "forward", KeyA: "left", KeyS: "back", KeyD: "right",
  ArrowUp: "look_up", ArrowLeft: "look_left", ArrowDown: "look_down", ArrowRight: "look_right",
  Space: "jump", KeyR: "reload", KeyF: "shield",
};
const RACING_KEYS = {
  KeyW: "forward", KeyA: "left", KeyS: "back", KeyD: "right",
  ShiftLeft: "brake", Space: "handbrake",
};
const KEYS = SHOOTER_KEYS; // RACING_KEYS for the racing scene
const held = new Set();
let inputSeq = 0;

function sendKeys() {
  game.socket.send(JSON.stringify({
    type: "set_input_state", input_seq: ++inputSeq, pressed_keys: [...held], client_ts: Date.now(),
  }));
}
addEventListener("keydown", (event) => {
  const key = KEYS[event.code];
  if (!key || held.has(key)) return;
  event.preventDefault(); // keep Space and the arrows from scrolling the page
  held.add(key);
  sendKeys();
});
addEventListener("keyup", (event) => {
  if (held.delete(KEYS[event.code])) sendKeys();
});
// Mouse buttons in the shooters: left fires, right aims down sights.
const canvas = document.querySelector("canvas");
canvas.addEventListener("contextmenu", (event) => event.preventDefault());
canvas.addEventListener("mousedown", (event) => {
  held.add(event.button === 2 ? "ads" : "fire");
  sendKeys();
});
addEventListener("mouseup", (event) => {
  if (held.delete(event.button === 2 ? "ads" : "fire")) sendKeys();
});
addEventListener("blur", () => {
  held.clear();
  sendKeys();
});

Snapshots replace each other, so there is no need to repeat one while a key stays down. Every snapshot is acknowledged with input_accepted; one whose input_seq is not ahead of the last applied one comes back with stale: true and is not applied.

Wait until the session is ready

Capacity is brought up on demand, so POST /v1/sessions can answer status: "queued" with a queue_position and no transport offers. Keep polling GET /v1/sessions/{id}: a queue ticket nobody polls expires with error: "scheduler_queue_expired". When the status leaves queued, POST /v1/sessions with the same session_id returns the WebSocket offer without creating a second session. Always ask for transport: "ws-h264", the only delivery Matrix-Game 4.0 uses.

On the socket, authenticate with the offer's launch_token must be the first message; client_hello with codecs: ["jpeg"] asks for one JPEG per binary message. Commands are accepted after worker_ready. Before it they fail with 409 queued or 409 no_active_session.

What the game owns and what the model draws

The game runs the rules, so a defeat in a shooter restarts the game without restarting the render: the stream continues while the player respawns. Each scene is its own model, trained for its bundled game, so the session takes no prompt or seed image. The stream carries the rendered picture; draw any score or health display, and add sound, on the page.

The model renders one opening frame, then four-frame chunks with one block of game lookahead. In a browser in the same region as the server, a key press shows on screen in about half a second.

Design for 60-second rounds

Each session is a 60-second round, which keeps the rendered scene consistent from start to finish. Design the experience as short rounds: show a countdown, and offer a new round that opens a new session.

Track arrived is not first frame visible

Start the 60-second countdown at the first painted frame. The session's transport.fallback_policy.first_frame_timeout_ms (10 seconds by default) is how long the platform's own client waits for media after generation_started.

Release the session on unmount

Call game.release() when the component unmounts, on pagehide, and when a round ends early. It sends { action: "detach", connection_id, release: true } with keepalive, which ends the session at the worker and hands the capacity back; the response reports session_released: true.

Troubleshooting by the stage where startup stopped

Rejected commands return 422 with a command_… code over HTTP and a command_error message on the socket.

FAQ

Can I change what Matrix-Game 4.0 renders with a prompt?

No. Each scene is a model trained for one bundled game, and the session takes no prompt or seed image. To render your own game with a prompt-driven look, use EdenLive.

How long is a Matrix-Game 4.0 session?

Each session is a 60-second round, which keeps the rendered scene consistent for the whole round. Open a new session for the next round.

Does the stream include game sound?

The stream carries the rendered picture. Add your own interface and sound effects on the page.