# How to use the Matrix-Game 4.0 API

> Stream a playable game re-rendered by Matrix-Game 4.0: pick a scene before start, send held-key snapshots on the socket, and plan for 60-second sessions.

Source: https://windflow.online/guides/matrix-game-4-api · Updated 2026-11-03 · Windflow Labs

The **Matrix-Game 4.0 API** on Windflow runs [Matrix-Game 4.0](https://windflow.online/models/matrix-game-4), Skywork's neural game renderer, as a realtime session. A real game runs on the server and owns movement, physics and combat; a scene-specific model draws every frame of it at 864 × 480 and 24 fps. You choose one of three games, send the player's held keys, and the rendered game streams back. There is no prompt and no seed image.

Matrix-Game 4.0 is delivered over the WebSocket transport.

## Start from something runnable

You need an approved Windflow account and an API key that begins with `wf_live_` ([request access](https://windflow.online/auth?mode=request)). Requests go to `https://api.windflow.online`.

**1. Check the key (server).** The response lists this model's commands, messages and limitations.

```bash
curl https://api.windflow.online/v1/models/matrix-game-4 \
  -H "Authorization: Bearer $WINDFLOW_API_KEY"
```

**2. Mint a browser token (server).** The account key never reaches the browser.

```ts
// For example app/api/windflow-token/route.ts. Authenticate your own user first.
export async function POST() {
  const res = await fetch("https://api.windflow.online/v1/tokens", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.WINDFLOW_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ model: "matrix-game-4", max_sessions: 1 }),
  });
  return Response.json(await res.json(), { status: res.status });
}
```

The token is bound to this model (another model gets `403 model_mismatch`) and can create one session. Read `expires_at` from the response instead of assuming a lifetime.

**3. Open the session and draw frames (browser).** Save this as `windflow-session.js`. It is the same file the [Helios guide](https://windflow.online/guides/helios-api) walks through line by line.

```js
// windflow-session.js: plain fetch and WebSocket, no dependencies
const API = "https://api.windflow.online";

export async function call(path, jwt, body) {
  const res = await fetch(API + path, {
    method: body ? "POST" : "GET",
    headers: { Authorization: `Bearer ${jwt}`, "Content-Type": "application/json" },
    body: body && JSON.stringify(body),
  });
  const data = await res.json();
  if (!res.ok) throw Object.assign(new Error(data.error.message), data.error, { status: res.status });
  return data;
}

export function openSession(model, canvas, onMessage = () => {}) {
  const ctx = canvas.getContext("2d");
  let session, ws, closed = false, released = false;
  let painting = Promise.resolve();

  const paint = async (bytes) => {
    const frame = await createImageBitmap(new Blob([bytes], { type: "image/jpeg" }));
    if (canvas.width !== frame.width) Object.assign(canvas, { width: frame.width, height: frame.height });
    ctx.drawImage(frame, 0, 0);
    frame.close();
  };

  const release = () => {
    closed = true;
    ws?.close();
    if (!session || released) return;
    released = true;
    fetch(`${API}/v1/sessions/${session.session_id}`, {
      method: "POST",
      keepalive: true, // still sent while the page unloads
      headers: { Authorization: `Bearer ${session.jwt}`, "Content-Type": "application/json" },
      body: JSON.stringify({ action: "detach", connection_id: session.connection_id, release: true }),
    });
  };

  const ready = (async () => {
    const { jwt } = await (await fetch("/api/windflow-token", { method: "POST" })).json();
    session = await call("/v1/sessions", jwt, { model, transport: "ws-h264" });
    while (session.status === "queued" && !closed) {
      await new Promise((resolve) => setTimeout(resolve, 2000));
      const poll = await call(`/v1/sessions/${session.session_id}`, session.jwt);
      onMessage({ type: "queued", position: poll.queue_position });
      if (poll.status !== "queued") {
        session = await call("/v1/sessions", session.jwt, { model, session_id: session.session_id, transport: "ws-h264" });
      }
    }
    const offer = session.transport.offers.find((o) => o.kind === "ws-h264");
    if (closed || !offer) throw new Error(closed ? "released" : `no stream offer (status ${session.status})`);
    const url = new URL(offer.relay_path, API);
    url.protocol = "wss:";
    ws = new WebSocket(url);
    ws.binaryType = "arraybuffer";
    return new Promise((resolve, reject) => {
      ws.onopen = () => {
        ws.send(JSON.stringify({ type: "authenticate", token: offer.launch_token }));
        ws.send(JSON.stringify({ type: "client_hello", codecs: ["jpeg"] }));
      };
      ws.onmessage = ({ data }) => {
        if (typeof data !== "string") {
          painting = painting.then(() => paint(data)).catch(console.warn);
          return;
        }
        const msg = JSON.parse(data);
        if (msg.type === "worker_ready") resolve(msg);
        if (msg.type === "failed") reject(new Error(msg.message ?? msg.error));
        onMessage(msg);
      };
      ws.onclose = (event) => {
        onMessage({ type: "socket_closed", code: event.code });
        reject(new Error(`socket closed (${event.code})`));
      };
    });
  })();
  ready.catch(release); // a failed start must not leave a session running

  return {
    ready,
    release,
    get session() { return session; },
    get socket() { return ws; },
    command: (type, fields = {}) =>
      call(`/v1/sessions/${session.session_id}/commands`, session.jwt, { command: { type, ...fields } }),
    upload: async (file) => {
      const intent = await call("/v1/uploads", session.jwt, {
        filename: file.name, mime: file.type, size: file.size, session_id: session.session_id,
      });
      const put = await fetch(intent.upload_url, { method: "PUT", headers: intent.headers, body: file });
      if (!put.ok) throw new Error(`upload failed (${put.status})`);
      return call(`/v1/uploads/${intent.file_id}/complete`, session.jwt, {});
    },
  };
}
```

`queued` and `socket_closed` are messages this file generates; every other message comes from the session.

**4. Pick the scene, then start.** The scene is fixed once the game starts, so choose it first and wait for `scene_accepted`:

```js
import { openSession } from "./windflow-session.js";

let markScene;
const sceneAccepted = new Promise((resolve) => (markScene = resolve));
const game = openSession("matrix-game-4", document.querySelector("canvas"), (msg) => {
  if (msg.type === "scene_accepted") markScene(msg.scene);
  console.log(msg.type);
});

await game.ready; // worker_ready: commands are accepted from here
await game.command("set_scene", { scene: "ancient-egypt" }); // "castle", "ancient-egypt" or "racing"
await sceneAccepted;
await game.command("start");
```

A session that never sends `set_scene` plays `castle`.

**5. Send held keys as one snapshot.** The game reads controls on its next frame. Send the complete held set with a rising `input_seq` on every press and release:

```js
const SHOOTER_KEYS = {
  KeyW: "forward", KeyA: "left", KeyS: "back", KeyD: "right",
  ArrowUp: "look_up", ArrowLeft: "look_left", ArrowDown: "look_down", ArrowRight: "look_right",
  Space: "jump", KeyR: "reload", KeyF: "shield",
};
const RACING_KEYS = {
  KeyW: "forward", KeyA: "left", KeyS: "back", KeyD: "right",
  ShiftLeft: "brake", Space: "handbrake",
};
const KEYS = SHOOTER_KEYS; // RACING_KEYS for the racing scene
const held = new Set();
let inputSeq = 0;

function sendKeys() {
  game.socket.send(JSON.stringify({
    type: "set_input_state", input_seq: ++inputSeq, pressed_keys: [...held], client_ts: Date.now(),
  }));
}
addEventListener("keydown", (event) => {
  const key = KEYS[event.code];
  if (!key || held.has(key)) return;
  event.preventDefault(); // keep Space and the arrows from scrolling the page
  held.add(key);
  sendKeys();
});
addEventListener("keyup", (event) => {
  if (held.delete(KEYS[event.code])) sendKeys();
});
// Mouse buttons in the shooters: left fires, right aims down sights.
const canvas = document.querySelector("canvas");
canvas.addEventListener("contextmenu", (event) => event.preventDefault());
canvas.addEventListener("mousedown", (event) => {
  held.add(event.button === 2 ? "ads" : "fire");
  sendKeys();
});
addEventListener("mouseup", (event) => {
  if (held.delete(event.button === 2 ? "ads" : "fire")) sendKeys();
});
addEventListener("blur", () => {
  held.clear();
  sendKeys();
});
```

Snapshots replace each other, so there is no need to repeat one while a key stays down. Every snapshot is acknowledged with `input_accepted`; one whose `input_seq` is not ahead of the last applied one comes back with `stale: true` and is not applied.

## Wait until the session is ready

Capacity is brought up on demand, so `POST /v1/sessions` can answer `status: "queued"` with a `queue_position` and no transport offers. Keep polling `GET /v1/sessions/{id}`: a queue ticket nobody polls expires with `error: "scheduler_queue_expired"`. When the status leaves `queued`, `POST /v1/sessions` with the same `session_id` returns the WebSocket offer without creating a second session. Always ask for `transport: "ws-h264"`, the only delivery Matrix-Game 4.0 uses.

On the socket, `authenticate` with the offer's `launch_token` must be the first message; `client_hello` with `codecs: ["jpeg"]` asks for one JPEG per binary message. Commands are accepted after `worker_ready`. Before it they fail with `409 queued` or `409 no_active_session`.

## What the game owns and what the model draws

| Scene | Game | Keys it reads |
|---|---|---|
| `castle` | Medieval castle shooter | `forward`, `back`, `left`, `right`, `look_left`, `look_right`, `look_up`, `look_down`, `fire`, `ads`, `reload`, `jump`, `shield` |
| `ancient-egypt` | Ancient Egypt shooter | The same as `castle` |
| `racing` | City racing | `forward`, `back`, `left`, `right`, `brake`, `handbrake` |

The game runs the rules, so a defeat in a shooter restarts the game without restarting the render: the stream continues while the player respawns. Each scene is its own model, trained for its bundled game, so the session takes no prompt or seed image. The stream carries the rendered picture; draw any score or health display, and add sound, on the page.

The model renders one opening frame, then four-frame chunks with one block of game lookahead. In a browser in the same region as the server, a key press shows on screen in about half a second.

## Design for 60-second rounds

Each session is a 60-second round, which keeps the rendered scene consistent from start to finish. Design the experience as short rounds: show a countdown, and offer a new round that opens a new session.

## Track arrived is not first frame visible

| Milestone | Signal | What the player should see |
|---|---|---|
| Ready | `worker_ready` | Scene picker |
| Scene chosen | `scene_accepted` | Loading state |
| Generating | `generation_started`, after `start` | Loading state |
| Stream arrived | `video_config`, then the first binary message | Loading state |
| First frame visible | The first `drawImage` returns | The game; enable the keys and start the countdown |

Start the 60-second countdown at the first painted frame. The session's `transport.fallback_policy.first_frame_timeout_ms` (10 seconds by default) is how long the platform's own client waits for media after `generation_started`.

## Release the session on unmount

Call `game.release()` when the component unmounts, on `pagehide`, and when a round ends early. It sends `{ action: "detach", connection_id, release: true }` with `keepalive`, which ends the session at the worker and hands the capacity back; the response reports `session_released: true`.

## Troubleshooting by the stage where startup stopped

| Stage | Symptom | Cause and fix |
|---|---|---|
| Token | `401 invalid_api_key` | Send the `wf_live_` key from your server only |
| Create | `403 model_mismatch` or `token_sessions_exhausted` | Token minted for another model, or its `max_sessions` used up |
| Create | `402 insufficient_balance` or `429 rate_limited` | Balance below the shortest useful session, or too many new sessions; honour `Retry-After` |
| Create | You asked for `transport: "livekit"` | Matrix-Game 4.0 is delivered over `ws-h264` only |
| Queue | `failed` with `scheduler_queue_expired` | Polling stopped while queued |
| Socket | Closes with `4401` or `4408` | `authenticate` not sent first; or the session already hit its time limit |
| Ready | The wrong game starts | `start` was sent before `scene_accepted`; wait for it |
| Running | `set_scene` rejected | The scene cannot change after `start`; open a new session |
| Running | `set_prompt` or `set_image` rejected | This model takes no prompt or seed image |
| Running | The player keeps moving after letting go | A release was missed; send an empty snapshot on `blur` |
| Running | The stream ends after a minute | Each session is a 60-second round; open a new session for the next one |

Rejected commands return `422` with a `command_…` code over HTTP and a `command_error` message on the socket.

## FAQ

### Can I change what Matrix-Game 4.0 renders with a prompt?

No. Each scene is a model trained for one bundled game, and the session takes no prompt or seed image. To render your own game with a prompt-driven look, use [EdenLive](https://windflow.online/guides/edenlive-api).

### How long is a Matrix-Game 4.0 session?

Each session is a 60-second round, which keeps the rendered scene consistent for the whole round. Open a new session for the next round.

### Does the stream include game sound?

The stream carries the rendered picture. Add your own interface and sound effects on the page.