# How to use the DreamX World API

> Use the DreamX-World-5B API to roll a seed image forward as a world you steer: stage prompt and seed together, send movement and look state, and release.

Source: https://windflow.online/guides/dreamx-world-api · Updated 2026-11-03 · Windflow Labs

The **DreamX World API** on Windflow runs [DreamX-World-5B](https://windflow.online/models/dreamx-world-5b), AMAP's controllable world model, as a realtime session. A seed image and a text prompt open a world; movement and look controls steer the camera through it, and the result streams back as a portrait 704 × 1280 rollout. Controls are sampled into the model's next action sequence rather than applied frame by frame, so the world responds in steps.

## Start from something runnable

You need an approved Windflow account and an API key that begins with `wf_live_` ([request access](https://windflow.online/auth?mode=request)). Requests go to `https://api.windflow.online`.

**1. Check the key (server).** The response lists this model's commands, messages and limitations.

```bash
curl https://api.windflow.online/v1/models/dreamx-world-5b \
  -H "Authorization: Bearer $WINDFLOW_API_KEY"
```

**2. Mint a browser token (server).** The account key never reaches the browser.

```ts
// For example app/api/windflow-token/route.ts. Authenticate your own user first.
export async function POST() {
  const res = await fetch("https://api.windflow.online/v1/tokens", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.WINDFLOW_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ model: "dreamx-world-5b", max_sessions: 1 }),
  });
  return Response.json(await res.json(), { status: res.status });
}
```

The token is bound to this model (another model gets `403 model_mismatch`) and can create one session. Read `expires_at` from the response instead of assuming a lifetime.

**3. Open the session and draw frames (browser).** Save this as `windflow-session.js`. It is the same file the [Helios guide](https://windflow.online/guides/helios-api) walks through line by line.

```js
// windflow-session.js: plain fetch and WebSocket, no dependencies
const API = "https://api.windflow.online";

export async function call(path, jwt, body) {
  const res = await fetch(API + path, {
    method: body ? "POST" : "GET",
    headers: { Authorization: `Bearer ${jwt}`, "Content-Type": "application/json" },
    body: body && JSON.stringify(body),
  });
  const data = await res.json();
  if (!res.ok) throw Object.assign(new Error(data.error.message), data.error, { status: res.status });
  return data;
}

export function openSession(model, canvas, onMessage = () => {}) {
  const ctx = canvas.getContext("2d");
  let session, ws, closed = false, released = false;
  let painting = Promise.resolve();

  const paint = async (bytes) => {
    const frame = await createImageBitmap(new Blob([bytes], { type: "image/jpeg" }));
    if (canvas.width !== frame.width) Object.assign(canvas, { width: frame.width, height: frame.height });
    ctx.drawImage(frame, 0, 0);
    frame.close();
  };

  const release = () => {
    closed = true;
    ws?.close();
    if (!session || released) return;
    released = true;
    fetch(`${API}/v1/sessions/${session.session_id}`, {
      method: "POST",
      keepalive: true, // still sent while the page unloads
      headers: { Authorization: `Bearer ${session.jwt}`, "Content-Type": "application/json" },
      body: JSON.stringify({ action: "detach", connection_id: session.connection_id, release: true }),
    });
  };

  const ready = (async () => {
    const { jwt } = await (await fetch("/api/windflow-token", { method: "POST" })).json();
    session = await call("/v1/sessions", jwt, { model, transport: "ws-h264" });
    while (session.status === "queued" && !closed) {
      await new Promise((resolve) => setTimeout(resolve, 2000));
      const poll = await call(`/v1/sessions/${session.session_id}`, session.jwt);
      onMessage({ type: "queued", position: poll.queue_position });
      if (poll.status !== "queued") {
        session = await call("/v1/sessions", session.jwt, { model, session_id: session.session_id, transport: "ws-h264" });
      }
    }
    const offer = session.transport.offers.find((o) => o.kind === "ws-h264");
    if (closed || !offer) throw new Error(closed ? "released" : `no stream offer (status ${session.status})`);
    const url = new URL(offer.relay_path, API);
    url.protocol = "wss:";
    ws = new WebSocket(url);
    ws.binaryType = "arraybuffer";
    return new Promise((resolve, reject) => {
      ws.onopen = () => {
        ws.send(JSON.stringify({ type: "authenticate", token: offer.launch_token }));
        ws.send(JSON.stringify({ type: "client_hello", codecs: ["jpeg"] }));
      };
      ws.onmessage = ({ data }) => {
        if (typeof data !== "string") {
          painting = painting.then(() => paint(data)).catch(console.warn);
          return;
        }
        const msg = JSON.parse(data);
        if (msg.type === "worker_ready") resolve(msg);
        if (msg.type === "failed") reject(new Error(msg.message ?? msg.error));
        onMessage(msg);
      };
      ws.onclose = (event) => {
        onMessage({ type: "socket_closed", code: event.code });
        reject(new Error(`socket closed (${event.code})`));
      };
    });
  })();
  ready.catch(release); // a failed start must not leave a session running

  return {
    ready,
    release,
    get session() { return session; },
    get socket() { return ws; },
    command: (type, fields = {}) =>
      call(`/v1/sessions/${session.session_id}/commands`, session.jwt, { command: { type, ...fields } }),
    upload: async (file) => {
      const intent = await call("/v1/uploads", session.jwt, {
        filename: file.name, mime: file.type, size: file.size, session_id: session.session_id,
      });
      const put = await fetch(intent.upload_url, { method: "PUT", headers: intent.headers, body: file });
      if (!put.ok) throw new Error(`upload failed (${put.status})`);
      return call(`/v1/uploads/${intent.file_id}/complete`, session.jwt, {});
    },
  };
}
```

`queued` and `socket_closed` are messages this file generates; every other message comes from the session.

**4. Stage the world and start it.** DreamX needs both a prompt and a seed image before `start`. Stage them in one `set_conditioning` call:

```js
import { openSession } from "./windflow-session.js";

const world = openSession("dreamx-world-5b", document.querySelector("canvas"), (msg) => console.log(msg.type));
await world.ready; // worker_ready: commands are accepted from here
const seed = await world.upload(document.querySelector("input[type=file]").files[0]);
await world.command("set_conditioning", {
  prompt: "A stone canal town at dusk, wet cobblestones reflecting lantern light, narrow alleys leading toward a bridge",
  image: { id: seed.id },
});
await world.command("start");
```

**5. Send held keys as one snapshot.** WASD moves and the arrow keys look around. Send the complete held set as a `set_input_state` snapshot with a rising `input_seq`, on the open socket:

```js
const KEYS = {
  KeyW: "forward", KeyA: "left", KeyS: "back", KeyD: "right",
  ArrowUp: "look_up", ArrowLeft: "look_left", ArrowDown: "look_down", ArrowRight: "look_right",
};
const held = new Set();
let inputSeq = 0;

function sendKeys() {
  world.socket.send(JSON.stringify({
    type: "set_input_state", input_seq: ++inputSeq, pressed_keys: [...held], client_ts: Date.now(),
  }));
}
addEventListener("keydown", (event) => {
  const key = KEYS[event.code];
  if (!key || held.has(key) || event.target.closest?.("input, textarea")) return;
  event.preventDefault(); // keep the arrow keys from scrolling the page
  held.add(key);
  sendKeys();
});
addEventListener("keyup", (event) => {
  if (held.delete(KEYS[event.code])) sendKeys();
});
addEventListener("blur", () => {
  held.clear();
  sendKeys();
});
```

## Wait until the session is ready

Capacity is brought up on demand, so `POST /v1/sessions` can answer `status: "queued"` with a `queue_position` and no transport offers. Keep polling `GET /v1/sessions/{id}`: a queue ticket nobody polls expires with `error: "scheduler_queue_expired"`. When the status leaves `queued`, `POST /v1/sessions` with the same `session_id` returns the WebSocket offer without creating a second session.

On the socket, `authenticate` with the offer's `launch_token` must be the first message; `client_hello` with `codecs: ["jpeg"]` asks for one JPEG per binary message. Commands are accepted after `worker_ready`. Before it they fail with `409 queued` or `409 no_active_session`.

## How controls reach the world

DreamX does not read a controller every frame. Your held keys are sampled into the next action sequence, the model generates that stretch of the world, and the frames stream back. Three things follow:

- **Movement is persistent state.** It keeps going through later sequences until you release it. That is why the example sends an empty snapshot on `keyup` and on window `blur`.
- **Steady intent works best.** One movement plus at most one look direction per sequence reads clearly. Steady keys give the clearest motion.
- **Acknowledge at once, expect the picture later.** `input_accepted` arrives for every snapshot; show the press in your interface immediately, and let the world catch up at the next sequence. A snapshot whose `input_seq` is not ahead of the last applied one comes back with `stale: true` and is not applied.

`set_movement`, `set_look_horizontal` and `set_look_vertical` are single-axis alternatives to the snapshot. `set_prompt` updates the world prompt during the run, and `pause`, `resume`, `reset` and `set_seed` control the run.

| Command | What it does |
|---|---|
| `set_conditioning` | Stages the world prompt and seed image together; both required |
| `set_prompt`, `set_image` | Update the prompt or the seed alone |
| `set_input_state` | The complete held-key set: `forward`, `back`, `left`, `right`, `look_left`, `look_right`, `look_up`, `look_down` |
| `set_movement` | `idle`, `forward`, `back`, `left`, `right`, `strafe_left`, `strafe_right` |
| `set_look_horizontal`, `set_look_vertical` | `idle`, `left`, `right`; `idle`, `up`, `down` |
| `start`, `pause`, `resume`, `reset`, `set_seed` | Run control |

The production runtime streams portrait 704 × 1280, and the resolution is negotiated per session. Read `video_config` rather than hard-coding the canvas size, and size the canvas with CSS. Compose seed images for a tall frame: a path that recedes up the picture navigates better than a wide vista that gets cropped. The [world model seed image and prompt guide](https://windflow.online/guides/world-model-seed-images-and-prompts) covers seed choice in detail.

## Track arrived is not first frame visible

| Milestone | Signal | What the player should see |
|---|---|---|
| Ready | `worker_ready` | The seed image, with a loading state |
| Generating | `generation_started`, after `start` | Still the seed image |
| Stream arrived | `video_config`, then the first binary message | Still the seed image |
| First frame visible | The first `drawImage` returns | The world; enable the keys |
| Key acknowledged | `input_accepted` | Immediate local feedback |
| Key visible | Frames from the next action sequence | The camera moving |

Enable the keys at the first painted frame, not at `worker_ready`. The session's `transport.fallback_policy.first_frame_timeout_ms` (10 seconds by default) is how long the platform's own client waits for media after `generation_started`.

## Release the session on unmount

Call `world.release()` when the component unmounts and on `pagehide`. It sends `{ action: "detach", connection_id, release: true }` with `keepalive`, which ends the session at the worker and hands the capacity back; the response reports `session_released: true`. A world nobody is steering is still a ready session, and paused time is billed, so release is what stops the meter. If a tab dies without releasing, the worker keeps the session for a reconnect grace (30 seconds by default). For drops, see [recovering a dropped stream](https://windflow.online/guides/recover-a-dropped-realtime-video-stream).

## Troubleshooting by the stage where startup stopped

| Stage | Symptom | Cause and fix |
|---|---|---|
| Token | `401 invalid_api_key` | Send the `wf_live_` key from your server only |
| Create | `403 model_mismatch` or `token_sessions_exhausted` | Token minted for another model, or its `max_sessions` used up |
| Create | `402 insufficient_balance` or `429 rate_limited` | Balance below the shortest useful session, or too many new sessions; honour `Retry-After` |
| Queue | `failed` with `scheduler_queue_expired` | Polling stopped while queued |
| Socket | Closes with `4401` or `4408` | `authenticate` not sent first; or the session already hit its time limit |
| Ready | `start` rejected: "seed image is required" or "prompt is required" | Stage both with `set_conditioning` first |
| Ready | `set_conditioning` rejected: "image is required" | `set_conditioning` needs the image; use `set_prompt` for a prompt alone |
| Running | The world keeps moving after the player lets go | A key release was missed; send an empty snapshot on `blur` |
| Running | Presses feel late | Controls land at the next action sequence; that step is part of the model |
| Running | `input_accepted` with `stale: true` | `input_seq` went backwards; keep one counter for the whole session |

Rejected commands return `422` with a `command_…` code over HTTP and a `command_error` message on the socket. `conditions_ready` reports `has_prompt` and `has_image` before `start`.

## FAQ

### Is DreamX first-person or third-person?

The controls move and turn the camera through the world, as in a first-person view. For a world where the keys drive an on-screen character, see [ABot-World](https://windflow.online/guides/abot-world-api).

### Can I use WebRTC with DreamX?

Yes. DreamX is offered over both `ws-h264` and `livekit`. Request `transport: "livekit"` and join the room with the offer's `url` and `participant_token`; commands go to `POST /v1/sessions/{id}/commands` on both transports.

### Why is the stream portrait?

The production runtime renders 704 × 1280. Resolution is negotiated per session, so read it from `video_config` instead of assuming it.