How to write edit instructions for realtime video editing
Write edit instructions that change only what you mean in a live webcam or clip: instructions not captions, what changes then what stays, and reference images.
An editing model works on a video you already have: a live webcam or an uploaded clip. It keeps the motion, the people and everything you do not mention, and it changes what your instruction names. That makes the instruction the whole product. A vague one changes too much or nothing; a precise one changes exactly the jacket, the sky or the style you asked for.
This guide applies to the two editing models on Windflow, JoyAI-Video-Edit and SANA-Streaming. For their APIs, see the JoyAI-Video-Edit API guide and the SANA-Streaming API guide.
Write an instruction, not a caption
A text-to-video prompt describes a scene. An edit instruction says what to do to a scene that already exists. JoyAI-Video-Edit reads your text together with the first frame of the source, so a description of the scene adds nothing; a verb and an object do.
| Weak (a caption) | Strong (an instruction) |
|---|---|
| A cinematic portrait. | Remove the glasses and change the jacket to red leather; keep the face, hair and background unchanged. |
| A beautiful sunset. | Turn the sky into a hand-painted sunset with soft watercolor edges; keep the people and camera motion unchanged. |
| Winter. | Cover the street and the parked cars in fresh snow; keep the people walking as they are. |
The JoyAI-Video-Edit project's own examples follow the same shape: transform a named set of things into a style, turn the whole video into a look, recolour or add specific items, dress a person in named clothes, remove named objects.
Say what changes, then what stays
Both models preserve what the instruction does not name, so the instruction's first job is to name the smallest region that should change.
- Name the target precisely. "The jacket", "the sky", "the two cats on the left", rather than "the image".
- Say the change. Colour, material, style, addition or removal.
- Add one preservation clause for anything that matters and is at risk: "keep the face unchanged", "keep the camera motion".
Use a global style word only when you want a global change. "Make it cinematic" applied to a webcam restyles everything, because nothing in it names a region.
| Edit type | Pattern |
|---|---|
| Style of the whole video | Turn the video into a watercolor wash style. |
| One object's appearance | Change the mug on the desk to brushed copper; keep everything else unchanged. |
| Wardrobe | Dress the person in a brown down jacket and a blue baseball cap. |
| Add items | Give each dog a colorful knitted hat. |
| Remove items | Remove the poster on the wall behind the person. |
| Setting | Make it a rainy night street; keep the cars and people moving as they are. |
Use a reference image for a look words cannot carry
JoyAI-Video-Edit takes an optional reference image. The source supplies motion and structure; the reference supplies appearance, such as a specific jacket, material, palette or lighting. Pick a clean reference whose look agrees with the instruction, and refer to it plainly: Dress them in this jacket.
Stage the instruction and the reference together with set_conditioning before start. Both are required in that command; to send the instruction alone, use set_prompt.
Change the instruction deliberately
The two models handle a new instruction differently, and it changes how you design the interaction.
- SANA-Streaming applies a new prompt at the next 24-frame chunk boundary, without stopping the stream. You can iterate on an edit while it plays.
- JoyAI-Video-Edit has no mid-stream conditioning swap. A new instruction or reference takes a moment to apply: the picture holds briefly, then continues from the same live source. Plan instructions as deliberate steps rather than sending several in a row.
Give the model a good source
The edit can only be as stable as what it edits.
- Prefer even exposure and subjects that are clearly visible.
- Keep the capture size fixed for the whole session; on a WebRTC camera track, set
contentHintto"detail". - In file mode, SANA-Streaming needs a clip of at least 41 frames.
- For long live sessions on SANA-Streaming,
set_anchor_intervalre-grounds the edit on the live source every few chunks, which keeps a long stream close to its source. Each refresh emitsanchored, which you can use for a short cross-fade.
Tune the edit
| Symptom | Likely cause | What to change |
|---|---|---|
| Too much of the picture changes | The instruction used a global word or no target | Name the smallest region; add a preservation clause |
| Nothing changes | The change is not visible enough, or not landed yet | Make the change concrete; wait for the next boundary before judging |
| The person's face changes | Nothing protected it | Add "keep the face unchanged" |
| A long live edit moves away from the source | Long session without re-grounding | On SANA-Streaming, set an anchor interval |
| A short hold after a new instruction | JoyAI-Video-Edit is applying the new instruction | Plan instructions as deliberate steps |
FAQ
Do I need to describe the whole scene?
No. Describe only the change. The model already sees the scene, and anything you do not mention is meant to stay as it is.
Can I change the edit while the video plays?
Yes. On SANA-Streaming the new prompt lands at the next chunk boundary. On JoyAI-Video-Edit the picture holds briefly while the model session reopens with the new instruction.
Which editing model should I use?
SANA-Streaming for quick, iterative edits on a live stream, with re-anchoring for long sessions. JoyAI-Video-Edit when you need a reference image or want the larger model, and can live with a short hold when the instruction changes.