generating-videos · diff
v0.1.0 to v0.5.0
75 added, 124 removed. Audit A to A.
---
name: generating-videos
- description: ALWAYS read this skill before generating or animating any video, or calling video_generate — text-to-video, image-to-video, a start→end transition, or a reference / motion / audio-guided clip. Turns a brief into a video clip — sets the model, picks the mode, and structures the video prompt. Use whenever the user asks to generate, create, make, or animate a video, produce b-roll, or bring an image to life.
+ description: ALWAYS read this skill before generating or animating any video, or calling video_generate — text-to-video, image-to-video, a start→end transition, or a reference / motion / audio-guided clip. Generates a video from a brief — one clip, or several clips joined into a single file at any length. Use when the user wants to make or generate a video, animate a photo, bring an image to life, produce b-roll, film a described scene, or move from one held frame to another. This skill should also be used when the video takes its motion or style from an existing video, or has to run to an existing audio track.
license: Apache-2.0
metadata:
- version: "0.1.0"
+ version: "0.5.0"
category: creative
- summary: "Turns text briefs or still images into videos. Generates everything from cinematic b-roll to animated product shots, automatically structuring the complex prompts required by top video models."
+ summary: "Routes your brief to the best of 10+ SOTA video models (like Veo, Seedance), writes detailed prompts as per model specifications and generates multi-clip stories that hold continuity across cuts, stitched into one file."
---
# Video Generation
- Turn a brief into a video clip via the `video_generate` tool.
- The decisions that drive quality are **which mode** you choose and **how you write the motion prompt**.
-
- **Scope.** This skill is the open-ended video engine — turning a brief into a clip, on its own or
- from supplied media. Purpose-built marketing formats — a creator talking to camera about a product
- (UGC), a premium product commercial, animated motion-graphics or brand reels, cartoon or anime
- animation — are their own skills and out of scope here.
+ Turn a brief into a video.
## Workflow
- ### Step 1: Pick the model and load its parameters
-
- **`seedance-2.0` is the default** — the general-purpose workhorse; a weak result is usually the
- prompt's fault, not the model's. Reach for another only on a clear signal:
-
- | Reach for another model when the brief… | Model | Guide |
- | --- | --- | --- |
- | Needs spoken dialogue with lip-sync, the tightest prompt-following, or top cinematic quality | `veo-3.1` | `references/prompt-veo.md` |
- | Is editing an existing clip (restyle, add or replace an element, fix on-screen text), or hinges on legible on-screen text | `gemini-omni` | `references/prompt-gemini-omni.md` |
- | Must hold one person or character consistent across references or several shots | `kling-3.0-pro` | `references/prompt-kling.md` |
- | Has an edgy or sensitive subject Seedance would refuse, or wants the fastest, cheapest draft | `grok-imagine-video` | `references/prompt-grok.md` |
- | Is a high-volume or cost-sensitive batch where top quality isn't essential | `wan-2.7` | `references/prompt-wan.md` |
-
- When several fit, take them in order:
-
- - The user **named** a model → use it.
- - Seedance **refused** (`nsfw` / `ip`) or keeps failing → the fallback above (edgy or blocked →
- `grok-imagine-video`).
- - A clear capability above → that model.
- - Otherwise stay on `seedance-2.0`; when two tie, prefer the newer.
-
- The `-fast` / `-lite` / `-turbo` / `-standard` / `-mini` variants share their family's guide, and
- Seedance filters hard on real people, trademarked characters, and graphic content — flag a clearly
- sensitive brief before you spend.
+ ### Step 1: Read the brief and the media
- Then **call `list_video_models` for the chosen model** and read its accepted modes, aspect ratios,
- durations, resolutions, and media — each validates its own subset, and you need these before picking
- a mode or writing the prompt.
+ **Look at everything the user supplied.** Run `image_analysis` on each image and `video_analysis` on
+ each video. Write down what each one shows — who is in it, where it is, how it is lit, how it is
+ framed — so the prompt matches it instead of describing something different. Send a product URL or
+ photo to `analyzing-products`. Skip an image you already know: you made it this turn, or you have
+ already analyzed it.
- ### Step 2: Pick the mode
+ **Then read the brief**: how long the video runs, what happens in it, where it will be posted, and
+ whether anyone speaks.
- From the modes that model accepts, pick what you feed the tool — each input below shapes the clip a
- different way, and a model can take several of them together (a start frame alongside references, for
- instance).
+ **Where the video has a product, write one description of it** and reuse it unchanged in every
+ storyboard and every clip prompt — rephrasing it between calls reads as a different product.
- | The brief is… | Mode | What you pass |
- | --- | --- | --- |
- | A scene described in words, no image | **text-to-video** | `prompt` only |
- | A still to bring to life ("animate this", "make this move") | **image-to-video** | `prompt` + `start_frame_image` |
- | A move from one held frame to another | **start→end transition** | `prompt` + `start_frame_image` + `end_frame_image` |
- | A subject, character, or style to carry (not tied to a specific frame) | **reference image** | `prompt` + `reference_images` |
- | Motion or style taken from existing footage | **video-to-video** | `prompt` + `reference_videos` |
- | Timed to an existing track (lip-sync, beat) | **audio-guided** | `prompt` + `reference_audios` |
+ - **Material and finish**, surface by surface.
+ - **Size** — its width and height, and how it sits against a hand.
+ - **Two to five visual anchors** — features you can verify in the product image and nothing else:
+ exact colours, the shape of a closure or handle, the gauge of a chain or strap, a surface finish, a
+ distinguishing mark.
+ - **What may be done with each part** — which parts are fixed to it, and the whole of what the moving
+ ones allow. Nothing later does anything to the product that isn't on this list.
- A single supplied image can serve as a **start frame** (animate that exact frame in place) or a
- **reference** (carry its subject or style into a new scene) — pick by what the user wants, from the
- inputs and combinations Step 1 showed the model accepts.
+ **Don't transcribe what the label says.** Spelling out the printing invites the model to redraw it,
+ and redrawn text comes back warped.
- ### Step 3: Get any missing inputs
+ ### Step 2: Pick the model and load its parameters
- Ask only for what would change the result and can't be sensibly defaulted — for example a start
- frame when the user says "animate this" but attaches nothing, the aspect ratio when the destination
- decides the crop, or whether the clip should carry sound. Bundle everything you need into one ask
- rather than a back-and-forth; if the user has signalled they don't want questions, choose sensible
- defaults, state them in a line, and proceed.
+ **`seedance-2.0-fast` is the default** — the general-purpose workhorse; a weak result is usually the
+ prompt's fault, not the model's.
- ### Step 4: Read the supplied media first
+ **The storyboard route always runs on `seedance-2.0-fast` unless the user named a model.** The table below
+ decides a single clip.
- When a frame or reference is supplied, read it before writing anything — call `image_analysis` on an
- image (a start or end frame, or a reference image) and `video_analysis` on a reference video — and
- name what it already fixes so your prompt animates or matches it instead of contradicting it. Skip
- this when you already know the media (you made it this turn, or it came with a description).
+ Reach past the default only on a clear signal:
- ### Step 5: Write the motion prompt
+ | Reach for another model when the brief… | Model |
+ | --- | --- |
+ | Needs spoken dialogue with lip-sync, the tightest prompt-following, or top cinematic quality | `veo-3.1-fast` |
+ | Is editing an existing clip — restyle it, add or replace an element, fix on-screen text — or hinges on legible on-screen text | `gemini-omni` |
+ | Must hold one person or character consistent across references or several shots | `kling-3.0-pro` |
+ | Has an edgy or sensitive subject Seedance would refuse, or wants the fastest, cheapest draft | `grok-imagine-video` |
+ | Is a high-volume or cost-sensitive batch where top quality isn't essential | `wan-2.7` |
- Build the prompt as the chosen model's guide specifies (from **Step 1**). Each model wants a different shape, so read its guide first. With a start frame or reference, the image already fixes the look — the prompt carries the **motion**: what moves, the camera behaviour, the physics, and the audio. A dense, concrete, physical description beats a thin one-liner.
+ When more than one fits, work down this list and stop at the first that applies:
- **Pace it to the length.** Fit the amount of action — and speech — to the clip's duration. Aim for roughly one distinct action or shot every couple of seconds, so a short clip carries only a few beats. Spoken dialogue runs at about two to three words a second, so each line has to fit its clip. When the brief needs more time, action, or dialogue than one clip allows — or the same subject has to hold across several shots — build it as several clips and stitch them (see **Longer or multi-clip videos** below).
+ 1. The user named a model — use it.
+ 2. The video takes the storyboard route — `seedance-2.0-fast`.
+ 3. Seedance refused the brief (`nsfw` / `ip`) or keeps failing — use the fallback from the table.
+ 4. The brief matches a row above — use that model.
+ 5. Otherwise use `seedance-2.0-fast`. If two models tie, use the newer one.
- ### Step 6: Generate
- Call `video_generate` with a `requests` list (one object per clip):
+ Then **call `list_video_models` for the chosen model** and read its accepted modes, aspect ratios, durations, resolutions, and media — each validates its own subset, and you need these before writing the prompt.
- - Per object: `prompt` (required); the mode inputs from Step 2; `duration`, `resolution`,
- `aspect_ratio`, and `generate_audio` as the task needs. Omit `model` to take the default; set it
- only when the user named one.
- - **A batch of different clips** (an A/B set, several shots) → add one request object per clip to the
- same call. For variations of one idea, repeat the same object.
+ ### Step 3: Route
- Video is long-running: the tool waits for each clip, but a clip that runs long comes back as
- `{status: "pending", …}` (a job handle, not a failure). Pass that exact handle to `job_status` to
- retrieve the finished clip — **never re-run a pending clip** with `video_generate`; that starts a new,
- separately-billed job. If it's still pending, call `job_status` again with the same handle.
+ Take one of three paths:
- ### Step 7: Return
+ - **The video is a person on camera presenting a product** — holding it and talking about it, opening
+ its package, putting it on, demonstrating it → Trigger the `generating-ugc-videos` skill workflow,
+ whether or not the brief says UGC.
+ - **One clip covers it** — the video runs no longer than the model's longest clip, and holds one
+ continuous shot → Read and follow the steps in `references/direct-route.md`.
+ - **It needs more than one clip** — the video runs longer than the model's longest clip, or the brief
+ needs a cut between shots → Read and follow the steps in `references/storyboard-route.md`.
- Once every clip has finished (rejoin any `pending` ones via `job_status` first), share the resulting
- video URL(s) and local file path(s) with the user.
+ `list_video_models` gives the model's longest clip. Storyboard sheets are what keep a person, a place
+ and a product looking the same from clip to clip, and what fixes the frame on both sides of a cut, so
+ anything with more than one clip takes the storyboard route.
- ## Longer or multi-clip videos
+ A supplied image is used one of two ways. Ask which, if the brief doesn't say:
- A single clip caps at the model's maximum length (`list_video_models` gives the exact range).
- Anything longer is built as a set of clips generated separately, then joined into one file.
- Work through these steps in order:
+ - As the **start frame**, the video begins on that exact image.
+ - As a **reference**, it shows what a subject or a style looks like and the video is a new scene.
- 1. **Plan the shots.** Break the brief into an ordered shot list — each shot one clip within the
- model's allowed durations, and together covering the whole story. Decide what happens in each shot
- and pick its length so the shots **add up to the target** (e.g. a longer target split into a few
- shots at the durations the model allows).
- 2. **Budget the script.** Spread any dialogue or voiceover across the shots so each line fits its
- clip's seconds (about two to three words a second). Set the ambient sound, music, or SFX for each
- shot.
- 3. **Anchor any recurring subject.** Clips are generated independently, so a back-reference ("the
- same woman", "that sneaker again") drifts into a different-looking thing each time. Whenever a
- person, product, or signature prop appears in more than one shot, before generating any clip:
- - Generate **one** clean still of that subject — a portrait for a person, a plain product shot for
- an object — by handing off to the `generating-images` skill (skip this if the user already
- supplied one).
- - Pass that **same** still as `reference_images` in **every** shot the subject appears in.
- - Describe the subject's look word-for-word the same in each prompt; vary only the shot, action,
- and framing. This anchoring works on **any** model.
- 4. **Keep the world continuous.** Repeat the palette, lighting, and setting in every shot's prompt,
- and open each shot on the state the previous one ended in, so a hard cut reads as continuous.
- 5. **Generate the clips.** Send all the shots in one `video_generate` call — one request object per
- shot, the same aspect ratio and resolution across the set.
- 6. **Stitch into one video.** Wait until every clip has finished — rejoin any that came back
- `pending` via `job_status` before stitching. Hand the finished clips, in order, to the
- `video_stitch` tool — it joins them with a hard cut between each and preserves each clip's audio.
- Don't assemble the clips by hand. Return the stitched video (and the individual clips if the user
- wants them).
+ **A start frame works for one clip only, and that clip can carry nothing else** —
+ `start_frame_image` cannot be combined with any `reference_*` input. Where a person, a product or a
+ style has to match an image that isn't the start frame, pass that image as a reference and drop the
+ start frame.
## Edge cases
- **Safety / NSFW rejection** → name a workable stand-in for whatever tripped the filter (cover the
wardrobe, change the setting) and resubmit. If the subject itself is the block, switch to
- `grok-imagine-video` (see **Step 1**) and read its guide. A second rejection: tell the user which
- element is blocked instead of retrying blind.
+ `grok-imagine-video` (see **Step 2**). A second rejection: tell the user which element is blocked
+ instead of retrying blind.
- **Still generating (`status: "pending"`) or the call times out** → the clip is still rendering on
the server, **not** a failure — do **not** resubmit (that starts a second billed job). If you got a
pending handle, call `job_status` with it to rejoin; if the call timed out with no handle, wait and
tell the user it's still processing rather than firing a fresh generation.
- **Generic failure** (an explicit error result, not a timeout) → read the error. If it names a
parameter or a limit, correct that and resubmit. Otherwise resubmit once; if it fails again, either
- switch to a fallback model (**Step 1**) and write to its guide, or give the user the error text
+ switch to a fallback model (**Step 2**) and write to its guide, or give the user the error text
rather than guessing.
- **Reference or frame rejected on count / mode** → the error states the model's limit or supported
- modes; adjust to it (or check `list_video_models`).
- - **The user wants a standalone audio file, not clip audio** (a voiceover to drop on a timeline, or
- a music track or sound effect they need to place or mix themselves) → native clip audio can't give
- them that; it bakes into the picture. Hand speech to `generating-audio`, and say plainly that
- separate music and effects tracks aren't generated here.
+ modes; adjust to it, or check `list_video_models`.
- **Unsure which model fits a named request** → call `list_video_models` and pick by `strengths`.
+ - **The user wants a standalone audio file, not clip audio** — a voiceover to drop on a timeline, or
+ a music track or sound effect they need to place themselves. Clip audio can't give them that; it is
+ baked into the picture. Send speech to `generating-audio`, and say plainly that separate music and
+ effects tracks aren't generated here.
- **`error: "no_provider_configured"`** → relay the tool's `hint` (the user must set their key).
## Reference
- **Model prompt guides** — read the one for the chosen model (see **Step 1**):
+ Each route is a complete workflow. Read the one Step 3 picked, and follow it to the end.
- - `references/prompt-seedance.md` — the default; native audio markers, one-move-per-shot, storyboards.
- - `references/prompt-veo.md` — cinematography-first order, `SFX:` / `Ambient noise:` markers, timecodes.
- - `references/prompt-gemini-omni.md` — conversational editing of an existing clip; edits, not extends.
- - `references/prompt-kling.md` — director-script style, `@Element` references, per-speaker dialogue labels.
- - `references/prompt-grok.md` — tight front-loaded briefs, positive-only exclusions, one action per clip.
- - `references/prompt-wan.md` — front-loaded prose, multi-shot timestamps, driving-audio sync.
+ - `references/direct-route.md` — one clip: the six modes and what each one passes, the beats, the
+ prompt and the call.
+ - `references/storyboard-route.md` — more than one clip: the video planned as beats, built through
+ storyboards and joined into one file.