generating-videos · diff
git:20260317.b3d2ddd to git:20260623.e3abc45
75 added, 459 removed. Audit A to A.
What this update changes in the checks
No check changed: every fixed check gives the same answer before and after this update.
A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.
---
name: generating-videos
- description: Generates videos using Grok (preferred) and Google Veo 3.1 models. Supports text-to-video, image-to-video, first-last-frame, video extension, and reference images. AI视频生成、文生视频、图生视频、首尾帧生成、视频拓展。
+ description: Generates videos using Grok. Supports text-to-video and image-to-video. AI视频生成、文生视频、图生视频。
---
# Video Generation
- Generates videos using **Grok** (preferred) and **Google Veo 3.1** models.
+ Generates videos using **Grok**.
## Model Selection Strategy
- **CRITICAL**: Always prefer Grok unless the task requires Veo-specific features.
+ Use `grok-imagine-video` for video generation.
| Scenario | Use Model | Reason |
|----------|-----------|--------|
- | Video ≤ 15s (text-to-video / image-to-video) | **Grok** (preferred) | Grok supports 1-15s directly, faster and cost-effective |
- | Video 16-36s (continuous) | **Veo** | Requires Video Extension, Grok max is 15s |
- | First-last-frame storyboard | **Veo** | Grok does not support first-last-frame |
- | Reference images (style consistency) | **Veo** (generate-preview) | Grok does not support reference images |
- | User explicitly requests Veo | **Veo** | User preference |
- | User explicitly requests Grok | **Grok** | User preference |
+ | Video 1-15s | `grok-imagine-video` | Grok supports direct 1-15s generation |
+ | Text-to-video | `grok-imagine-video` | Prompt-only generation |
+ | Image-to-video | `grok-imagine-video` | Uses one reference image |
+ | Video >15s | Generate multiple clips, then load `editing-videos` to concatenate | Grok max duration is 15s |
## Language Rule
- **CRITICAL**: Video prompts MUST be generated in:
-
- 1. **User's language** - If user writes in Chinese, generate Chinese prompts; if in English, generate English prompts
- 2. **User-specified language** - If user explicitly requests a specific language, use that language
-
- ---
+ Video prompts MUST follow the user's language. If the user writes in Chinese, generate Chinese prompts; if in English, generate English prompts. If the user explicitly requests a specific language, use that language.
- ## Grok Video Models (Preferred)
+ ## Grok Video Model
| Model | Speed | Quality | Max Duration | Use Case |
|-------|-------|---------|--------------|----------|
- | grok-imagine-video | Fast | Good | 15s | Default - Most use cases |
-
- ### Grok Parameters
-
- | Parameter | Values | Default | Description |
- |-------------|-------------------------------------------------|---------|-------------|
- | duration | 1-15 | - | Video duration in seconds |
- | resolution | 480p, 720p | 720p | Video resolution |
- | aspectRatio | 1:1, 16:9, 9:16, 4:3, 3:4, 3:2, 2:3 | 9:16 | Video aspect ratio |
- | imageUrl | string | - | Reference image URL for image-to-video |
-
- ### Grok Generation Modes
-
- | Mode | Parameters | Description |
- |------|-----------|-------------|
- | Text-to-video | prompt only | Generate video from text description |
- | Image-to-video | prompt + imageUrl | Generate video using a reference image |
-
- ### Grok Workflow
-
- 1. Call `generateVideoWithGrok` with prompt and optional parameters
- 2. Poll `getGrokVideoStatus` every 30 seconds
- 3. Maximum wait: 5 minutes
-
- ---
-
- ## Veo 3.1 Models (Advanced Features)
-
- Use Veo when you need: video extension, first-last-frame, or reference images.
-
- | Model | Speed | Quality | Video Extension | Reference Images | Use Case |
- | ----------------------------- | ----- | ------- | --------------- | ---------------- | -------------------------------- |
- | veo-3.1-fast-generate-preview | Fast | Good | Yes | No | Default Veo - Most use cases |
- | veo-3.1-generate-preview | Slow | Better | Yes | Yes (max 3) | Best quality + all features |
- | veo-3.1-fast-generate-001 | Fast | Good | No | No | Simple generation (no extension) |
- | veo-3.1-generate-001 | Slow | Better | No | No | Higher quality (no extension) |
-
- **Veo Model Selection**:
-
- - **Default Veo**: Use `veo-3.1-fast-generate-preview` (supports video extension)
- - **Use non-preview models when**: User explicitly requests and doesn't need video extension
- - **Use non-fast models when**: User explicitly requests higher quality
-
- ### Veo Parameters
-
- | Parameter | Values | Default | Description |
- | -------------- | ----------------- | ------- | ---------------------------------------- |
- | duration | 4, 6, 8 | 8 | Video duration in seconds |
- | resolution | 720p, 1080p, 4000 | 720p | Video resolution (1080p/4K takes longer) |
- | aspectRatio | 16:9, 9:16 | 9:16 | Video aspect ratio (vertical by default for social media) |
- | negativePrompt | string | - | What to exclude from the video |
- | seed | number | - | Random seed for reproducibility |
-
- **Notes**:
-
- - Video extension only supports 720p
- - Reference images require 8-second duration
-
- ---
-
- ## Prompt Guide (applies to both Grok and Veo)
-
- ### Prompt Structure
-
- `[Subject & Background] + [Action] + [Style] + [Camera] + [Atmosphere] + [Audio]`
-
- ### 1. Subject & Background (主题与背景)
-
- Specify your main focus (object, person, animal) and environmental context.
-
- **Examples**:
-
- - "A young woman with long black hair wearing a red dress" / "一位长发黑发、身穿红裙的年轻女子"
- - "White concrete apartment building with organic shapes and lush greenery" / "白色混凝土公寓楼,有机形状,绿植茂盛"
-
- ### 2. Action (动作)
-
- Describe what the subject is doing—walking, running, transforming, etc.
-
- **Examples**:
-
- - "walks slowly towards the camera" / "缓缓走向镜头"
- - "transforms from liquid to solid" / "从液态变为固态"
-
- ### 3. Style (风格)
-
- Add aesthetic direction using keywords.
-
- **Common Styles**:
-
- - Film noir / 黑色电影
- - Surrealism / 超现实主义
- - Cyberpunk / 赛博朋克
- - 3D cartoon animation / 3D卡通动画
- - Cinematic documentary / 电影纪录片
- - Japanese anime / 日系动漫
-
- ### 4. Camera Work (镜头)
-
- Control perspective and movement.
-
- **Camera Angles**:
-
- - Aerial shot / 航拍镜头
- - POV (Point of View) / 第一人称视角
- - Close-up / 特写
- - Wide angle / 广角
- - Low angle / 低角度
- - Bird's eye view / 鸟瞰
-
- **Camera Movements**:
-
- - Tracking shot / 跟踪镜头
- - Pan left/right / 左右平移
- - Dolly in/out / 推拉镜头
- - Static shot / 固定镜头
-
- ### 5. Atmosphere (氛围)
-
- Specify lighting and color mood.
-
- **Examples**:
-
- - "warm golden hour lighting" / "温暖的黄金时段光线"
- - "cool blue tones" / "冷蓝色调"
- - "neon-lit cyberpunk atmosphere" / "霓虹灯赛博朋克氛围"
- - "foggy moonlight" / "雾蒙蒙的月光"
-
- ---
-
- ## Audio Generation Guide
-
- Veo 3.1 natively generates synchronized audio. Include audio descriptions in your prompt.
-
- ### 1. Dialogue (对话)
-
- Use **quotation marks** for specific speech.
-
- **Example**:
-
- ```
- A man in a suit stands at a podium, speaking confidently: "Welcome to the future of technology."
- 一位穿西装的男子站在讲台上,自信地说:"欢迎来到科技的未来。"
- ```
-
- ### 2. Sound Effects (音效)
-
- Describe sounds explicitly.
-
- **Examples**:
-
- - "screeching tires" / "轮胎尖叫声"
- - "crackling fire" / "噼啪作响的火焰"
- - "thunder rumbling in the distance" / "远处雷声隆隆"
- - "footsteps echoing in the hallway" / "走廊里回响的脚步声"
-
- ### 3. Ambient Noise (环境音)
-
- Describe environmental soundscapes.
-
- **Examples**:
-
- - "busy city street with traffic and chatter" / "繁忙的城市街道,车流和人声"
- - "peaceful forest with birds chirping" / "宁静的森林,鸟鸣声"
- - "ocean waves crashing on the shore" / "海浪拍打海岸"
-
- - "busy city street with traffic and chatter" / "繁忙的城市街道,车流和人声"
- - "peaceful forest with birds chirping" / "宁静的森林,鸟鸣声"
- - "ocean waves crashing on the shore" / "海浪拍打海岸"
-
- ## Generation Modes
-
- Modes are auto-detected based on parameters:
-
- | Mode | Parameters | Models Supported |
- | ---------------- | -------------------------- | ----------------------------- |
- | Text-to-video | prompt only | All |
- | Image-to-video | prompt + image | All |
- | First-last-frame | prompt + image + lastFrame | All |
- | Video extension | prompt + video | Preview models only |
- | Reference images | prompt + referenceImages | veo-3.1-generate-preview only |
-
- ---
-
- ## Video Extension
-
- Extend existing Veo-generated videos.
-
- **IMPORTANT**: Each extension returns a **COMPLETE video** (initial + all extensions combined). NO concatenation needed - the API automatically merges the extended content into a single video file.
-
- ### Limitations
-
- - **Models**: Only preview models (`veo-3.1-generate-preview`, `veo-3.1-fast-generate-preview`)
- - **Extension length**: ~7 seconds per extension
- - **Max extensions**: 4 times
- - **Max total duration**: 36 seconds (8s initial + 4×7s extensions)
- - **Resolution**: 720p only
- - **Aspect ratio**: 16:9 or 9:16 only
-
- ### Workflow
-
- 1. Generate initial video with preview model
- 2. Call `generateVideoWithVeo` with `video` parameter (prefer GCS URI `gs://bucket/path` for efficiency)
- 3. Repeat up to 4 times to reach 36 seconds max
+ | `grok-imagine-video` | Fast | Good | 15s | Default video generation |
- **Note**: When extending videos, prefer using GCS URI over HTTP URL for better performance (no download/base64 conversion needed).
+ ## Parameters
- - **Models**: Only preview models (`veo-3.1-generate-preview`, `veo-3.1-fast-generate-preview`)
- - **Extension length**: ~7 seconds per extension
- - **Max extensions**: five times
- - **Max total duration**: 37 seconds
- - **Resolution**: 720p only
- - **Aspect ratio**: 16:9 or 9:16 only
+ | Parameter | Values | Default | Description |
+ |-----------|--------|---------|-------------|
+ | `duration` | 1-15 | model default | Video duration in seconds |
+ | `resolution` | `480p`, `720p` | `720p` | Video resolution |
+ | `aspectRatio` | `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, `2:3` | `9:16` | Video aspect ratio |
+ | `imageUrl` | URL | none | Reference image URL for image-to-video |
## Workflow
- ### Grok Video Generation (Preferred)
-
- 1. Call `generateVideoWithGrok` with prompt and optional parameters
- 2. Poll `getGrokVideoStatus` every 30 seconds
- 3. Maximum wait: 5 minutes
-
- ### Veo Video Generation
-
- 1. Call `generateVideoWithVeo` with prompt and optional parameters
- 2. Poll `getVeoVideoStatus` every 30 seconds
- 3. Maximum wait: 10 minutes
-
- ### First-Last-Frame Generation (Veo only)
-
- 1. Generate keyframes with `generateImage`
- 2. Call `generateVideoWithVeo` with `image` (first frame) and `lastFrame`
- 3. Poll `getVeoVideoStatus` until completed
-
- ---
-
- ## Duration-Based Model Selection Strategy
-
- **CRITICAL**: For videos ≤ 15 seconds, ALWAYS prefer **Grok direct generation**. Only use Veo Extension for videos longer than 15 seconds.
-
- ### When to Use Each Approach
-
- | Target Duration | Recommended Approach |
- | --------------- | ---------------------------------------- |
- | ≤ 15 seconds | **Grok direct generation** (preferred, faster and cost-effective) |
- | 16-36 seconds | Veo Video Extension (dynamic initial + N extensions) |
- | > 36 seconds | First-Last-Frame Storyboard |
-
- ### Why Grok for ≤ 15s?
-
- - **Faster**: Grok generates directly without extension polling
- - **Cost-effective**: Single generation call vs initial + extensions
- - **Simpler workflow**: No multi-step extension process needed
- - **Full duration support**: Grok supports 1-15 seconds natively
-
- ### Veo Extension (for 16-36s only)
-
- **Dynamic Initial Duration Calculation**:
-
- **Parameters**:
- - Initial duration options: 4s, 6s, 8s
- - Each extension adds: ~7 seconds
- - Maximum extensions: 4 times
- - Maximum total duration: 36 seconds (8s + 4×7s)
-
- **Calculation Formula**:
-
- 1. Extension count `n` = ⌈(target - 8) / 7⌉
- 2. Initial duration = target - (n × 7)
- 3. If initial < 4, then n += 1, recalculate initial
-
- **Reference Table**:
-
- | Target | Initial | Extensions | Actual |
- |--------|---------|------------|--------|
- | 18s | 4s | 2 | 18s |
- | 22s | 8s | 2 | 22s |
- | 27s | 6s | 3 | 27s |
- | 29s | 8s | 3 | 29s |
- | 36s | 8s | 4 | 36s |
-
- ### Extension Workflow
-
- 1. Calculate optimal initial duration using the formula above
- 2. Generate initial video with preview model (720p) using calculated duration
- 3. Extend N times with `generateVideoWithVeo(video=<previous_video_gcs_uri_or_url>)`
- 4. Each extension returns a **COMPLETE accumulated video** (no concatenation needed)
-
- **IMPORTANT**: The Gemini video extension API returns a complete video file containing the initial video plus all extensions merged together. You do NOT need to concatenate segments - just use the final extended video directly.
-
- **Note**: Prefer using GCS URI (`gs://bucket/path`) for video extension - it's more efficient than HTTP URL.
-
- **Example**: Target 22 seconds
- - n = ⌈(22 - 8) / 7⌉ = 2
- - Initial = 22 - (2 × 7) = 8s
- - Result: 8s initial + 2 extensions = 22s ✓
-
- **Note**: Extension uses the video URL directly - no need to call `uploadAndGetVid`.
-
- ---
-
- ## Long Videos Strategy
-
- ### Approach 1: First-Last-Frame Storyboard
-
- **CRITICAL**: This workflow has strict sequential dependencies. Do NOT skip steps.
-
- Use Veo's first-last-frame for precise visual control in multi-segment videos.
-
- **Frame Sharing Principle**: `N video segments require N+1 keyframes`
-
- **Workflow** (MUST follow in order):
-
- **Step 1: Generate ALL keyframes FIRST**
-
- - Generate N+1 keyframes with `generateImage`
- - Pass previous frames as `imageUrls` for style consistency
- - Example: 6 videos need 7 keyframes (frame1 to frame7)
-
- **Step 2: Generate videos with shared frames**
-
- - Video 1: `generateVideoWithVeo(image=frame1, lastFrame=frame2)`
- - Video 2: `generateVideoWithVeo(image=frame2, lastFrame=frame3)` ← frame2 shared!
- - ... continue for all segments
- - Videos CAN be generated in parallel AFTER all keyframes are ready
-
- **Step 3: Concatenate**
-
- - Load `editing-videos` skill, then use `submitDirectEditTask` to combine all segments
- - **CRITICAL: NO transitions** - shared keyframes ensure seamless visual continuity
-
- ### Approach 2: Video Extension
-
- For continuous long videos without cuts.
-
- **Workflow**:
-
- 1. Generate initial 8-second video with preview model (720p)
- 2. Extend with `generateVideoWithVeo(video=<previous_video_gcs_uri_or_url>)`
- 3. Repeat up to 4 times (max 36 seconds total)
-
- **Note**: Prefer using GCS URI (`gs://bucket/path`) for video extension - it's more efficient than HTTP URL.
-
- **Limitations**:
-
- - 720p only
- - 16:9 or 9:16 only
- - Must use preview models
-
- ---
-
- ## Video Concatenation Guide
-
- When concatenating multiple generated video segments, use `submitDirectEditTask` with the Track structure.
-
- ### Prerequisites
-
- 1. Load the `editing-videos` skill first
- 2. Upload each video segment using `uploadAndGetVid` to get VID
-
- ### Concatenation Workflow
-
- **Step 1**: Upload all video segments
-
- ```
- For each segment URL:
- uploadAndGetVid(url) → vid://segment_N, duration_N, width, height
- ```
-
- **Step 2**: Build Track structure
-
- **CRITICAL**: Choose the correct method based on how videos were generated:
+ 1. Call `generateVideoWithGrok` with prompt and optional parameters.
+ 2. Poll `getGrokVideoStatus` every 30 seconds.
+ 3. Maximum wait: 5 minutes.
+ 4. Save completed output to draft unless the user explicitly asks for media library only.
- #### For First-Last-Frame Videos - NO TRANSITIONS
+ ## Prompt Structure
- ⚠️ **MUST use direct concatenation. DO NOT add transitions.**
+ `[Subject & Background] + [Action] + [Style] + [Camera] + [Atmosphere] + [Audio]`
- When videos are generated using Veo first-last-frame with shared keyframes:
+ ### Subject & Background
- ```json
- {
- "Canvas": { "Width": 1920, "Height": 1080 },
- "Track": [[
- { "Type": "video", "Source": "vid://segment_1", "TargetTime": [0, 8000] },
- { "Type": "video", "Source": "vid://segment_2", "TargetTime": [8000, 16000] },
- { "Type": "video", "Source": "vid://segment_3", "TargetTime": [16000, 24000] }
- ]]
- }
- ```
+ Specify the main focus and environmental context.
- **Why no transitions?** The shared keyframes between segments already ensure seamless visual continuity.
+ Examples:
+ - "A young woman with long black hair wearing a red dress"
+ - "White concrete apartment building with organic shapes and lush greenery"
- #### For Independent Segments - Optional Transitions
+ ### Action
- When videos are independently generated (not using shared keyframes), transitions can help smooth the connection:
+ Describe what the subject is doing.
- ```json
- {
- "Canvas": { "Width": 1920, "Height": 1080 },
- "Track": [[
- {
- "Type": "video",
- "Source": "vid://segment_1",
- "TargetTime": [0, 8000],
- "Extra": [{ "Type": "transition", "Source": "1182376", "Duration": 500 }]
- },
- {
- "Type": "video",
- "Source": "vid://segment_2",
- "TargetTime": [7500, 15500],
- "Extra": [{ "Type": "transition", "Source": "1182376", "Duration": 500 }]
- },
- { "Type": "video", "Source": "vid://segment_3", "TargetTime": [15000, 23000] }
- ]]
- }
- ```
+ Examples:
+ - "walks slowly towards the camera"
+ - "transforms from liquid to solid"
- **Key Points**:
+ ### Style
- - Time unit is **milliseconds** (1s = 1000ms)
- - For transitions, overlap TargetTime by the transition Duration
- - Last segment does NOT need a transition
- - **First-last-frame videos MUST use direct concatenation (no transitions)**
+ Add aesthetic direction.
- **Step 3**: Submit and poll
+ Examples:
+ - Film noir
+ - Surrealism
+ - Cyberpunk
+ - 3D cartoon animation
+ - Cinematic documentary
- ```
- submitDirectEditTask(Canvas, Track) → taskId
- wait 90 seconds
- poll getVideoEditTaskStatus(taskId) every 30 seconds until completed
- ```
+ ### Camera
- ---
+ Describe movement and framing.
- ## Output Requirements
+ Examples:
+ - "slow dolly-in"
+ - "handheld tracking shot"
+ - "close-up with shallow depth of field"
- **MANDATORY**: After generating videos, you MUST output ALL video URLs clearly.
+ ### Atmosphere
- ### Single Video Output
+ Describe lighting, color, mood, and setting.
- ```
- **Generated Video**:
- 
+ Examples:
+ - "warm sunset lighting"
+ - "misty morning atmosphere"
+ - "high contrast studio lighting"
- Video URL: url
- ```
+ ### Audio
- ### Multiple Videos Output (Batch/Storyboard)
+ Include useful audio cues when relevant.
- ```
- All N video segments completed!
+ Examples:
+ - "soft ambient city sounds"
+ - "gentle piano music"
+ - "footsteps on wet pavement"
- | # | Status | Preview | URL |
- |---|--------|---------|-----|
- | 1 | ✅ |  | url1 |
- | 2 | ✅ |  | url2 |
- | ... | ... | ... | ... |
+ ## Aspect Ratio Guidance
- **All Video URLs**:
- 1. url1
- 2. url2
- ...
- ```
+ | Platform / Use | Aspect Ratio |
+ |----------------|--------------|
+ | TikTok, Reels, Shorts, mobile social video | `9:16` |
+ | YouTube landscape, website hero, presentation | `16:9` |
+ | Profile or square feed asset | `1:1` |
+ | Portrait-style content | `3:4` or `2:3` |
+ | Landscape social feed | `4:3` or `3:2` |
- ### After Concatenation
+ ## Long Video Strategy
- ```
- **Final Video** (concatenated from N segments):
- 
+ For videos longer than 15 seconds:
- Final Video URL: final_url
- ```
+ 1. Split the video into independent 15-second-or-shorter segments.
+ 2. Generate each segment with `generateVideoWithGrok`.
+ 3. Poll every task until complete.
+ 4. Load `editing-videos`.
+ 5. Use the video editing tool to concatenate the completed segments.
- **IMPORTANT**: Never end a video generation task without explicitly listing all video URLs.
+ Keep prompts visually consistent across segments by repeating subject, style, lighting, camera language, and environment details.