spark-local-media-generation · diff
v0.1.0 to v0.1.0
38 added, 18 removed. Audit A to A.
---
name: spark-local-media-generation
version: "0.1.0"
- description: Generate advertisement frames and video clips on the local DGX Spark with the cineloom CLI (ComfyUI running Qwen-Image, Qwen-Image-Edit and Wan2.2) instead of a cloud API. Use when storyboard frames, product or character shots, or shot clips must be produced, when material is confidential or unreleased, or when choosing between local and cloud generation.
+ description: Generate a product hero still, storyboard frames anchored to it, and video clips on the local DGX Spark with the cineloom CLI (ComfyUI running Qwen-Image, Qwen-Image-Edit and Wan2.2 14B) instead of a cloud API. Use when product shots, storyboard frames or shot clips must be produced, when material is confidential or unreleased, or when choosing between local and cloud generation.
license: MIT
metadata:
author: 方舟团队 (Team Ark)
tags:
- dgx-spark
- comfyui
- qwen-image
- wan2.2
---
# Spark local media generation
- Frames first, clips second. A clip is generated from its approved frame, so product
- identity is decided at the frame stage where a retry costs seconds, not minutes.
+ Hero still first, frames second, clips last. Each step is anchored to the one before it,
+ so product identity is decided where a retry costs seconds, not minutes.
## Workflow
- 1. Check memory with `spark-model-scheduler` before the first frame and before the first clip.
- 2. Generate each storyboard frame. References switch the request to the edit model:
+ 1. Check memory with `spark-model-scheduler` before each phase. Only one diffusion family should be resident at a time; run `cineloom mem free` when switching between them.
+ 2. Generate the **product hero still** from the storyboard's `product` description on a plain backdrop, and gate it:
```bash
+ cineloom image --out projects/<id>/refs/product.png --size 1328x1328 --prompt "Studio product photograph of <product> ..."
+ cineloom qa --frame projects/<id>/refs/product.png --expect "one container, upright, label reads \"<brand>\" clearly"
+ ```
+
+ If the user supplied real product photos, use those instead and skip this step.
+ 3. Generate every storyboard frame **from the hero still**. A reference switches the request to the edit model and keeps the product the same product:
+
+ ```bash
cineloom image --project <id> --shot 1 --size 1080x1920 \
- --prompt "<image_prompt>" --ref projects/<id>/refs/can_front.png
+ --prompt "<image_prompt>. Image 1 is the product: keep its shape, colours and label exactly as in image 1. No captions, no subtitles, no lettering except the product label. <style>" \
+ --ref projects/<id>/refs/product.png
```
- Sizes above the native budget (1328x1328 pixels in total) are generated natively and upscaled; the output says `upscaled: true`.
- 3. Run `shot-quality-gate` on every frame. Only approved frames move on.
- 4. Generate each clip from its approved frame:
+ Generate two candidates per shot and let `shot-quality-gate` pick: the spread between seeds is often larger than what a prompt edit buys.
+ 4. Only gated frames move on. Generate each clip from its approved frame:
```bash
- cineloom video --project <id> --shot 1 --duration 5 --ratio 9:16 --resolution 720p \
- --prompt "<video_prompt>" --first-frame projects/<id>/frames/shot_001.png
+ cineloom video --project <id> --shot 1 --model wan22-14b --duration 5 --ratio 9:16 --resolution 720p \
+ --prompt "<video_prompt>" --first-frame projects/<id>/frames/shot_001.png --keep-loaded
```
- Durations are multiples of 5 s. Ten seconds is two segments, the second continued from the first one's last frame. Add `--keep-loaded` on every clip except the last so the video model stays in memory between shots.
- 5. Every command prints measured `seconds` and records the asset in the project with `execution: local-dgx-spark`. Keep these numbers for the delivery report.
+ Drop `--keep-loaded` on the last clip so the video model is released.
+ 5. Every command prints measured `seconds` and records the asset with `execution: local-dgx-spark`. Keep these numbers for the delivery report.
+ ## What the models do, measured on this machine
+
+ | Step | Model | Cost |
+ |---|---|---|
+ | Frame, no reference, 1080x1920 | Qwen-Image + 8-step LoRA | 22.7 s warm, 45.8 s cold |
+ | Frame from the hero still | Qwen-Image-Edit-2509 + 8-step LoRA | about 77 s |
+ | 5 s clip, 720p, from a first frame | Wan2.2 I2V A14B fp8 + 4-step LoRAs (default) | 353 s |
+ | 5 s clip, 720p | Wan2.2 TI2V-5B (also text-to-video) | 365 s |
+
+ Sizes above the native budget (1328x1328 pixels in total) are generated natively and upscaled.
+
## Prompting the local models
- - Qwen-Image renders Chinese and English text. Quote the exact text and say where it sits.
- - With references, name them in order: "image 1 is the product, image 2 is the model". Product first; more than three are ignored.
- - Wan2.2 wants one camera move and one action per segment, described plainly. It produces no audio.
+ - **Never ask the image model for captions or slogans.** Titles are typeset in post. Measured failures: a caption painted twice, the style token "50mm" painted as text, the rare character "泠" drawn as "冷". The only lettering in a frame is the product label, and it comes from the hero still.
+ - Keep numbers and lens specs out of image prompts for the same reason.
+ - With references, say what each one is: "Image 1 is the product". At most three; product first.
+ - The 14B video model wants one camera move and one action, described plainly. For product-only shots end the prompt with "No people and no hands enter the frame" - otherwise it tends to add a hand reaching for the product. It produces no audio.
## When something fails
- - `ComfyUI rejected the workflow` or a missing model file: run `cineloom doctor`. It compares the workflow templates with the running ComfyUI and lists unknown nodes and missing weights.
+ - `ComfyUI rejected the workflow` or a missing model file: run `cineloom doctor`. It compares every workflow template with the running ComfyUI and lists unknown nodes and missing weights.
- Connection refused: ComfyUI is not up. See `deploy/spark/`.
- - Out of memory or heavy swapping: run `cineloom mem free`, then follow `spark-model-scheduler`.
+ - Allocation failures in the GPU driver log, or heavy swapping: more than one diffusion family is resident. Run `cineloom mem free`, then follow `spark-model-scheduler`.
Model files and graphs behind each route: `references/workflows.md`.