broll-actor-copy · diff
git:20260717.a87eb1b to git:20260908.f4e5dbb
67 added, 15 removed. Audit A to A.
---
name: broll-actor-copy
description: >-
Generate a B-roll clip where OUR avatar COPIES the motion of a driving video —
- same movement, gestures, expressions and lip movements, but performed by the
- avatar — via ByteDance DreamActor M2.0 (bytedance/dreamactor-m2.0 on Replicate).
- Takes an avatar, a location/look (or its default) and a driving video, resolves
- the look's identity-anchored hero image, and transfers the video's performance
- onto it. DreamActor is universal (humans, cartoons, animals) and needs no pose
- estimation. The driving video must show exactly ONE animated character (person
- or animal); multi-character sources are split into single-character segments and
- stitched. Output keeps the reference image's resolution (9:16 hero →
- 9:16 clip); the clip is muted and avatar-reel-composer re-lays narration. Use
- when the user wants the avatar to mimic a reference video's movement, a motion
- copy / performance transfer / actor copy / "copiar el movimiento del video"
- B-roll, or mentions dreamactor / DreamActor M2.0.
+ same movement, gestures, expressions and lip movements — via ByteDance
+ DreamActor M2.0 (bytedance/dreamactor-m2.0 on Replicate). Takes an avatar, a
+ location/look (or its default) and a driving video, resolves the look's hero
+ image, and transfers the video's performance onto it. Works with humans,
+ cartoons and animals. The driving video must show exactly ONE animated
+ character; multi-character sources are split into single-character segments
+ and stitched. A human driver should match a human avatar's gender (else warn +
+ ask: the face distorts) and the reference scene should include the objects the
+ driver interacts with. Output keeps the reference image's resolution (9:16
+ hero → 9:16 clip); the clip is muted and avatar-reel-composer re-lays
+ narration. Use when the user wants the avatar to mimic a video's movement, a
+ motion copy / performance transfer / actor copy / "copiar el movimiento del
+ video" B-roll, or mentions dreamactor.
disable-model-invocation: true
---
# B-roll Actor Copy (the avatar copies a video's motion)
The **motion-transfer** member of the broll family. Where `broll-avatar-camera`
animates ONE start frame with a text action prompt and `broll-story` animates an
illustrated storyboard, **`broll-actor-copy` copies a real driving video's whole
performance onto our avatar** — the avatar re-does the exact movement, gestures,
expressions and lip movements of the video, keeping its own identity + look.
It is a thin **orchestrator** over one Replicate model:
**`bytedance/dreamactor-m2.0`** — image (the avatar's look hero) + driving video
→ a video of the avatar performing the video's motion.
> **Why DreamActor M2.0:** it learns motion from raw video pixels instead of
> extracting a human skeleton, so it animates realistic humans, stylized
> drawings, cartoons and animals alike — ideal for our varied avatars. It keeps
> the reference identity while accurately copying the motion, and the **output
> resolution equals the reference image's resolution**.
## Inputs (gather what's missing)
| Input | Used for |
|---|---|
| **Avatar** | a bare name (e.g. `nora` → `avatares/nora/`) or a path — its hero image is the appearance source |
| **Location / look** | which look to wear: `default` (top-level `scene.json`+`refs/`) or a name under `<avatar>/locations/<loc>/` (default: `default`) |
| **Driving video** | the MOTION source — its movement/expressions/lips are copied onto the avatar. **Must show exactly ONE animated character** (person or animal) in scene |
The look resolves to the identity-anchored hero automatically:
- default → `<avatar>/refs/<slug>_hero.png`
- a location → `<avatar>/locations/<loc>/refs/<slug>__<loc>_hero.png`
Override the appearance entirely with `--image PATH` (its resolution then sets
the output resolution).
## Prerequisites
- **Replicate token** (shared, auto-discovered) — for `bytedance/dreamactor-m2.0`.
Set/refresh: `python3 scripts/setup_key.py YOUR_REPLICATE_API_TOKEN`.
- `pip3 install -r scripts/requirements.txt` (replicate, pillow).
- **ffmpeg / ffprobe** on PATH — trims/scales the driving video and mutes the clip.
- An **existing avatar** with a hero (as produced by `avatar-invent` / `avatar-location`).
## Pipeline
```
avatar look -> hero image + driving video
│
1 resolve the look's hero (or --image); fit it to 480x480..1920x1080, <=4.7MB (only if needed)
│ fit/trim the driving video to <=30s within a 2048x1440 box (only if needed)
│
2 make_actor_copy.py -> bytedance/dreamactor-m2.0
│ inputs: image (avatar hero) + video (driver) + cut_first_second
│ -> download -> mute (-an) -> <avatar>/broll/actor-copy/<NNN>_<slug>.mp4 + manifest.json
│
3 hand off to avatar-reel-composer as a broll scene (broll_source: existing)
```
## Hard rules
- **One character in the driving video.** DreamActor copies a SINGLE subject, so
the driving video must show **exactly one animated character** (a person OR an
animal) visible in scene. Our reference hero is one avatar → match it with a
one-character driver. See the multi-character workflow below.
+ - **Human driver → same gender as the avatar (or warn + ask).** DreamActor also
+ transfers the driver's facial gestures and the position of facial features. When
+ both driver and avatar are human but of different genders, the avatar's face
+ gets distorted / drifts off-identity. Before generating with a human driver,
+ check its apparent gender against the avatar's: if they differ, **tell the user
+ and ask (AskQuestion) whether to continue anyway or pick another driver**.
+ A human driver animating an ANIMAL avatar is fine — the face-matching issue is
+ human-to-human.
+ - **The reference scene must support the driver's interactions.** If the driving
+ actor interacts with objects (a bench, a mat, a bar, a chair…), the reference
+ image's location should contain a **similar interactable element** — not
+ necessarily identical, just coherent with the avatar's scene — placed where the
+ motion needs it. Otherwise DreamActor invents props mid-clip and the motion
+ looks unnatural. Also frame the reference **half-body (mid-thigh up)** when the
+ detail that matters is the face/torso: DreamActor plausibly fills in whatever
+ is outside the reference frame, and a closer reference keeps the face sharp.
- **The reference image sets the output resolution.** Feed a **9:16 hero** for a
9:16 reel clip, or a **16:9 hero** for a 16:9 YouTube clip. When it falls back
to a camera angle, pass `--aspect 16:9` to prefer the avatar's `_169.png`
angle. Keep the subject clear and framed like the shot you want.
- **Driving video ≤ 30s.** Longer input is auto-trimmed to the first 30s; pick a
segment with `--trim-start` / `--trim-duration` (or `--segments`).
- **Muted output.** The clip is muted on disk; `avatar-reel-composer` re-lays the
single master narration. `--keep-audio` only for a standalone QA preview.
- **The motion comes from the VIDEO, not a text prompt.** There is no
`video_prompt`; choose a driving clip that already performs the movement you want.
## If the driving video has more than one character
DreamActor can only copy ONE subject. Before generating, **verify the driving
video shows a single animated character** (visually, or with the
`video-scene-analysis` skill). If it shows more than one:
1. **Tell the user** the driving video must have only one animated character to
copy (this is a model limitation, not a preference).
2. **Ask the user** (use the AskQuestion tool) whether to:
- **(a)** provide/pick a different clip that already has a single character, or
- **(b)** split the video into time **segments where only one character
appears**, process each, and **stitch** them into one clip via `--segments`.
For **(b)**, pass the single-character ranges to `--segments` (seconds or
`M:SS`/`H:MM:SS`); each range is animated separately with the same avatar hero and
the muted results are concatenated into one clip:
```bash
python3 broll-actor-copy/scripts/make_actor_copy.py nora \
--video downloads/interview.mp4 --segments "0-8,15-22,0:40-0:52" \
--slug nora-copies-host
```
Each segment must be ≤ 30s and contain a single character; the stitched output is
always muted (VO is re-laid later).
+ ## If the driver's gender differs from the avatar's (human ↔ human)
+ DreamActor mirrors the driver's facial gestures and feature placement, so a
+ female driver on a male avatar (or vice versa) tends to **distort the face**.
+ Before generating with a human driver:
+
+ 1. Compare the driver's apparent gender with the avatar's.
+ 2. If they differ, **inform the user** of the face-distortion risk and **ask**
+ (AskQuestion) whether to (a) use/pick a same-gender driving clip, or
+ (b) continue anyway accepting possible facial drift.
+
+ This only applies human-to-human: driving an **animal** avatar with a human
+ video is fine and supported.
+
+ ## Match the reference scene to the driver's interactions
+ Watch what the driving actor touches or leans on (bench, mat, bar, wall, chair…)
+ and make sure the reference image's location has a **coherent interactable
+ counterpart** in a workable position — the avatar's own version of it, not a copy
+ of the driver's scene. If the hero look lacks it, generate a scene-matched
+ reference (e.g. with `gpt-image-2` / `avatar-location`) instead of feeding a
+ mismatched hero: DreamActor will otherwise hallucinate props and the contact
+ points will look wrong. Prioritize detail where it matters — a **half-body
+ (mid-thigh up) reference** keeps face/torso sharp and lets the model infer legs
+ and clothing below the frame.
+
## Workflow
### 1 — Pick the driver + the look
Choose (or download) the driving video whose performance you want the avatar to
copy, and decide the avatar look. **Confirm the driver shows a single animated
- character** (see the multi-character workflow above if not). Preview the resolved
- inputs with `--dry-run` (no spend): it prints the resolved hero, prepared
- image/video sizes and the plan.
+ character** (see the multi-character workflow above if not), **check the
+ driver/avatar gender match** (warn + ask if they differ), and **check the
+ reference scene supports the driver's object interactions** (see above). Preview
+ the resolved inputs with `--dry-run` (no spend): it prints the resolved hero,
+ prepared image/video sizes and the plan.
### 2 — Generate
```bash
python3 broll-actor-copy/scripts/make_actor_copy.py nora \
--video downloads/dance_take.mp4 \
--slug nora-copies-dance
```
Writes `<avatar>/broll/actor-copy/<NNN>_<slug>.mp4` (muted, resolution = the hero)
plus a manifest entry, and prints a JSON summary. Useful flags:
- `--location <loc>` — use a specific look (else `default`).
- `--image PATH` — explicit appearance image (skips look resolution).
- `--segments "a-b,c-d"` — single-character time ranges to process + stitch (for
multi-character sources). Mutually exclusive with `--trim-*`.
- `--trim-start S` / `--trim-duration S` — use a single segment of a long driver.
- `--keep-first-second` — keep the model's 1s lead-in (default: cut it).
- `--keep-audio` — keep the generated audio (default: mute).
- `--model-version HASH` — pin a specific Replicate version.
- `--dry-run` — resolve/prepare inputs and print the plan without calling the model.
### 3 — Hand off to avatar-reel-composer
Drop the clip into a storyboard `broll` scene; the composer lays the master
narration over it:
```json
{ "id": "s3", "type": "broll", "broll_source": "existing",
"broll_clip": "avatares/nora/broll/actor-copy/001_nora-copies-dance.mp4",
"motion": "none",
"text": "the contiguous slice of the narration spoken over this beat" }
```
## Notes / troubleshooting
- **Identity drifts / off-look** → the reference hero controls the appearance;
use a cleaner, front-ish hero (or a specific `--location`) and re-run.
+ - **Face looks distorted / wrong gender vibes** → likely a human driver of a
+ different gender than the avatar (DreamActor copies facial gesture/feature
+ placement). Use a same-gender driving clip, or accept the drift knowingly.
+ - **Face mushy in full-body shots** → the face is too small in the reference;
+ switch to a half-body (mid-thigh up) reference so face/torso carry the detail
+ and let the model infer the rest.
+ - **Props appear/change mid-clip, unnatural contact** → the reference scene lacks
+ the element the driver interacts with; regenerate the reference with a coherent
+ interactable counterpart (bench/mat/bar…) in position.
- **Motion looks wrong / cropped** → the driver's framing matters; prefer a clean,
well-lit driving clip with the same shot scale you want (portrait vs full-body
both work — the model adapts).
- **Output wrong ratio** → it follows the reference image; crop the hero to 9:16
first (or pass a 9:16 `--image`).
- **Input too big / too long** → handled automatically (image ≤4.7MB within
1920x1080; video ≤30s within 2048x1440); pass `--no-fit-image` to send the
image untouched.
- **Cost / time** — billed per run on Replicate (video generation takes a few
minutes); check the model page for current pricing.
## Additional resources
- Model schema, resolution rules, look resolution details and examples: [REFERENCE.md](REFERENCE.md)
- Siblings: `broll-avatar-camera` (one frame + action prompt, p-video-avatar),
`broll-story` (illustrated storyboard, seedance-2), `broll-generator` (synthetic),
`broll-finder` (real footage). Looks come from `avatar-location`.
Consumer: `avatar-reel-composer` (`broll_source: existing`).