remotion-to-hyperframes · diff
git:20261004.0c3e244 to git:20261021.6882293
32 added, 115 removed. Audit A to A.
---
name: remotion-to-hyperframes
- description: 'Port an existing Remotion (React) composition''s source to HyperFrames HTML. Use ONLY on an explicit ask to port/convert/migrate/translate a Remotion source — one-way, Remotion-only. A passing Remotion mention, reference-only code, or "make something like my Remotion video" is a fresh build (/general-video). Unclear → /hyperframes.'
+ description: Bring existing video code or footage into HyperFrames and verify it frame by frame against the original - Remotion first, also Motion Canvas, Lottie, Three.js, p5.js, canvas or plain HTML/CSS animation, or renders from Manim, After Effects or Blender. Use to port, convert, migrate or reuse work made with another tool.
---
- **Plugin installs:** Before setup or freshness commands, follow [plugin execution rules](../hyperframes/references/plugin-installation.md) when this skill is inside a HyperFrames plugin. Standalone installs keep the update instructions below.
-
- > **First, keep this skill fresh — confirm with the user before running:** `npx hyperframes skills update remotion-to-hyperframes`. A fast no-op when everything is current; otherwise it refreshes this skill plus the core domain skills it depends on before you rely on them.
-
- # Remotion to HyperFrames
-
- > **The front door is `/hyperframes`.** Use this **only** to port an existing **Remotion** (React) composition's source into HyperFrames, one way. Authoring a **new** composition, re-creating from a non-Remotion source (After Effects, Framer Motion, plain React / CSS — there is no Remotion source to translate), a passing Remotion mention, or any uncertainty → read `/hyperframes` first: the intent layer owns every route decision.
-
- ## Overview
-
- Translate Remotion (React-based) video compositions into HyperFrames (HTML + GSAP) compositions. Most Remotion idioms have direct HyperFrames equivalents — the translation is mechanical for ~80% of typical compositions. This skill encodes the mapping and guards against the lossy 20% by refusing to translate patterns that don't fit HF's seek-driven model and recommending the runtime interop pattern from [PR #214](https://github.com/heygen-com/hyperframes/pull/214) instead.
-
- The skill ships with a **tiered test corpus** (T1–T4, 4 fixtures total) that grades translations against measured SSIM thresholds. Don't translate without running the eval — a translation that "looks right" but renders 0.05 SSIM lower than the validated baseline is silently wrong.
-
- ## When to use
-
- **Use this skill ONLY when the user explicitly asks to migrate from Remotion.** Example trigger phrases:
-
- - "port my Remotion project to HyperFrames"
- - "convert this Remotion code to HyperFrames"
- - "migrate from Remotion"
- - "translate this Remotion comp"
- - "rewrite this as HyperFrames HTML"
-
- **Do NOT use this skill when:**
-
- - (a) The user is authoring a **new** HyperFrames composition, even if they have or are A/B-testing a similar Remotion video.
- - (b) The user mentions Remotion in passing without asking for migration.
- - (c) The user shares Remotion code as reference material rather than asking for a translation.
- - (d) The user asks for "the same video as my Remotion one" without explicitly asking to migrate the source — treat that as a fresh HyperFrames build.
-
- **NOT SUPPORTED (decline — this is not what this skill does):**
-
- - **The reverse direction.** Exporting a HyperFrames composition back out _to_ Remotion (or to any other framework) is not a workflow — the translation is Remotion → HyperFrames only. Say so plainly.
- - **Non-Remotion sources.** An After Effects project (`.aep`), a Framer Motion / plain-React / CSS animation, or any other tool's source is not a Remotion composition — there is no Remotion source to translate. Re-create it natively via `/general-video`, or decline if HyperFrames can't represent it.
-
- When in doubt, return to `/hyperframes`; its intent layer picks the route.
-
- ## Workflow
-
- ### Step 1: Lint the source
-
- Run [`scripts/lint_source.py`](scripts/lint_source.py) over the Remotion source directory. The lint detects patterns that can't translate cleanly:
-
- - **Blockers** (refuse + recommend interop): `useState`, `useReducer`, `useEffect`/`useLayoutEffect` with non-empty deps, async `calculateMetadata`, third-party React UI libraries (MUI, Chakra, Mantine, antd, shadcn, Radix, NextUI).
- - **Warnings** (translate after dropping the construct): `@remotion/lambda` config, `delayRender`, `useCallback`, `useMemo`, custom hooks.
- - **Info** (translate with note): `staticFile`, `interpolateColors`.
-
- If any blocker fires, **stop**. Read [`references/escape-hatch.md`](references/escape-hatch.md) and surface the recommendation message. Warnings don't stop translation — drop the offending construct in step 3 and note the gap in `TRANSLATION_NOTES.md`. `@remotion/lambda` config is the canonical warning case: the skill drops the import + `renderMediaOnLambda(...)` calls but translates the rest of the composition.
-
- ### Step 2: Plan the translation
-
- Read [`references/api-map.md`](references/api-map.md) — the index of every Remotion API and its HF equivalent or per-topic reference. Identify which topic references you'll need based on what the source uses:
-
- | Source contains | Load reference |
- | ------------------------------------------------------------------------- | --------------------------------------------- |
- | `Composition`, `defaultProps`, `schema`, `calculateMetadata` | [`parameters.md`](references/parameters.md) |
- | `Sequence`, `Series`, `Loop`, `AbsoluteFill`, `Freeze` | [`sequencing.md`](references/sequencing.md) |
- | `useCurrentFrame`, `interpolate`, `spring`, `Easing`, `interpolateColors` | [`timing.md`](references/timing.md) |
- | `Audio`, `Video`, `Img`, `IFrame`, `staticFile`, `delayRender` | [`media.md`](references/media.md) |
- | `TransitionSeries`, `@remotion/transitions` | [`transitions.md`](references/transitions.md) |
- | `@remotion/lottie` | [`lottie.md`](references/lottie.md) |
- | `@remotion/google-fonts/<Family>`, `Font.loadFont`, `@font-face` | [`fonts.md`](references/fonts.md) |
-
- Don't load all of them — load only what the specific source needs.
-
- **Search the live catalog for any visual effect the table does not map.** When the source paints a look with no HF API equivalent — a scanline/CRT overlay, a glitch or chromatic-aberration pass, a shader wipe, a film-grain treatment — run `npx hyperframes catalog --query "<the effect, in plain English>" --json` before hand-writing it in GSAP. The search needs **nothing installed**: no project, no prior `add`, no account. It ranks the whole hosted registry (~400 blocks and components) from any directory, and `transitions.md` already takes this route for `clockWipe()` / `iris()` via `npx hyperframes add sdf-iris`. A real component is closer to the source than a hand-approximation, so it usually raises the SSIM rather than lowering it — but the render diff in Step 4 is still the arbiter. Hand-write the effect when a search returns nothing that fits, and record the substitution in `TRANSLATION_NOTES.md` either way.
-
- ### Step 3: Generate the HF composition
-
- Emit `index.html` with:
-
- - Root `<div id="stage">` carrying the composition's `data-composition-id`, `data-start="0"`, `data-duration` (in seconds), `data-fps`, `data-width`, `data-height`, plus one `data-*` per scalar prop.
- - One host `<div>` per scene with `data-composition-src="compositions/<scene>.html"` and `data-start` / `data-duration` / `data-track-index`. The root holds no nested layout.
- - One `compositions/<scene>.html` per scene (a `<template>` sub-composition): its inline `<style>` for layout (CSS sets the `from` state of every animated property), its markup, and one paused `gsap.timeline({paused: true})` in the scene's local time. Every Remotion `useCurrentFrame()` derivation becomes a tween on that timeline at the offset within the scene.
- - `window.__timelines["<scene-id>"] = tl;` in each scene file, and `window.__timelines["<composition-id>"]` for the root's own (possibly empty) timeline.
-
- Custom React subcomponents inline as repeated HTML using the prop interface as the template (see [`parameters.md`](references/parameters.md) for the per-instance `data-*` pattern).
-
- ### Step 4: Validate
-
- Run the eval harness — [`references/eval.md`](references/eval.md) for the full guide. Quick path:
-
- ```bash
- # Render Remotion baseline (after npm install in the fixture)
- cd remotion-src && npx remotion render <CompositionId> out/baseline.mp4
-
- # Render HF translation
- cd ../hf-src && npx hyperframes render --skill=remotion-to-hyperframes --output ../hf.mp4
-
- # SSIM diff
- ../../scripts/render_diff.sh ./remotion-src/out/baseline.mp4 ./hf.mp4 ./diff
- ```
+ **Plugin installs:** Before setup or freshness commands, follow [plugin execution rules](../hyperframes/references/plugin-installation.md) when this skill is inside a HyperFrames plugin.
- Threshold: ~0.02 below `p05` of the source's complexity tier (see `eval.md`'s validated thresholds table). If the diff fails, run [`scripts/frame_strip.sh`](scripts/frame_strip.sh) to see _which_ frames diverged, then re-read the relevant timing/sequencing/media reference.
+ # Import into HyperFrames
- **Critical**: both renders must use matching pixel format. Set `Config.setVideoImageFormat("png")` + `Config.setColorSpace("bt709")` in the Remotion source's `remotion.config.ts` — otherwise the diff measures encoder differences (~0.05 SSIM hit), not translation fidelity.
+ HyperFrames is the container: its tracks, sub-compositions and variables are editable in Studio whatever runs inside a clip. Bring each part in at the highest level it supports and record which level you used.
- ### Step 5: Document gaps
+ | Level | For | How | Editable in Studio afterwards |
+ | --- | --- | --- | --- |
+ | Native | code that can become HTML + GSAP: most Remotion, HTML/CSS/GSAP animation, simple Motion Canvas | translate it, then verify frame by frame | every element; variables, catalog, presets apply |
+ | Hosted | code that can draw any requested frame in a browser: canvas, p5.js, Three.js, Lottie, WebGL, other JS animation | keep the code and drive it from HyperFrames time | timing, position, exposed parameters |
+ | Media | anything that outputs video or frames: Manim, After Effects, Blender, PIL, moviepy | render to a file and use it as a clip | cut, place, layer; not the pixels inside |
- Anything that didn't translate cleanly (volume ramps dropped, custom presentations approximated, fonts substituted) gets a `TRANSLATION_NOTES.md` written next to the HF output. See [`references/limitations.md`](references/limitations.md) for the format.
+ Code that depends on wall-clock time, live physics or the network can only come in as media.
- ## What this skill explicitly does NOT do
+ ## Native: translate, then verify
- - **Translate React state machines.** Compositions that drive animation via `useState` + `useEffect` are not deterministic frame-capture targets in HyperFrames' seek-driven model. Recommend the runtime interop pattern.
- - **Run Remotion's render pipeline alongside HyperFrames.** That's the runtime interop pattern from [PR #214](https://github.com/heygen-com/hyperframes/pull/214) — a separate solution for compositions that fail this skill's lint.
+ 1. For Remotion, run `python3 <SKILL_DIR>/scripts/lint_source.py <source dir>` first. Blockers (`useState`, `useReducer`, `useEffect` with dependencies, async `calculateMetadata`, React UI kits such as MUI, Chakra, Mantine, antd, shadcn, Radix, NextUI) don't translate: host them or refactor them out, never approximate (`references/escape-hatch.md`). Warnings (`@remotion/lambda` config, `delayRender`, `useMemo`, custom hooks) are dropped and noted.
+ 2. Translate to the `/hyperframes` contract: one sub-composition per scene, times in seconds (`frame / fps`), asset paths root-relative. Each Remotion API's equivalent is in `references/api-map.md`.
+ 3. Keep it editable: the source's props become `data-composition-variables` bound with `data-var-text`, `data-var-src` or `var(--id)`; each `Sequence` becomes a timed clip or a sub-composition; a reused component becomes one sub-composition mounted several times with `data-variable-values`.
+ 4. Compare frame by frame: `npx hyperframes snapshot --at <t,…> --against <original.mp4>` saves the original's frame at each time next to yours. Sample every scene and both sides of every cut; look for one late or recoloured element, not only the overall picture. For a scored check, `scripts/render_diff.sh <original.mp4> <hf.mp4> <out>` gives SSIM per frame and `scripts/frame_strip.sh` shows which frames diverged (thresholds per complexity tier in `references/eval.md`). Render the Remotion side with `Config.setVideoImageFormat("png")` and `Config.setColorSpace("bt709")`, or the diff measures the encoders (about 0.05 SSIM).
+ 5. Write what didn't translate cleanly (dropped volume ramps, approximated presentations, substituted fonts) to `TRANSLATION_NOTES.md`.
- (`@remotion/lambda` is _not_ a blocker — Lambda config is deployment, not animation. The skill drops it as a warning and translates the rest. See [`references/escape-hatch.md`](references/escape-hatch.md).)
+ | Remotion | Read |
+ | --- | --- |
+ | `Composition`, `defaultProps`, `schema`, `calculateMetadata` | `references/parameters.md` |
+ | `Sequence`, `Series`, `Loop`, `AbsoluteFill`, `Freeze` | `references/sequencing.md` |
+ | `useCurrentFrame`, `interpolate`, `spring`, `Easing`, `interpolateColors` | `references/timing.md` |
+ | `Audio`, `Video`, `Img`, `IFrame`, `staticFile`, `delayRender` | `references/media.md` |
+ | `TransitionSeries`, `@remotion/transitions` | `references/transitions.md` |
+ | `@remotion/lottie` | `references/lottie.md` |
+ | `@remotion/google-fonts`, `Font.loadFont`, `@font-face` | `references/fonts.md` |
+ | What can't be ported | `references/limitations.md` |
- ## How to grade your own translation
+ Before hand-building a named visual the source used, search `npx hyperframes catalog --query "<the visual, in English>" --json`: it ranks the whole registry with nothing installed.
- Run the test corpus orchestrator:
+ ## Hosted
- ```bash
- ./assets/test-corpus/run.sh
- ```
+ Keep the code and let HyperFrames own the clock: it draws the frame for the time it is given. The seek protocol and the built-in adapters for Lottie, Three.js, anime.js, WAAPI and CSS animations are in `../hyperframes/references/runtimes.md`. No `requestAnimationFrame` loop or timer as the time source, seeded randomness only, and `data-duration` on the root when nothing else defines the length.
- It runs T1, T2, T3 (render + diff) and T4 (lint validation), prints a per-tier pass/fail table, and emits an aggregate JSON report. Use this to verify the skill is working end-to-end on a clean checkout — and as a regression check after editing any reference.
+ ## Media
- Validated baseline (as of 2026-04-27):
+ `<video class="clip" data-has-audio="true" src="assets/x.mp4" ...>`, or `muted` when it is silent. Nothing else is needed.
- | Tier | Composition shape | Mean SSIM | Threshold |
- | ---- | ------------------------------------------- | --------- | --------- |
- | T1 | single-element fade-in | 0.974 | 0.95 |
- | T2 | multi-scene + spring + audio + image | 0.985 | 0.95 |
- | T3 | data-driven, custom subcomponents, count-up | 0.953 | 0.90 |
- | T4 | escape-hatch (8 lint cases) | 8/8 pass | n/a |
+ In the final message, list each imported part with its level, what a person can change in Studio, and how the frame-by-frame comparison came out.