foley git:20261014.71d43e0A

Immutable. This exact content never changes and is served at /api/v1/blob/705ab16dd5e67d92.

---
name: foley
description: >
  Spatial Foley pipeline: tile a video into a grid, generate Foley audio
  per tile via fal.ai, review and flag bad tiles, then build a viewer page
  that mixes Foley tracks anchored to spatial rectangles via Web Audio.
---

# Foley

The foley pack covers three actions that together form the spatial Foley
pipeline.

## Capabilities

| ID | Kind | What it does |
|---|---|---|
| [`foley.tile_video`](../actions/tile_video/STAGE.md) | Action | Crop a video into an M×N grid of overlapping spatial tiles, emitting one clip and one first-frame PNG per tile, plus a `tiles.json` manifest. |
| [`foley.foley_review`](../actions/foley_review/STAGE.md) | Action | Build a static `review.html` pairing each tile clip with its generated Foley audio for sense-checking; reviewers flag bad tiles into `flagged.json`. |
| [`foley.foley_map`](../actions/foley_map/STAGE.md) | Action | The spatial Foley action: tiles the video, prompts a VLM for per-tile scene descriptions, generates Foley audio per tile via `fal.fal_foley`, runs review, and emits the spatial audio viewer. |

## Spatial Foley pipeline

The canonical Foley pipeline flows through these stages:

```
tile_video → (per-tile Foley generation via fal.fal_foley) → foley_review → foley_map action → spatial_audio_page
```

1. **`foley.tile_video`** — Crop the source video into an M×N grid of
   overlapping spatial tiles. Produces tile clips, first-frame PNGs, and
   a `tiles.json` manifest with each tile's rectangle in original-video
   coordinates.

2. **Foley generation per tile** — The `foley_map` action uses a
   vision-language model (`understanding.visual_understand`) to describe
   each tile's scene, then calls `fal.fal_foley` (fal.ai's
   hunyuan-video-foley model) to generate matching Foley audio for each
   tile clip. Requires `FAL_KEY`.

3. **`foley.foley_review`** — Build a static HTML review page that pairs
   each tile clip with its generated Foley audio. Human reviewers can
   flag bad tiles; flags are written to a `flagged.json` sidecar.

4. **`foley.foley_map` action** — Coordinates the full pipeline:
   tiles the video, dispatches VLM scene descriptions, generates Foley
   per tile, and runs the review gate. This is
   the single entry point for the end-to-end spatial Foley workflow.

## When to use

- Use `foley.tile_video` standalone when you only need to grid a video
  into tiles for downstream processing.
- Use `foley.foley_review` standalone when you have generated Foley audio
  that needs human sense-checking against the tile clips.
- Use `foley.foley_map` as the single entry point for the full spatial
  Foley pipeline — from source video to a reviewable audio manifest.

## Credentials

| Env var | Used by |
|---|---|
| `FAL_KEY` | `fal.fal_foley` (called by the foley_map action) |
| `OPENAI_API_KEY` | VLM scene description step in foley_map |

## SDK quick-start

```python
import astrid.sdk as sdk

# Run tile_video standalone
result = sdk.invoke(
    "foley.tile_video",
    inputs={"video": "./input.mp4"},
    out="./tiles",
)

# Run foley_review standalone
result = sdk.invoke(
    "foley.foley_review",
    inputs={"manifest": "./tiles/tiles.json"},
    out="./review",
)

# Run the full spatial Foley pipeline (action)
result = sdk.invoke(
    "foley.foley_map",
    inputs={"video": "./input.mp4"},
    out="./foley-run",
)
```