lai-align · diff
git:20260412.257645d to git:20260412.c7d541e
33 added, 205 removed. Audit A to A.
---
name: lai-align
- description: Use when user needs accurate/precise caption timing, or aligning captions with audio/video using forced alignment. Corrects caption timing to match actual speech. Uses LattifAI Lattice-1 model. Triggers on "align captions", "fix timing", "forced alignment", "对齐字幕", "校准时间轴", or when user has both audio and text that need synchronization.
+ description: Align existing captions to audio/video with word-level precision using the Lattice-1 model. Trigger when the user has both a media file AND a caption/transcript that need to be synchronized, or says "fix caption timing", "字幕对不上", "对齐字幕", "word-level timestamps", "karaoke timing", "timestamps are off". Do NOT trigger without existing text — use `/lai-transcribe` first.
allowed-tools: Read, Bash(lai:*)
---
# LattifAI Forced Alignment
- Precision audio-text forced alignment powered by the Lattice-1 model. Produces
- word-level and segment-level timestamps with sub-frame accuracy.
-
- **Requires LattifAI API Key** -- get one free via `lai auth trial` (see `/lai-setup`).
-
- ## When to Use
-
- - **Precise timing needed** -- user requests accurate timestamps or precise alignment
- - **Sync misaligned captions** -- fix timing drift in downloaded or auto-generated captions
- - **Align manual transcripts** -- match hand-written text to speech precisely
- - **Post-transcription alignment** -- improve coarse timestamps from ASR output
- - **Word-level timestamps** -- need per-word timing for karaoke, highlighting, or TTS
- - **Multi-format input** -- SRT, VTT, ASS, TXT, MD, JSON, TextGrid, TSV, and 20+ more
-
- ## When NOT to Use
-
- - No existing text/transcript -- use `/lai-transcribe` first to generate one
- - Only need format conversion -- use `/lai-caption` instead
- - Need speaker identification -- use `/lai-diarize` after alignment
- - Very short clips (<3 seconds) -- alignment needs sufficient audio context
-
- ## Prerequisites
-
- ```bash
- lai doctor # Verify environment
- lai auth whoami # Verify API key
- ```
-
- If not set up, run `/lai-setup` first.
-
- ## CLI Usage
+ Produces word-level and segment-level timestamps with sub-frame accuracy. Requires an API key — see `/lai-setup`.
- ### Basic Alignment
+ ## Basic Command
```bash
lai alignment align audio.wav caption.srt output.srt
+ # shortcut:
+ lai-align audio.wav caption.srt output.srt
```
- Shortcut form:
+ Format is inferred from the output file extension (30+ formats — see `/lai-caption`).
- ```bash
- lai-align audio.wav caption.srt output.srt
- ```
+ ## Common Options
- ### Word-Level Timestamps
+ Append as `key=value` pairs:
- ```bash
- lai alignment align audio.mp3 caption.srt output.json \
- caption.render.word_level=true
- ```
+ - `caption.input.split_sentence=true` — re-segment into natural sentences (wtpsplit). Useful for YouTube auto-captions that break mid-sentence. **Ask the user before enabling** — it rewrites segment boundaries and can hurt manually-crafted SRT/VTT
+ - `media.streaming_chunk_secs=300` — process audio >10 min with bounded memory (up to 20 h)
+ - `alignment.device=auto` — override auto device (cuda/mps/cpu)
- Word-level output is best in JSON format (preserves per-word timing data):
+ Full list: `lai alignment align --help`.
+ ## Word-Level JSON Output
+
```json
{
- "supervisions": [
- {
- "text": "Hello world",
- "start": 0.0,
- "end": 2.5,
- "alignment": [
- {"word": "Hello", "start": 0.0, "end": 0.5},
- {"word": "world", "start": 0.6, "end": 2.5}
- ]
- }
- ]
+ "supervisions": [{
+ "text": "Hello world",
+ "start": 0.0,
+ "end": 2.5,
+ "words": [
+ {"word": "Hello", "start": 0.0, "end": 0.5},
+ {"word": "world", "start": 0.6, "end": 2.5}
+ ]
+ }]
}
```
- ### Smart Sentence Segmentation
-
- For word-level or poorly segmented input (e.g., YouTube VTT):
-
- ```bash
- lai alignment align video.mp4 caption.vtt output.srt \
- caption.input.split_sentence=true
- ```
-
- Uses `wtpsplit` AI model to re-segment into natural sentences. Especially
- useful for YouTube auto-captions where segments break mid-sentence.
-
- ### Long Audio (Streaming Mode)
-
- For audio longer than 10 minutes, enable streaming to limit memory usage:
-
- ```bash
- lai alignment align long_podcast.mp3 transcript.srt output.srt \
- media.streaming_chunk_secs=300
- ```
-
- Supports audio up to 20 hours. Default chunk: 300 seconds (5 minutes).
-
- ## Parameters
-
- | Parameter | CLI Syntax | Default | Description |
- |-----------|-----------|---------|-------------|
- | Input media | positional arg 1 | required | Audio or video file path |
- | Input caption | positional arg 2 | required | Caption/transcript file path |
- | Output caption | positional arg 3 | required | Output file path (format from extension) |
- | Word-level | `caption.render.word_level=true` | `false` | Include per-word timestamps |
- | Split sentence | `caption.input.split_sentence=true` | `false` | AI sentence re-segmentation |
- | Streaming | `media.streaming_chunk_secs=N` | off | Chunk size for long audio (seconds) |
- | Include speaker | `caption.render.include_speaker_in_text=true` | `false` | Embed speaker labels in text |
- | Normalize text | `caption.input.normalize_text=true` | `false` | Normalize input text before alignment |
- | API key | `client.api_key=xxx` | from config | Override API key |
- | Device | `alignment.device=cuda` | auto | Force device (cuda/mps/cpu) |
-
- ## Supported Formats
-
- **Input** (auto-detected from extension/content):
-
- | Format | Extensions |
- |--------|-----------|
- | SubRip | `.srt` |
- | WebVTT | `.vtt` |
- | ASS/SSA | `.ass`, `.ssa` |
- | Plain text | `.txt` |
- | Markdown (Gemini) | `.md` |
- | JSON | `.json` |
- | TextGrid (Praat) | `.TextGrid` |
- | TSV/CSV | `.tsv`, `.csv` |
- | Audacity | `.aud` |
- | LRC (lyrics) | `.lrc` |
- | TTML | `.ttml`, `.xml` |
- | And 15+ more | See `lai alignment align --help` |
-
- **Output**: Same formats as input. Format is inferred from output file extension.
-
- **Recommended output format**: JSON (preserves word-level timing for downstream use).
- Convert to other formats later with `/lai-caption`.
-
- ## Workflow
-
- ### Step 1 -- Verify setup
-
- ```bash
- lai auth whoami
- ```
-
- If not authenticated, guide user through `/lai-setup`.
-
- ### Step 2 -- Identify inputs
-
- Determine:
- 1. **Audio/video file** -- the media source
- 2. **Caption/transcript file** -- the text to align
- 3. **Desired output format** -- SRT for playback, JSON for downstream processing
-
- ### Step 3 -- Decide options
-
- | Scenario | Recommended Options |
- |----------|-------------------|
- | YouTube auto-captions (word-level VTT) | `caption.input.split_sentence=true` |
- | Long podcast (>10 min) | `media.streaming_chunk_secs=300` |
- | Need per-word timing | `caption.render.word_level=true`, output as `.json` |
- | Multi-speaker content | Align first, then `/lai-diarize` |
- | Want ASS with styling | Output as `.json` first, then `/lai-caption` to convert |
-
- ### Step 4 -- Run alignment
-
- ```bash
- lai alignment align <audio> <caption> <output> [options]
- ```
-
- ### Step 5 -- Verify output
-
- Read the first few segments of the output to confirm timing looks correct.
- For JSON output, check that `start`/`end` values are reasonable.
-
- ## Output Contract
-
- The output preserves all input text content. Only timestamps change.
-
- - **Segment-level**: Each caption segment gets precise `start` and `end` timestamps
- - **Word-level** (when enabled): Each word within a segment gets its own `start`/`end`
- - **Speaker labels**: Preserved from input if present
- - **Non-speech events**: `[MUSIC]`, `[APPLAUSE]` etc. are preserved with timing
+ Fan out to other formats (karaoke ASS, TextGrid, SRT, …) via `/lai-caption`.
- ## Error Handling
+ ## Common Issues
- | Condition | Action |
- |-----------|--------|
- | `API KEY verification error` | Key invalid/expired. Run `lai auth whoami` to check, then `lai auth login` or `lai auth trial` |
- | `No segments found` | Input caption is empty or unreadable. Check file format |
- | `Audio format error` | Convert to WAV/MP3/M4A first: `ffmpeg -i input.xxx -ar 16000 output.wav` |
- | `CUDA out of memory` | Add `alignment.device=cpu` or reduce `media.streaming_chunk_secs` |
- | `Model download failed` | Check network. Model auto-downloads from HuggingFace on first run |
- | Timing looks wrong | Try `caption.input.normalize_text=true` or `caption.input.split_sentence=true` |
+ | Problem | Fix |
+ |---------|-----|
+ | `API KEY verification error` | Run `lai auth trial` or `lai auth login` — see `/lai-setup` |
+ | Timing looks wrong | Try `caption.input.split_sentence=true` or `caption.input.normalize_text=true` |
## Related Skills
- | Skill | Use When |
- |-------|----------|
- | `/lai-setup` | Need to install or authenticate first |
- | `/lai-transcribe` | No transcript exists yet -- generate one |
- | `/lai-diarize` | Need speaker labels after alignment |
- | `/lai-caption` | Convert aligned output to another format |
- | `/lai-youtube` | Working with YouTube videos (download + align combined) |
- | `/lai-translate` | Translate aligned captions to another language |
-
- ### Common Workflow Chains
-
- ```bash
- # Transcribe then align (most common)
- lai transcribe run video.mp4 transcript.json
- lai alignment align video.mp4 transcript.json aligned.json
-
- # Align then convert to multiple formats
- lai alignment align audio.mp3 caption.srt aligned.json caption.render.word_level=true
- laicap-convert aligned.json aligned.srt
- laicap-convert aligned.json aligned.vtt
- laicap-convert aligned.json aligned.ass
-
- # Align then diarize
- lai alignment align podcast.mp3 transcript.srt aligned.json
- lai diarize run podcast.mp3 aligned.json diarized.json
- ```
-
- ## Non-Goals
-
- - Does NOT generate transcripts -- use `/lai-transcribe` for that
- - Does NOT identify speakers -- use `/lai-diarize` after alignment
- - Does NOT translate text -- use `/lai-translate` for translation
- - Does NOT download media -- use `/lai-youtube` or `/lai-caption` for downloads
- - Does NOT modify caption text content -- only timestamps change
+ - `/lai-transcribe` — generate a transcript first
+ - `/lai-diarize` — add speaker labels to aligned output
+ - `/lai-caption` — convert aligned JSON to any format
+ - `/lai-youtube` — YouTube URL → aligned captions in one command