v1.0.0 to v1.0.1

---

name: whisper-preprocess

metadata:

---

# Whisper Preprocess Pipeline

You are setting up and running an audio preprocessing + transcription pipeline. This pipeline solves four critical Whisper problems:

1. **Hallucinations during silence** — Whisper invents text like "Obrigado", "E ai", "Subscribe" during quiet segments

2. **Hallucination cascade in long audio** — after the first hallucination, Whisper repeats it for the rest of the file

3. **Poor transcription from noisy audio** — background noise reduces accuracy

4. **Silent failures on Windows due to encoding** — the `whisper` CLI crashes when printing accented/CJK characters in a cp1252 console, producing empty `.txt` files with no clear error

The pipeline runs **100% offline** when the Whisper model is already cached locally — only needs ffmpeg, Python, and the downloaded model.

## Critical lessons learned (from real-world testing)

These are hard-won insights from iterating on a 2h13min lecture recording and a multilingual (PT+JA) meeting. Follow them carefully:

### 1. NEVER use `afftdn` (FFT noise reduction)

The `afftdn` filter creates **tonal artifacts** ("musical noise") that sound like constant beeping/whistling. This completely destroys the audio for Whisper — every segment will transcribe as garbage. The same applies to `lowpass=f=8000` which can interact badly with other filters.

**Safe enhancement chain:** `highpass=f=100, acompressor, loudnorm` — nothing else.

### 2. Remove silence BEFORE enhancement

The `loudnorm` filter normalizes everything to -16 LUFS, including silence. If you run `silenceremove` after `loudnorm`, it finds nothing to remove because the "silence" has been boosted. Always: extract → remove silence → enhance.

### 3. Silence threshold should be -30dB, not -40dB

Most recordings have background noise above -40dB. At -40dB threshold, `silenceremove` finds zero silence. Use -30dB as default. For noisier recordings, try -25dB.

### 4. Segment long audio (mandatory for >10 minutes)

Whisper hallucinates on long audio files. A 2h13min file produced only ~19 lines of real content before devolving into repeated "Obrigado" or "E ai" for the remaining 2 hours. Segmenting into 10-minute chunks solved this completely — each segment produced ~6-7K characters of accurate transcription.

### 5. Use `--condition_on_previous_text False`

Without this, once Whisper starts hallucinating, it conditions on its own hallucinated output and the error cascades. With `False`, each segment starts fresh. This doesn't prevent hallucinations entirely (segmentation does that), but limits the damage.

### 6. Compress output audio to OPUS (listening copy, decoupled — see lesson #9)

Enhanced WAV files are enormous (2.6 GB for 2h). The listenable `*_enhanced.opus` is now

encoded at **48 kHz, 64 kbps, `-application audio`** (wideband, fuller voice). The old

`-application voip` is narrowband telephony tuning and was dropped. Bitrate/sample-rate are

configurable via `--listen-bitrate` / `--listen-sr`.

### 9. NEVER feed the listening copy through silence-removal + dynamic AGC (anti-"picotamento")

**Symptom:** the voice in `*_enhanced.opus` is choppy / cuts between syllables ("picotando"),

worst for a low-volume or impaired (dysarthric, slow) speaker.

**Cause:** the listenable file used to inherit the whole transcription chain — `silenceremove`

(cuts quiet speech) **+** `acompressor release=50ms` (pumps every syllable gap) **+**

single-pass `loudnorm` (dynamic AGC that "breathes" on pauses). Three gain-modulation/chopping

sources stacked.

**Fix (now built in):** the listening copy is **decoupled** from the transcription path and

generated by `build_listenable()` from the **original** file:

- wideband (48 kHz), **no silence removal** by default → continuous audio, the listener follows

every pause and nothing is clipped;

- **stable gain only** — slow-release compressor (`release=400ms`) + fixed makeup + true-peak

`alimiter` with lookahead. **No dynamic AGC.** Verified A/B on a real recording: a 2-pass

`loudnorm linear=true` *reverted to "Dynamic"* on clipping sources, so it is NOT used here.

- limiter ceiling has headroom (`--listen-limit 0.6`) for Opus inter-sample overshoot → final

true-peak stays below 0 dBFS (the old recipe actually clipped at +0.7 dBFS).

The **transcription path** is unchanged in spirit (16 kHz, `silenceremove` at -30dB, then

enhance) — only made gentler: `detection=rms`, `window=0.025`, `stop_silence=0.5`,

`stop_duration=2.0`, which also helps Whisper on slow speakers. Lessons #1 (no `afftdn`) and #3

(-30dB) still hold; denoise on the listening copy is **opt-in and off by default** (`--denoise`).

### 7. UTF-8 everywhere on Windows (handled automatically)

On Windows the default console encoding is cp1252. When `whisper` CLI prints transcription progress containing accented or CJK characters it crashes with `UnicodeEncodeError`, and the Python `subprocess.run` reader thread separately crashes with `UnicodeDecodeError` on bytes > 0x7F. Result: all segments come back empty with the misleading log line "Segmento vazio ou falhou". The pipeline now sets `PYTHONIOENCODING=utf-8`, reconfigures `sys.stdout/stderr`, and forces `encoding="utf-8", errors="replace"` on every `subprocess.run` — no user action needed. If you ever invoke `whisper` standalone on Windows, set `$env:PYTHONIOENCODING="utf-8"` first.

### 8. For multilingual recordings, run two passes and merge

A single `--language` choice loses everything spoken in another script. Run a primary pass in the dominant language, a secondary pass in the second language, and merge the SRTs choosing the secondary version where the script ratio is high. See "Multilingual recordings" below — the script does this for you with `--secondary-language`.

## The Pipeline (6 stages)

```

Input (MKV/MP4/etc.)

|

v

[1] Extract audio → WAV 16kHz mono

|

v

[2] Remove silence (transcription path, on RAW 16kHz, before enhancement)

| silenceremove threshold=-30dB, duration=2.0s, padding=0.5s, detection=rms

v

[3] Enhance audio (transcription path)

| highpass=100Hz → acompressor → loudnorm EBU R128

| NO afftdn, NO lowpass (they cause artifacts)

v

[4] Segment into ~10min chunks → cached by input hash

|

v

[5] Transcribe each segment with Whisper

| temperature=0, condition_on_previous_text=False

| optional 2nd pass in --secondary-language + merge

|

| (PARALLEL, decoupled) build_listenable() from the ORIGINAL file →

| 48kHz, NO silence removal, stable gain (slow compressor + true-peak limiter,

| no dynamic AGC) → continuous, no "picotamento" — see lesson #9

v

[6] Compress listening copy to OPUS (48k, application=audio) + cleanup

|

v

Output: *_transcricao_<lang>.txt/.srt, *_transcricao_merged.txt, *_enhanced.opus

```

## Prerequisites

```bash

ffmpeg -version # audio processing

whisper --help # transcription (pip install -U openai-whisper)

python --version # 3.9+

```

Check required ffmpeg filters:

```bash

ffmpeg -filters 2>/dev/null | grep -E "silenceremove|loudnorm|highpass|acompressor"

```

## How to use

Copy `scripts/whisper_preprocess.py` to the user's working directory, then:

```bash

# Basic — full pipeline with auto-cleanup

python whisper_preprocess.py "lecture.mkv"

# With context for proper nouns and technical terms

python whisper_preprocess.py "lecture.mkv" --initial-prompt "Seminario de Teologia, professor Carlos"

# Test preprocessing only (listen before transcribing)

python whisper_preprocess.py "lecture.mkv" --only-preprocess

# Keep intermediate files for debugging / segment reuse

python whisper_preprocess.py "lecture.mkv" --keep-intermediate

# CPU only (no GPU)

python whisper_preprocess.py "lecture.mkv" --device cpu

# Better quality model (needs ~10GB VRAM)

python whisper_preprocess.py "lecture.mkv" --model large-v2

# Noisier recording — more aggressive silence detection

python whisper_preprocess.py "lecture.mkv" --silence-threshold="-25dB"

# Quiet / impaired (slow, low-volume) speaker — louder, fuller listening copy

python whisper_preprocess.py "call.m4a" --listen-makeup 7 --listen-comp-threshold="-24dB"

# Longer segments for very clean audio

python whisper_preprocess.py "lecture.mkv" --segment-minutes 15

# Multi-idioma (ex.: reuniao PT + JA com merge automatico)

python whisper_preprocess.py "meeting.mkv" \

--language Portuguese --secondary-language Japanese \

--initial-prompt "Reuniao tecnica sobre UPS, participantes Kusaba, Murata, Heitor" \

--secondary-initial-prompt "Kusaba Murata UPS battery substation"

# Modo offline (modelo ja baixado, sem internet)

python whisper_preprocess.py "lecture.mkv" --offline

# Forcar reprocessamento ignorando cache de segmentos

python whisper_preprocess.py "lecture.mkv" --force-reprocess

```

**Important:** when passing negative thresholds, use `=` syntax: `--silence-threshold="-30dB"` (not `--silence-threshold "-30dB"`, which argparse interprets as a flag).

## Multilingual recordings

Use `--secondary-language` when the audio mixes two languages (e.g., a Portuguese meeting with Japanese speakers, or English narration with Spanish quotes). The script will:

1. Preprocess once

2. Transcribe all segments in `--language` (primary)

3. Transcribe the **same segments** in `--secondary-language`

4. Merge the two SRTs into a unified `*_transcricao_merged.txt`. The merge prefers the primary version, but for cues where the secondary script (kanji/hiragana, cyrillic, etc.) appears prominently in both passes (sign of foreign speech mistranscribed phonetically), it substitutes the secondary version prefixed with a tag like `[JA]`.

Outputs in this mode:

- `*_transcricao_pt.txt` / `.srt` — primary pass

- `*_transcricao_ja.txt` / `.srt` — secondary pass

- `*_transcricao_merged.txt` — unified, with `[JA]` markers

The merge logic also lives in `scripts/merge_passes.py` and can be called standalone:

```bash

python merge_passes.py --primary meeting_pt.srt --secondary meeting_ja.srt \

--output merged.txt --secondary-script japanese --marker-tag JA

```

Supported scripts: `japanese`, `chinese`, `korean`, `cyrillic`, `arabic`, `greek`, `hebrew`, `thai`. For anything else pass a regex via `--secondary-script-regex`.

## Offline use

When `--offline` is passed (or env var `WHISPER_OFFLINE=1` is set), the script:

- Sets `HF_HUB_OFFLINE=1`, `TRANSFORMERS_OFFLINE=1` so Whisper won't touch the network

- Pre-validates that the model is in local cache; fails fast with a clear path if it isn't

Whisper caches models in:

- `~/.cache/whisper/<model>.pt` (classic models: tiny, base, small, medium, large-v1, large-v2, large-v3)

- `~/.cache/huggingface/hub/models--openai--whisper-large-v3-turbo/snapshots/` (turbo)

To pre-download a model while online:

```bash

python -c "import whisper; whisper.load_model('turbo')"

# or

python -c "import whisper; whisper.load_model('large-v2')"

```

You can also pass `--model-dir DIR` to use a portable copy of the models.

## Segment cache (reuse across runs)

Intermediate processing is cached per-input. The script computes a short hash from the input's path + size + mtime and uses it to name the intermediate directory (e.g., `meeting_intermediate_a1b2c3d4/`). On subsequent runs with the same input (e.g., changing only `--secondary-initial-prompt`), the extract → silence → enhance → segment pipeline is skipped and the existing segments are reused.

To force a clean reprocess, pass `--force-reprocess`.

Combined with `--keep-intermediate`, this lets you iterate cheaply on prompts or experiment with alternate secondary languages without redoing the heavy ffmpeg work.

## All parameters

| Parameter | Default | Description |

|-----------|---------|-------------|

| `input` | (required) | Input file (MKV, MP4, WAV, M4A, etc.) |

| `--language` | Portuguese | Primary transcription language |

| `--secondary-language` | (none) | Secondary language for 2nd pass + auto merge |

| `--model` | turbo | Whisper model: tiny, base, small, medium, large-v2, turbo |

| `--model-dir` | (none) | Pass-through to whisper's `--model_dir` |

| `--device` | cuda | Device: cuda (GPU) or cpu |

| `--initial-prompt` | (none) | Context for primary pass (names, acronyms) |

| `--secondary-initial-prompt` | (none) | Context for secondary pass |

| `--silence-duration` | 2.0 | Minimum silence to remove (seconds) |

| `--silence-threshold` | -30dB | Volume threshold for silence detection |

| `--silence-keep` | 0.5 | Silence kept at each cut (stop_silence, seconds) |

| `--segment-minutes` | 10 | Duration of each transcription segment |

| `--listen-sr` | 48000 | Sample rate of the listening `*_enhanced.opus` (48000/24000/16000) |

| `--listen-bitrate` | 64k | Opus bitrate of the listening copy |

| `--listen-makeup` | 5.0 | Listening compressor makeup gain (dB) — raise for a very quiet speaker |

| `--listen-limit` | 0.6 | Listening limiter ceiling 0..1 (lower = more Opus headroom) |

| `--listen-highpass` | 80 | High-pass (Hz) for the listening copy |

| `--listen-comp-threshold` | -22dB | Listening compressor threshold |

| `--listen-comp-ratio` | 3.0 | Listening compressor ratio |

| `--trim-listen` / `--no-trim-listen` | off | Also (gently) remove silence from the listening copy |

| `--denoise` / `--no-denoise` | off | Denoise listening copy (afftdn, or arnndn with `--denoise-model`) |

| `--denoise-strength` | 12 | afftdn noise reduction in dB (6..24) |

| `--denoise-model` | (none) | Path to a `.rnnn` model → uses arnndn instead of afftdn |

| `--skip-enhance` | false | Skip audio enhancement |

| `--skip-silence` | false | Skip silence removal |

| `--keep-intermediate` | false | Keep intermediate audio files |

| `--only-preprocess` | false | Only preprocess, don't transcribe |

| `--offline` | false | Force local model; fail if not cached (also via env `WHISPER_OFFLINE=1`) |

| `--force-reprocess` | false | Ignore segment cache and reprocess from scratch |

## Output files

| File | Description |

|------|-------------|

| `*_transcricao_<lang>.txt` | Plain text transcription (primary pass) |

| `*_transcricao_<lang>.srt` | SubRip subtitles for primary pass |

| `*_transcricao_<lang2>.txt` / `.srt` | Secondary pass (only with `--secondary-language`) |

| `*_transcricao_merged.txt` | Merged transcript with `[TAG]` markers (only with `--secondary-language`) |

| `*_enhanced.opus` | Compressed enhanced audio (~54 MB for 2h) |

Intermediate files are cached under `<input>_intermediate_<hash>/` and removed at the end unless `--keep-intermediate` is used (or until `--force-reprocess` discards the cache).

## Whisper model recommendations

| Model | VRAM | Speed | Quality | Notes |

|-------|------|-------|---------|-------|

| turbo | ~6GB | Fast | Good | Recommended default |

| large-v2 | ~10GB | Slow | Best | Fewer hallucinations than v3 |

| medium | ~5GB | Medium | OK | Fallback if VRAM limited |

| small | ~2GB | Fast | Basic | Quick drafts only |

Prefer `large-v2` over `large-v3` — v3 has more hallucination issues for non-English languages.

## Troubleshooting

| Problem | Cause | Fix |

|---------|-------|-----|

| "Obrigado" / "E ai" repeated | Hallucination in silence | Reduce `--segment-minutes` to 5 |

| 0% silence removed | Threshold too low | Use `--silence-threshold="-25dB"` |

| Beeping/whistling in audio | `afftdn` filter | Already removed from default pipeline (denoise is opt-in) |

| Voice choppy / cutting between syllables ("picotando") in `*_enhanced.opus` | Old: silence-removal + fast compressor + dynamic loudnorm AGC stacked on the listening copy | Fixed (lesson #9): listening copy decoupled, stable gain, no silence cut. For a very quiet speaker raise `--listen-makeup` (e.g. 7) |

| Listening copy still too quiet | Low-volume speaker, default makeup | Increase `--listen-makeup` (6–9) and/or lower `--listen-comp-threshold` |

| Argparse error with threshold | `-30dB` parsed as flag | Use `--silence-threshold="-30dB"` |

| Huge intermediate files | WAV format | Default behavior now auto-cleans |

| "Segmento vazio" with no real error | Old issue (Windows cp1252) | Fixed — script now logs whisper's stderr when exit≠0 or output is empty |

| "Modelo nao encontrado" in `--offline` | Model not pre-cached | Run `python -c "import whisper; whisper.load_model('turbo')"` once online |

| Mixed-language audio losing JA/RU/AR speech | Single-pass `--language` only sees one script | Add `--secondary-language` for an auto 2-pass + merge |