song-analysis git:20260925.4486774A

Immutable. This exact content never changes and is served at /api/v1/blob/f2859d9c9a2d502a.

---
name: song-analysis
description: Use when analyzing a song recording — extracting tempo/downbeats/key, separating stems, transcribing bass to MIDI, identifying chords per bar, timing lyrics, reading drum patterns, or producing a chord+lyric chart for a band. Covers madmom, demucs (htdemucs_ft), CREPE, basic-pitch, chroma chord proposal, and Whisper word timing.
---

# Song Analysis — recording → bars, chords, lyrics, chart

Song-agnostic pipeline, validated over many real projects (dream pop, Greek
laiko, punk reinterpretations). Works for any song, genre, or key.

**References (read on demand):**
- `references/environment-setup.md` — first-time venv install + per-song folder layout
- `references/chart-and-lyrics.md` — chart structure, the 6 lyric-placement rules, HTML output
- `references/drum-pattern-analysis.md` — kick/snare pattern reading from the drums stem
- `references/modal-theory.md` — power-chord shorthand, mode tables, Byzantine ≠ Phrygian

## Inputs

1. **Audio file** — mp3/wav of the song (full mix).
2. **Lyrics** — canonical text from an official source. Section labels (Verse, Chorus,
   Bridge…) come from this, not from the analyzer.

## Tool per step

| Step | Tool | Why |
|---|---|---|
| Pulse (beats only) | **madmom** `DBNBeatTrackingProcessor` | No meter assumed — see Phase 1a |
| Meter / bar length | **`scripts/detect_meter.py`** | Sweeps every cycle 2–25; finds odd meters |
| Key (melodic mode) | **librosa** chroma + Krumhansl | Signal only — see Trap 2 |
| Stem separation | **demucs `htdemucs_ft`** (4-stem) | Cleanest bass; NEVER `htdemucs_6s` (bleeds piano/guitar into bass) |
| Bass → MIDI | **librosa pyin** (40–220 Hz) | Monophonic tracker; gives the ROOT, not quality. Not CREPE — see Phase 3 |
| Polyphonic parts → MIDI | **basic-pitch** | Piano/guitar/pads, double-stops |
| Chords | **chroma + bass-root** (+ slash relax) | See Phase 5b |
| Lyrics + word timing | **mlx-whisper `large-v3-turbo`** (Apple Silicon) | Word timestamps, hallucination-filterable |
| Drum-pattern read | **librosa** band-limited onsets | see reference |
| Chart output | Python → self-contained **HTML** | see reference |

## Phase 1 — Foundation (pulse + meter + key)

### 1a. Track the pulse WITHOUT assuming a meter

```python
from madmom.features.beats import RNNBeatProcessor, DBNBeatTrackingProcessor
import librosa, numpy as np

act  = RNNBeatProcessor()(audio_path)
beat_times = DBNBeatTrackingProcessor(fps=100)(act)   # beats only, NO bar assumption
beat = float(np.median(np.diff(beat_times)))
pulse_bpm = 60.0 / beat
# Key via librosa chroma_cqt mean + Krumhansl correlation → melodic mode ONLY (Trap 2)
```

**Never pass `beats_per_bar=(3,4)`.** A constrained candidate list cannot return "none of
these" — it returns the least-bad of the options you supplied, with full confidence, and
every downstream number (bar counts, per-bar chord cells, drum histograms) silently
inherits the error. Get the pulse first; derive the meter in 1b.

### 1b. Derive the meter by sweeping every cycle length

```bash
<venv>/bin/python scripts/detect_meter.py <song.mp3> --drums stems/<song>/drums.wav \
    --bass stems/<song>/bass.wav --other stems/<song>/other.wav \
    --foundation analysis/foundation.json
```

**Always pass `--bass` and `--other` too.** Plenty of songs have no drum kit, or a
percussion part that is an undifferentiated pulse carrying no accent — and the cycle is
still there, articulated by the bass and the harmony instead. The signature is a drum
contrast stuck near 1.2 while the bass reaches 3–4 and resolves the cycle cleanly. The
script falls back to these stems automatically when no two drum bands agree.

Phase-folds band-limited kick / snare / hat onsets onto every cycle length from 2 to 25
pulses and ranks by **accent contrast** = loudest position in the cycle ÷ quietest.
Run it AFTER Phase 2 so it has a drums stem; a full mix works but reads muddier.

Reading the output:

| Output | Means | Do |
|---|---|---|
| Consensus, confidence HIGH | Real cycle, clear of unrelated periods | Use it. Odd winner → odd meter |
| Consensus on 8, 16, 24 (script prints a PHRASE note) | Usually plain 4/4 with an *n*-bar pattern | Report 4/4 + phrase length, not an *n*-beat bar |
| Consensus on 2 | Duple; the bar-level accent is too faint to fix 2/4 vs 4/4 | Report duple; ask the user to count |
| INCONCLUSIVE | Bands disagree, no usable accent, or no band clear of unrelated periods (margin < 1.3) | Report the ambiguity. Do NOT pick one |
| Winner is prime (7, 11, 13) | Cannot be a phrase of anything smaller | Almost certainly the meter |
| Winner = 2× a strong period | The half is the bar, the double is two bars | Take the fundamental |

**The sweep can only find cycles on the pulse grid you hand it.** If the tempogram peaks
at twice the tracked pulse, the beat tracker is reading half-time: re-run on the doubled
grid, because a cycle of 8 there shows up as 4 here. Check the tempo octave *before*
trusting any cycle length.

Needs librosa, plus madmom unless `--foundation` supplies the beat grid. `--json` writes
every band's sweep, per-band verdicts (cycle, margin, accented positions) and the
consensus. Smoke test: `<venv>/bin/python tests/test_detect_meter.py` (synthetic 4/4,
3/4, 6/8, 5/4, 7/8, 11/8 and flat clicks; ~8 s).

**A song with no percussion at all cannot be metered this way** — say so and move on
rather than straining the harmonic stem for an answer it does not contain.

**The two strongest positions in an odd cycle mark its internal split.** An 11 with peaks
at 1 and 7 is 6+5; peaks at 1 and 5 would be 4+7.

Save as `analysis/foundation.json`: `pulse_bpm`, `beat_times`, `beats_per_bar` (from 1b,
**not** from madmom), `pulse_unit` (4 if the pulse is a quarter note, 8 if an eighth),
`grouping` (the internal split, e.g. `[6, 5]`; `[2, 2]` for 4/4), `downbeat_times` (every
*n*-th beat from the winning phase), `num_bars`, `key`.

**Cells.** Everything downstream (bass, chords, lyrics, chart) is keyed by `(bar, cell)`,
one cell per group in `grouping`: 4/4 → 2+2 (the familiar half-bars), 3/4 → one cell,
6/8 → 3+3, 11/8 as 6+5 → two unequal cells. Never split an odd bar at its midpoint — that
lands mid-pulse.

**Sanity checks:** BPM stable (no jumps); watch half/double-time on slow songs (60–90
BPM) — count along the audio; cross-check `num_bars` against
`duration ÷ (beats_per_bar × beat)`. A large mismatch means rubato, a wrong tempo octave,
or a wrong meter.

## Phase 2 — Stems

```bash
demucs -n htdemucs_ft -d mps -o stems <song.mp3>
# → stems/htdemucs_ft/<song>/{bass,drums,vocals,other}.wav
```

**Always pass `-d mps` on Apple Silicon.** demucs 4.0.1 defaults to CUDA-else-CPU, so
without the flag it runs on the CPU. Measured on an M3 (30 s clip): CPU 64.5 s, MPS 23.0 s,
stems within noise of each other. Meter-only runs need just the drums:
`--two-stems=drums`.

**CHECK THE CACHE FIRST — `du -sh ~/.cache/torch/hub/checkpoints`** (demucs 4.0.1; 4.1.x
loads from the Hugging Face cache instead). `htdemucs_ft` is a bag of 4 models (4 × 84
MB). If it isn't cached, that command silently downloads for as long as your connection
takes — measured at ~3 MB/min (≈100 min) on 2026-08-06 — and shows no progress at all if
you pipe it through `tail`. Don't assume it's cached because you used it
on a previous song.

**Fast offline fallback (Apple Silicon):** plain htdemucs via MLX, weights already local —
28 s for a 4-minute track (8.7× realtime):

```bash
<audio-analysis>/mlx-demucs/.venv/bin/mlx-demucs <song.mp3> -o stems_mlx -v
```

Call the venv binary **directly**. `uv run mlx-demucs` re-resolves dependencies over the
network and hangs 10+ min even though the venv is already complete. The MLX port is within
0.03% of the PyTorch htdemucs reference, so it's a sound basis for chords and bass; re-run
on `_ft` later only if quality looks marginal. Note its README lists `htdemucs_ft` as a
supported model, but only plain htdemucs weights ship in `weights/`.

**Quality check:** listen to `bass.wav` alone — piano/guitar bleed means separation
struggled; try plain `htdemucs` as fallback.

## Phase 3 — Bass → MIDI

Default: **pyin** on `bass.wav` (monophonic → root line). For the pyin→MIDI code and
cleanup rules, use the `bass-transcribe` skill — same pipeline. Use **basic-pitch** only
if the part is polyphonic. CREPE is for vocals, not bass (reasons in `bass-transcribe`).

Then aggregate per `(bar, cell)`: for each cell, total each pitch class's overlapping
note duration and rank → `analysis/bass_per_cell.json`.

**Spell pitch classes from the key signature, not from a fixed table.** Flat spelling
(C, D♭, D, E♭ … B♭, B) is the right default for flat and neutral keys — it matches how
guitarists read pop/indie charts. But applying it blindly in a **sharp key produces
nonsense**: in F♯ minor (three sharps) the flat table renders the tonic as `G♭m`
instead of `F♯m`, and the dominant as `D♭7` instead of `C♯7`. Rule: if the
key signature has sharps, use sharp spelling (C♯, D♯, F♯, G♯, A♯); if flats or none, use
flats.

**Pitfalls:**
- Sub-0.5 s detections are noisy; ≥0.7 s sustained notes are usually right.
- Bass doesn't always play roots — it pedals, walks, sits on 3rds/5ths. Bass note is a
  *clue*; annotate in chart only when bass ≠ chord root.
- **Bass enters late** in most songs. Find the bass-entry bar (first clearly audible bar)
  and suppress all bass-derived annotations before it — pre-entry content is stem bleed.

## Phase 4 — Lyric timing (Whisper on the VOCALS STEM)

Whisper is far more accurate on isolated vocals than the full mix.

Settings that work: model `mlx-community/whisper-large-v3-turbo` (Apple Silicon; the
non-turbo `large-v3` is a separate ~3 GB download for no gain on lyrics);
`word_timestamps=True`; `condition_on_previous_text=False` (avoids hallucination drift);
post-filter known hallucinations ("Thank you.", "Blah Blah", ".") and 3+ identical repeats.

`no_speech_threshold=0.8` suppresses hallucinated text in silence **but also drops real,
sparse vocals** — it can return nothing for a track that has singing. If the result looks
empty or thin, re-run at `0.45` and diff: what only the second pass finds is either sparse
real text or hallucination, and the vocal stem's RMS in that span tells you which.

Map each word's start time to (bar, beat):

```python
def locate(t, downbeats, beats_per_bar):     # from foundation.json — no default, never assume 4
    for i in range(len(downbeats)-1):
        if downbeats[i] <= t < downbeats[i+1]:
            return (i+1, (t - downbeats[i]) / (downbeats[i+1] - downbeats[i]) * beats_per_bar)
    return None
```

## Phase 5 — Sections (manual, from lyrics + audio)

For each canonical lyric line, find its first bar via Whisper + `locate()`. Section
boundaries = where each labeled section's first line lands. **Don't trust automatic
section detection** (librosa recurrence segmentation is ±1 bar off around bridges/outros);
the lyrics + word timing are far more reliable.

## Phase 5b — Chord proposal (chroma + bass-root)

Per cell, on the `other.wav` stem (everything harmonic except bass/drums/vocals).

1. **Chroma**: average `librosa.feature.chroma_cqt(other.wav)` over the cell's frames.
2. **Score all 24 triad templates** (12 maj + 12 min, binary root/3rd/5th, sum-normalized)
   by dot product with the cell chroma.
3. **Bass-root constraint** (only when `bar >= BASS_ENTRY_BAR`): if one bass pitch class
   has ≥0.5 s sustain in the cell, restrict candidates to the maj+min triads on that root —
   **unless step 4 relaxes it**.
4. **Slash-chord relaxation (REQUIRED — +17.7 points).** The constraint assumes the bass
   note *is* the root. On a slash chord the bass plays the 3rd or 5th, and constraining to
   it forces a guaranteed-wrong answer:

   ```python
   allc = sorted(all 24 templates by score, desc)
   con  = [c for c in allc if root_of(c) == bass_pc]
   use_constrained = con and con[0].score >= RELAX_FACTOR * allc[0].score   # 0.85
   ```

   If the best chord rooted on the bass note scores materially worse than the
   unconstrained best, the bass is a non-root chord tone — drop the constraint and let
   chroma decide. `RELAX_FACTOR = 0.85`; anything ≥0.85 behaves identically, 0.0 = off.
   Measured on a minor-tonic track built around a dominant-7th slash chord (C♯7/F, bass
   on the 3rd, E♯): without it every C♯7 cell became `Fm`, the minor triad on the bass
   note (14 misses, a third of all errors). With it: **73.8% → 91.5%** root+quality.
5. **Emit top-3** per cell + `beat3_re_attack` flag from onset detection on the same stem
   (corroborating evidence only — noisy).
6. **Major-bias diff pass.** Re-run with `MAJOR_BIAS_MARGIN = 0.05` (if the top template is
   minor and the same-root major is within the margin, swap) and **report the flip
   count**. Never adopt the biased result silently.

### `MAJOR_BIAS_MARGIN`: default 0, never gated on the key

It is the single highest-leverage parameter. Same minor-tonic track (F♯m tonic), same
audio, only this changed:

| `MAJOR_BIAS_MARGIN` | root+quality |
|---|---|
| 0.05 | **27.4%** |
| 0.02 | 54.9% |
| 0 | **73.8%** |

A 46-point swing: at 0.05 the tonic minor flips to major on nearly every cell. The 0.05
bias was tuned on modal-major dream pop, where it reached 99% top-1 / 100% top-3 — that
figure does not transfer to other material.

**Never gate the bias on the detected key.** The key estimator reports the *melody's*
mode, so a modal song pedalling on a major triad returns "major" — which switches on the
very bias that destroys its minor chords. Measured: gating on a major key estimate flipped
the majority of cells in a track (70%+), turning a minor loop into a phantom major one.

A large flip count means the cells are near-ties and the quality call is genuinely
uncertain — say so rather than picking silently. To settle major vs minor on one chord,
compare chroma energy of the major third against the minor third directly: G♯ at 14.4%
vs G♮ at 4.7% closes the question in a single number.

Why the combination works: bass MIDI gives the root but not the quality; chroma gives the
quality but confuses 4th/5th-related roots. Errors concentrate in pre-bass-entry bars,
sustained-organ smear, and extended chords (expand templates with 7th/sus for jazz/R&B).

Per-song tuning: `BASS_ENTRY_BAR`, template set.

## Phases 6–9 — Chart

Read `references/chart-and-lyrics.md` before building the chart. Core invariants:
- One row per section; parallel sections get identical row splits.
- Chart bar = audio bar. A virtual bar only where the audio has none (e.g. a rubato hold
  the beat tracker skipped) — and mark it on the chart.
- Chord dict keyed by `(bar, cell)` (cells from `grouping`, Phase 1).
- **Lyric phrases anchor at the chord they resolve INTO** (Rule 1 — the big one).
- Output: single self-contained HTML, print-friendly, harmonic-notes block at bottom.

## Hard rules — the cardinal traps

### Trap 1 — A bass walk is one chord or two; the upper voicing decides

Bass moving within a bar (`C → A♭`) has two readings, and both are common: (1) one held
chord over a moving bass (`Cm/A♭`, or a drone with a walking bass) or (2) **two chords**
(`Cm → A♭`). The bass alone cannot tell them apart — and a bass-constrained matcher
simply follows the bass, so it reports (2) either way.

**Discriminating test — look above the bass:**
- **Re-attack**: a new voicing struck where the bass moves (in `other.wav`) = two chords.
- **Register-pooled chroma**: pool chroma per cell in the chord instrument's own register
  (high-pass above ~165 Hz). If the pitch-class set changes with the bass = two chords;
  if it stays put = one chord over a moving bass.

Bass movement alone is a suspicion signal, never the decision.

### Trap 2 — "Minor key" ≠ minor chords

Key estimators report the **melody's mode** (the PCP is melody-dominant), not chord
quality. Many songs put **major triads under a minor melody** — that's the bittersweet
sound. Default each chord to major, then verify: play root+maj3+5 against the melody; if
the maj-3rd fights the melody's ♭3, go minor. Never propagate the key label to chord
quality. Also: many modal songs pivot between relative major and minor — label the key as
a pair (`E♭ / Cm`) when both centers appear.

### Trap 3 — The meter you never tested for looks like no meter at all

Folding onsets onto the wrong period does not produce a *low* score, it produces a
**flat** one. An 11-beat cycle folded onto 2, 3, 4 or 6 smears to near-perfect uniformity,
because 11 shares no factor with any of them.

Calibration — read your own contrast numbers against this:

| Top contrast in the sweep | Reading |
|---|---|
| 1.0–1.3 at *every* period tested | Wrong periods tested, or genuinely no accent. Widen the sweep |
| 1.5–2.5 | Weak or ambiguous — report it as such |
| 3–10 at one period, ≤2.5 at every unrelated one | A real cycle. Trust it |

A coprime cycle read through the wrong fold lands squarely in the first row: uniform, not
low.

**Uniform contrast at every period you tested is not a finding. It means you have not
tested the right period yet.** Widen the sweep before concluding anything, and never
report "no time signature" on the strength of a narrow sweep.

Corollary: assume odd meters are live. 5/4, 7/8 and 11/8 are ordinary in Balkan,
Byzantine, prog and much devotional music — exactly the material this pipeline gets used
on. Coprime blindness makes them invisible to a 4/4-shaped test.

**Verify with a player.** All automated signals are wrong ~30% of the time on
modal-mixture songs. A musician who knows the song is ground truth — plan a verification
pass before declaring the chart done.

## Lessons learned

1. `htdemucs_ft`, never `htdemucs_6s` — 6-stem bleeds piano/guitar into bass MIDI.
2. Trust the user's ear over any analyzer disagreement.
3. Chart bar = audio bar.
4. Whisper timing is precise; musical placement is not strict timing (Rule 1).
5. A mid-bar bass walk may or may not change the chord (Trap 1) → key by `(bar, cell)`
   so either reading can be written down.
6. Chord-recognition tools are starting points, never ground truth.
7. Each chart correction touches ~3 places (chord dict, row layout, cascading lyrics).
8. Mirror parallel sections visually.
9. Section-boundary pickups: overlay in the same cell, don't displace.
10. Never hand an analyzer a candidate list it can't say "none of these" to — a
    constrained `beats_per_bar` and a key-gated `MAJOR_BIAS_MARGIN` both produced
    confident wrong answers that survived every downstream check.
11. Ask the user to count. A player's count settles in seconds what a sweep can only
    rank, and it outranks the sweep.
12. A static upper voicing over a moving bass reads as several chords to a triad matcher.
    Before reporting a progression, pool chroma **per bar in the instrument's own
    register** (high-pass away the bass) and check whether the set actually changes — a
    drone with a walking bass is one chord, not four.
13. `basic-pitch` duplicates notes at exact octaves and invents low-register content that
    the stem does not contain. Floor the output at the instrument's real range (C3 for a
    piano out of `other.wav`) before pushing anything to Ableton.

## Local environment (this machine)

- Tooling root: `/Users/peripan/dev/abletonAI/audio-analysis/`
- `.venv-bp/bin/python` — madmom, librosa, basic-pitch, crepe, pretty_midi.
  `.venv-demucs/bin/python` — demucs.
- **Whisper is NOT in any audio-analysis venv.** It lives at
  `/Users/peripan/mlx-openai-whisper/bin/python`, and only
  `mlx-community/whisper-large-v3-turbo` is cached — naming `whisper-large-v3-mlx`
  triggers a ~3 GB download.
- `mlx-demucs/.venv/bin/mlx-demucs` — for any multi-song batch (plain htdemucs; 5 songs in
  3.5 min). `htdemucs_ft` without `-d mps` runs on the CPU at ~2 s per second of audio.
- Existing helpers: `analyze_chords.py` (Phase 5b — stale: set `MAJOR_BIAS_MARGIN = 0`),
  `analyze_chord_onsets.py`, `scripts/whisper_vocals.py` (Phase 4), chart generators
  `gen_*.py` per song folder.
- Source docs (read-only): `WORKFLOW-song-analysis.md` in the tooling root.