git:20260925.dca1d7f to git:20260925.2eb4071

- description: Use when analyzing a song recording — extracting tempo/downbeats/key, separating stems, transcribing bass to MIDI, identifying chords per bar, timing lyrics, reading drum patterns, or producing a chord+lyric chart for a band. Covers madmom, demucs (htdemucs_ft), CREPE, basic-pitch, chroma chord proposal, and Whisper word timing.

- Song-agnostic pipeline, validated over many real projects (dream pop, Greek

- laiko, punk reinterpretations). Works for any song, genre, or key.

- - `references/environment-setup.md` — first-time venv install + per-song folder layout

- - `references/chart-and-lyrics.md` — chart structure, the 6 lyric-placement rules, HTML output

- - `references/drum-pattern-analysis.md` — kick/snare pattern reading from the drums stem

- - `references/modal-theory.md` — power-chord shorthand, the mode test, Byzantine ≠ Phrygian

- | Step | Tool | Why |

- | Pulse (beats only) | **madmom** `DBNBeatTrackingProcessor` | No meter assumed — see Phase 1a |

- | Meter / bar length | **`scripts/detect_meter.py`** | Sweeps every cycle 2–25; finds odd meters |

- | Key (melodic mode) | **madmom** `CNNKeyRecognitionProcessor` + bass-pedal check | 24-key probabilities; signal only — see Trap 2 |

- | Stem separation | **demucs `htdemucs_ft`** (4-stem) | Cleanest bass. `htdemucs_6s` only for an isolated guitar/piano stem — never for bass |

- | Bass → MIDI | **librosa pyin** (40–220 Hz) | Monophonic tracker; gives the ROOT, not quality. Not CREPE — see Phase 3 |

- | Polyphonic parts → MIDI | **basic-pitch** | Piano/guitar/pads, double-stops |

- | Chords | **chroma + bass-root** (+ slash relax) | See Phase 5b |

- | Lyrics + word timing | **mlx-whisper `large-v3-turbo`** (Apple Silicon) | Word timestamps, hallucination-filterable |

- | Drum-pattern read | **librosa** band-limited onsets (ADTOF optional) | see reference |

- | Chart output | Python → self-contained **HTML** | see reference |

- ## Phase 1 — Foundation (pulse + meter + key)

- ### 1a. Track the pulse WITHOUT assuming a meter

- import librosa, numpy as np

-

- act = RNNBeatProcessor()(audio_path)

- beat_times = DBNBeatTrackingProcessor(fps=100)(act) # beats only, NO bar assumption

- beat = float(np.median(np.diff(beat_times)))

- pulse_bpm = 60.0 / beat

-

- **Key: report the top two with probabilities, then check the bass pedal.** The CNN beats

- chroma + Krumhansl by ~20 MIREX points and fixes relative-major/minor flips, but it knows

- only 24 major/minor keys and its probability is not a confidence on modal or drone

- material (a Lydian track came out as the wrong key at p = 0.59). So also take the pitch

- class the bass sustains longest (duration-weighted, after Phase 3): if it disagrees with

- the key estimate, trust the pedal as the tonic and name the mode with the mode test in

- `references/modal-theory.md`. Either way this is the *melody's* key — Trap 2.

- **Never pass `beats_per_bar=(3,4)`.** A constrained candidate list cannot return "none of

- these" — it returns the least-bad of the options you supplied, with full confidence, and

- every downstream number (bar counts, per-bar chord cells, drum histograms) silently

- inherits the error. Get the pulse first; derive the meter in 1b.

- ### 1b. Derive the meter by sweeping every cycle length

- --foundation analysis/foundation.json

- **Always pass `--bass` and `--other` too.** Plenty of songs have no drum kit, or a

- percussion part that is an undifferentiated pulse carrying no accent — and the cycle is

- still there, articulated by the bass and the harmony instead. The signature is a drum

- contrast stuck near 1.2 while the bass reaches 3–4 and resolves the cycle cleanly. The

- script falls back to these stems automatically when no two drum bands agree.

-

- Phase-folds band-limited kick / snare / hat onsets onto every cycle length from 2 to 25

- pulses and ranks by **accent contrast** = loudest position in the cycle ÷ quietest.

- Run it AFTER Phase 2 so it has a drums stem; a full mix works but reads muddier.

-

- Reading the output:

-

- | Output | Means | Do |

- |---|---|---|

- | Consensus, confidence HIGH | Real cycle, clear of unrelated periods | Use it. Odd winner → odd meter |

- | Consensus on 8, 16, 24 (script prints a PHRASE note) | Usually plain 4/4 with an *n*-bar pattern | Report 4/4 + phrase length, not an *n*-beat bar |

- | Consensus on 2 | Duple; the bar-level accent is too faint to fix 2/4 vs 4/4 | Report duple; ask the user to count |

- | INCONCLUSIVE | Bands disagree, no usable accent, or no band clear of unrelated periods (margin < 1.3) | Report the ambiguity. Do NOT pick one |

- | Winner is prime (7, 11, 13) | Cannot be a phrase of anything smaller | Almost certainly the meter |

- | Winner = 2× a strong period | The half is the bar, the double is two bars | Take the fundamental |

-

- **The sweep can only find cycles on the pulse grid you hand it.** If the tempogram peaks

- at twice the tracked pulse, the beat tracker is reading half-time: re-run on the doubled

- grid, because a cycle of 8 there shows up as 4 here. Check the tempo octave *before*

- trusting any cycle length.

-

- Needs librosa, plus madmom unless `--foundation` supplies the beat grid. `--json` writes

- every band's sweep, per-band verdicts (cycle, margin, accented positions) and the

- consensus. Smoke test: `<venv>/bin/python tests/test_detect_meter.py` (synthetic 4/4,

- 3/4, 6/8, 5/4, 7/8, 11/8 and flat clicks; ~8 s).

-

- **A song with no percussion at all cannot be metered this way** — say so and move on

- rather than straining the harmonic stem for an answer it does not contain.

- **The two strongest positions in an odd cycle mark its internal split.** An 11 with peaks

- at 1 and 7 is 6+5; peaks at 1 and 5 would be 4+7.

- Save as `analysis/foundation.json`: `pulse_bpm`, `beat_times`, `beats_per_bar` (from 1b,

- **not** from madmom), `pulse_unit` (4 if the pulse is a quarter note, 8 if an eighth),

- `grouping` (the internal split, e.g. `[6, 5]`; `[2, 2]` for 4/4), `downbeat_times` (every

- *n*-th beat from the winning phase), `num_bars`, `key` (top two + probabilities),

- `bar_bpm` (per bar: 60 / median inter-beat interval in that bar, same unit as the pulse) and

- `tempo_drift_pct` ((max − min) / median of `bar_bpm`).

- 6/8 → 3+3, 11/8 as 6+5 → two unequal cells. Never split an odd bar at its midpoint — that

- lands mid-pulse.

- **Sanity checks:** read `bar_bpm`, not just one BPM — played music drifts, and above

- ~±3% a single tempo misplaces notes by the end (`ableton-mcp` → "Timing"). Watch

- half/double-time on slow songs (60–90 BPM) — count along the audio; cross-check `num_bars` against

- `duration ÷ (beats_per_bar × beat)`. A large mismatch means rubato, a wrong tempo octave,

- **Always pass `-d mps` on Apple Silicon.** demucs 4.0.1 defaults to CUDA-else-CPU, so

- without the flag it runs on the CPU. Measured on an M3 (30 s clip): CPU 64.5 s, MPS 23.0 s,

- stems within noise of each other. Meter-only runs need just the drums:

- `--two-stems=drums`.

-

- **CHECK THE CACHE FIRST — `du -sh ~/.cache/torch/hub/checkpoints`** (demucs 4.0.1; 4.1.x

- loads from the Hugging Face cache instead). `htdemucs_ft` is a bag of 4 models (4 × 84

- MB). If it isn't cached, that command silently downloads for as long as your connection

- takes — measured at ~3 MB/min (≈100 min) on 2026-08-06 — and shows no progress at all if

- you pipe it through `tail`. Don't assume it's cached because you used it

- on a previous song.

-

- **Fast offline fallback (Apple Silicon):** plain htdemucs via MLX, weights already local —

- 28 s for a 4-minute track (8.7× realtime):

-

- ```bash

- <audio-analysis>/mlx-demucs/.venv/bin/mlx-demucs <song.mp3> -o stems_mlx -v

- ```

-

- Call the venv binary **directly**. `uv run mlx-demucs` re-resolves dependencies over the

- network and hangs 10+ min even though the venv is already complete. The MLX port is within

- 0.03% of the PyTorch htdemucs reference, so it's a sound basis for chords and bass; re-run

- on `_ft` later only if quality looks marginal. Note its README lists `htdemucs_ft` as a

- supported model, but only plain htdemucs weights ship in `weights/`.

-

- **Quality check:** listen to `bass.wav` alone — piano/guitar bleed means separation

- struggled; try plain `htdemucs` as fallback.

- Default: **pyin** on `bass.wav` (monophonic → root line). For the pyin→MIDI code and

- cleanup rules, use the `bass-transcribe` skill — same pipeline. Use **basic-pitch** only

- if the part is polyphonic. CREPE is for vocals, not bass (reasons in `bass-transcribe`).

-

- Then aggregate per `(bar, cell)`: for each cell, total each pitch class's overlapping

- note duration and rank → `analysis/bass_per_cell.json`.

-

- **Spell pitch classes from the key signature, not from a fixed table.** Flat spelling

- (C, D♭, D, E♭ … B♭, B) is the right default for flat and neutral keys — it matches how

- guitarists read pop/indie charts. But applying it blindly in a **sharp key produces

- nonsense**: in F♯ minor (three sharps) the flat table renders the tonic as `G♭m`

- instead of `F♯m`, and the dominant as `D♭7` instead of `C♯7`. Rule: if the

- key signature has sharps, use sharp spelling (C♯, D♯, F♯, G♯, A♯); if flats or none, use

- flats.

-

- **Pitfalls:**

- - Sub-0.5 s detections are noisy; ≥0.7 s sustained notes are usually right.

- - Bass doesn't always play roots — it pedals, walks, sits on 3rds/5ths. Bass note is a

- *clue*; annotate in chart only when bass ≠ chord root.

- - **Bass enters late** in most songs. Find the bass-entry bar (first clearly audible bar)

- and suppress all bass-derived annotations before it — pre-entry content is stem bleed.

-

- ## Phase 4 — Lyric timing (Whisper on the VOCALS STEM)

-

- Whisper is far more accurate on isolated vocals than the full mix. **Gate it on where

- the stem is actually sung** instead of fiddling with `no_speech_threshold`: an RMS gate

- on the vocal stem, fed in as `clip_timestamps`, cut long-form lyric WER 22.9% → 20.7% and

- removed most filler hallucinations on separated vocals (arXiv 2506.15514).

-

- ```python

- import librosa, numpy as np, mlx.core as mx, mlx_whisper

- from mlx_whisper.transcribe import ModelHolder

- REPO = "mlx-community/whisper-large-v3-turbo" # large-v3 = extra ~3 GB, no gain

- y, sr = librosa.load(VOCALS, sr=16000, mono=True)

- rms = librosa.feature.rms(y=y, frame_length=2048, hop_length=512)[0]

- t = librosa.frames_to_time(np.arange(len(rms)), sr=sr, hop_length=512)

- sung = rms > 0.1 * rms.max() # the gate

- segs, start = [], None # sung runs → [start, end]

- for ti, on in zip(t, sung):

- if on and start is None: start = ti

- if not on and start is not None: segs.append([start, ti]); start = None

- if start is not None: segs.append([start, t[-1]])

- merged = [] # merge gaps < 1 s, cap at 30 s

- for s0, e0 in segs:

- if merged and s0 - merged[-1][1] < 1.0 and e0 - merged[-1][0] <= 30: merged[-1][1] = e0

- else: merged.append([s0, e0])

- dur = len(y) / sr

- clips = [(max(0, s0 - 0.4), min(dur, e0 + 0.4)) for s0, e0 in merged] # pad: onsets lead

- # turbo's curated word-alignment heads — mlx-whisper never sets them itself

- ModelHolder.get_model(REPO, mx.float16).set_alignment_heads(

- b"ABzY8j^C+e0{>%RARaKHP%t(lGR*)0g!tONPyhe`")

- res = mlx_whisper.transcribe(

- VOCALS, path_or_hf_repo=REPO, word_timestamps=True,

- clip_timestamps=[x for c in clips for x in c],

- condition_on_previous_text=False, # avoids hallucination drift

- hallucination_silence_threshold=2.0) # add language="en"/"el"/… if known

- def in_gate(w): # keep a word if ANY of its span is sung

- m = (t >= w["start"]) & (t <= w["end"])

- return bool(m.any() and sung[m].any())

- words = [w for seg in res["segments"] for w in seg.get("words", []) if in_gate(w)]

- Judge a word by its **whole span**, not its start — Whisper often starts a sung word

- before the voice is audible, and filtering on start time drops real words. Force

- `language=` when the auto-detect wobbles (chant, non-English). Then post-filter known

- hallucinations ("Thank you.", "Blah Blah", ".") and 3+ identical repeats. On a

- three-minute stem the gated run took ~4 min on an M3 including model load.

-

- `no_speech_threshold=0.8` (if you skip the gate) also drops real, sparse vocals — it can

- return nothing for a track that has singing. The curated alignment heads are what

- openai-whisper uses for this model; their effect on sung vocals hasn't been A/B'd.

- Map each word's start time to (bar, beat):

- def locate(t, downbeats, beats_per_bar): # from foundation.json — no default, never assume 4

- ## Phase 5 — Sections (manual, from lyrics + audio)

-

- For each canonical lyric line, find its first bar via Whisper + `locate()`. Section

- boundaries = where each labeled section's first line lands. **Don't trust automatic

- section detection** (librosa recurrence segmentation is ±1 bar off around bridges/outros);

- the lyrics + word timing are far more reliable.

-

- ## Phase 5b — Chord proposal (chroma + bass-root)

-

- Per cell, on the `other.wav` stem (everything harmonic except bass/drums/vocals).

-

- 1. **Chroma**: average `librosa.feature.chroma_cqt(other.wav)` over the cell's frames.

- 2. **Score all 24 triad templates** (12 maj + 12 min, binary root/3rd/5th, sum-normalized)

- by dot product with the cell chroma.

- 3. **Bass-root constraint** (only when `bar >= BASS_ENTRY_BAR`): if one bass pitch class

- has ≥0.5 s sustain in the cell, restrict candidates to the maj+min triads on that root —

- **unless step 4 relaxes it**.

- 4. **Slash-chord relaxation (REQUIRED — +17.7 points).** The constraint assumes the bass

- note *is* the root. On a slash chord the bass plays the 3rd or 5th, and constraining to

- it forces a guaranteed-wrong answer:

-

- ```python

- allc = sorted(all 24 templates by score, desc)

- con = [c for c in allc if root_of(c) == bass_pc]

- use_constrained = con and con[0].score >= RELAX_FACTOR * allc[0].score # 0.85

- ```

-

- If the best chord rooted on the bass note scores materially worse than the

- unconstrained best, the bass is a non-root chord tone — drop the constraint and let

- chroma decide. `RELAX_FACTOR = 0.85`; anything ≥0.85 behaves identically, 0.0 = off.

- Measured on a minor-tonic track built around a dominant-7th slash chord (C♯7/F, bass

- on the 3rd, E♯): without it every C♯7 cell became `Fm`, the minor triad on the bass

- note (14 misses, a third of all errors). With it: **73.8% → 91.5%** root+quality.

- 5. **Emit top-3** per cell + `beat3_re_attack` flag from onset detection on the same stem

- (corroborating evidence only — noisy).

- 6. **Major-bias diff pass.** Re-run with `MAJOR_BIAS_MARGIN = 0.05` (if the top template is

- minor and the same-root major is within the margin, swap) and **report the flip

- count**. Never adopt the biased result silently.

-

- ### `MAJOR_BIAS_MARGIN`: default 0, never gated on the key

-

- It is the single highest-leverage parameter. Same minor-tonic track (F♯m tonic), same

- audio, only this changed:

-

- | `MAJOR_BIAS_MARGIN` | root+quality |

- |---|---|

- | 0.05 | **27.4%** |

- | 0.02 | 54.9% |

- | 0 | **73.8%** |

-

- A 46-point swing: at 0.05 the tonic minor flips to major on nearly every cell. The 0.05

- bias was tuned on modal-major dream pop, where it reached 99% top-1 / 100% top-3 — that

- figure does not transfer to other material.

- **Never gate the bias on the detected key.** The key estimator reports the *melody's*

- mode, so a modal song pedalling on a major triad returns "major" — which switches on the

- very bias that destroys its minor chords. Measured: gating on a major key estimate flipped

- the majority of cells in a track (70%+), turning a minor loop into a phantom major one.

- A large flip count means the cells are near-ties and the quality call is genuinely

- uncertain — say so rather than picking silently. To settle major vs minor on one chord,

- compare chroma energy of the major third against the minor third directly: G♯ at 14.4%

- vs G♮ at 4.7% closes the question in a single number.

- Why the combination works: bass MIDI gives the root but not the quality; chroma gives the

- quality but confuses 4th/5th-related roots. Errors concentrate in pre-bass-entry bars,

- sustained-organ smear, and extended chords (expand templates with 7th/sus for jazz/R&B).

- Per-song tuning: `BASS_ENTRY_BAR`, template set.

- the beat tracker skipped) — and mark it on the chart.

- - Chord dict keyed by `(bar, cell)` (cells from `grouping`, Phase 1).

- - Output: single self-contained HTML, print-friendly, harmonic-notes block at bottom.

- Bass moving within a bar (`C → A♭`) has two readings, and both are common: (1) one held

- chord over a moving bass (`Cm/A♭`, or a drone with a walking bass) or (2) **two chords**

- (`Cm → A♭`). The bass alone cannot tell them apart — and a bass-constrained matcher

- simply follows the bass, so it reports (2) either way.

-

- **Discriminating test — look above the bass:**

- - **Re-attack**: a new voicing struck where the bass moves (in `other.wav`) = two chords.

- - **Register-pooled chroma**: pool chroma per cell in the chord instrument's own register

- (high-pass above ~165 Hz). If the pitch-class set changes with the bass = two chords;

- if it stays put = one chord over a moving bass.

-

- Bass movement alone is a suspicion signal, never the decision.

- Key estimators report the **melody's mode** (the PCP is melody-dominant), not chord

- quality. Many songs put **major triads under a minor melody** — that's the bittersweet

- sound. Default each chord to major, then verify: play root+maj3+5 against the melody; if

- the maj-3rd fights the melody's ♭3, go minor. Never propagate the key label to chord

- quality. Also: many modal songs pivot between relative major and minor — label the key as

- a pair (`E♭ / Cm`) when both centers appear.

- Folding onsets onto the wrong period does not produce a *low* score, it produces a

- **flat** one. An 11-beat cycle folded onto 2, 3, 4 or 6 smears to near-perfect uniformity,

- because 11 shares no factor with any of them.

-

- Calibration — read your own contrast numbers against this:

-

- | Top contrast in the sweep | Reading |

- |---|---|

- | 1.0–1.3 at *every* period tested | Wrong periods tested, or genuinely no accent. Widen the sweep |

- | 1.5–2.5 | Weak or ambiguous — report it as such |

- | 3–10 at one period, ≤2.5 at every unrelated one | A real cycle. Trust it |

-

- A coprime cycle read through the wrong fold lands squarely in the first row: uniform, not

- low.

-

- **Uniform contrast at every period you tested is not a finding. It means you have not

- tested the right period yet.** Widen the sweep before concluding anything, and never

- report "no time signature" on the strength of a narrow sweep.

-

- Corollary: assume odd meters are live. 5/4, 7/8 and 11/8 are ordinary in Balkan,

- Byzantine, prog and much devotional music — exactly the material this pipeline gets used

- on. Coprime blindness makes them invisible to a 4/4-shaped test.

- **Verify with a player.** All automated signals are wrong ~30% of the time on

- modal-mixture songs. A musician who knows the song is ground truth — plan a verification

- pass before declaring the chart done.

- 1. `htdemucs_ft` for bass, never `htdemucs_6s` — 6-stem bleeds piano/guitar into bass

- MIDI. `htdemucs_6s` is for isolating guitar or piano only.

- 2. Trust the user's ear over any analyzer disagreement.

- 3. Chart bar = audio bar.

- 5. A mid-bar bass walk may or may not change the chord (Trap 1) → key by `(bar, cell)`

- so either reading can be written down.

- 6. Chord-recognition tools are starting points, never ground truth.

- 7. Each chart correction touches ~3 places (chord dict, row layout, cascading lyrics).

- 8. Mirror parallel sections visually.

- 9. Section-boundary pickups: overlay in the same cell, don't displace.

- 10. Never hand an analyzer a candidate list it can't say "none of these" to — a

- constrained `beats_per_bar` and a key-gated `MAJOR_BIAS_MARGIN` both produced

- confident wrong answers that survived every downstream check.

- 11. Ask the user to count. A player's count settles in seconds what a sweep can only

- rank, and it outranks the sweep.

- 12. A static upper voicing over a moving bass reads as several chords to a triad matcher.

- Before reporting a progression, pool chroma **per bar in the instrument's own

- register** (high-pass away the bass) and check whether the set actually changes — a

- drone with a walking bass is one chord, not four.

- 13. `basic-pitch` duplicates notes at exact octaves and invents low-register content that

- the stem does not contain. Floor the output at the instrument's real range (C3 for a

- piano out of `other.wav`) before pushing anything to Ableton.

- - Tooling root: `/Users/peripan/dev/abletonAI/audio-analysis/`

- - `.venv-bp/bin/python` — madmom, librosa, basic-pitch, crepe, pretty_midi.

- `.venv-demucs/bin/python` — demucs.

- - **Whisper is NOT in any audio-analysis venv.** It lives at

- `/Users/peripan/mlx-openai-whisper/bin/python`, and only

- `mlx-community/whisper-large-v3-turbo` is cached — naming `whisper-large-v3-mlx`

- triggers a ~3 GB download.

- - `mlx-demucs/.venv/bin/mlx-demucs` — for any multi-song batch (plain htdemucs; 5 songs in

- 3.5 min). `htdemucs_ft` without `-d mps` runs on the CPU at ~2 s per second of audio.

- - Existing helpers: `analyze_chords.py` (Phase 5b — stale: set `MAJOR_BIAS_MARGIN = 0`),

- `analyze_chord_onsets.py`, `scripts/whisper_vocals.py` (Phase 4), chart generators

- `gen_*.py` per song folder.

- - Source docs (read-only): `WORKFLOW-song-analysis.md` in the tooling root.