CLAUDE.md · git:20261005.491ea00 · 2026-10-05 · sha256 052638cf63f05c7f
CLAUDE.md git:20261005.491ea00A
Immutable. This exact content never changes and is served at /api/v1/blob/052638cf63f05c7f.
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## System Instructions
Before executing any task, you MUST read and strictly adhere to the constraints defined in `specs/agent-rules.md`.
## Backlog
Use GitHub Issues for feature requests and bugs. Do not create file-based todos.
## What This Is
A Claude Code skill (open Agent Skills format) that uses Gemini's multimodal API as a proxy to analyze YouTube videos. Gemini sees video frames at 1 FPS, reads on-screen text, and hears audio simultaneously. Claude's role is triage and conversation over the resulting markdown artifacts — it never calls Gemini directly during triage. The reading layer that produces mind maps, briefings and topic digests also digests non-video sources - a community's newsletter issues and the talk pages behind them - through the same templates; see `docs/reading-layer.md`.
## Commands
```bash
# Scan all configured channels (generates mind maps, optionally transcripts)
python scripts/video_intel.py scan
# Scan one channel
python scripts/video_intel.py scan --channel natebjones
# Preview what would be scanned (no API calls)
python scripts/video_intel.py scan --dry-run
# Override lookback window
python scripts/video_intel.py scan --since 30d
# Transcribe a specific video (channel auto-detected from config)
python scripts/video_intel.py transcript --url "https://www.youtube.com/watch?v=XXXXX"
# Full pipeline on a local MP4 with one Gemini upload
# (mindmap + transcript + concepts; lazy-skips already-done steps without re-uploading)
python scripts/video_intel.py process --file "./video-intel/earlyaidopters/some-talk.mp4"
# Override Gemini model (e.g., Pro for transcripts when Flash truncates)
python scripts/video_intel.py --model gemini-2.5-pro transcript --url "URL"
# Install dependencies
pip install google-genai google-api-python-client pyyaml
# Optional: vector search
pip install lancedb voyageai
# Build vector search index (requires VOYAGE_API_KEY)
python scripts/video_intel.py index
# Semantic search over transcript chunks
python scripts/video_intel.py search "permission problems" --vector
# Catch-up briefing: surface corpus videos not yet in any _briefings/ guide
python scripts/video_intel.py briefings --unseen --dry-run # preview only
python scripts/video_intel.py briefings --unseen # write the briefing
# Personalization: what ranks briefings + the headline digest (one shared profile)
python scripts/video_intel.py profile show # read-only; prints the model + both file paths
python scripts/video_intel.py profile init # persist profile.yaml + scaffold audience.md
```
Required env vars: `GEMINI_API_KEY`, `YOUTUBE_API_KEY`.
Optional: `VOYAGE_API_KEY` (for vector search, free at https://dash.voyageai.com/).
Optional: `VIDEO_INTEL_OUTPUT_DIR` (absolute path to the corpus; reached only when the plugin's `config.yaml` is absent — see Corpus Discovery below).
## Architecture
**Plugin, not single skill.** This repo ships as a **plugin** — a container that holds multiple independent skills — per the current Anthropic plugin format. Layout:
```text
video-intel/ ← plugin root (git repo root)
├── .claude-plugin/plugin.json ← plugin manifest (name, version, skill list)
├── skills/
│ ├── video-intel/SKILL.md ← curate: scan / transcript / mindmap / process / index / dedupe / concepts / taxonomy-build
│ ├── video-intel-search/SKILL.md ← query: search / nugget (writes only its own brief) / status / profile show (globally installable)
│ └── translate-bcs/SKILL.md ← BCS subtitle translation
├── scripts/ ← shared by all skills
├── prompts/ ← shared by all skills
├── config.yaml ← gitignored; per-user. Copy config.yaml.example to create.
├── config.yaml.example ← committed template
└── tests/ ← covers shared scripts
```
Each `SKILL.md` has its own frontmatter description and is independently triggered by Claude Code. Scripts and prompts are shared at the plugin root so the skills can reuse `translate_video.py`, `gemini_common.py`, etc., without duplication. The operational-separation rule (translate_video.py does not read video-intel's config/taxonomy/meta.json) is unchanged — scripts stay independent even though they ship in the same plugin.
**When the plugin is installed**, all three skills become available to Claude. A user asking "scan my channels" triggers `video-intel` (curate); a user asking "find videos about MCP" triggers `video-intel-search` (read-only); a user asking "translate this video to Bosnian" triggers `translate-bcs`. No cross-skill invocation is required — each skill's body tells Claude which CLI commands to run, and Claude executes them from the plugin's shared scripts directory.
### Corpus Discovery
`load_config()` in `scripts/video_intel.py` resolves `output_dir` via a four-step precedence chain (see KD1 of `docs/plans/2026-04-23-001-feat-search-skill-portability-plan.md`):
1. **`SKILL_DIR/config.yaml`** — the plugin-local config, gitignored. Authored by the developer running curate workflows from a source checkout. Wins when present so a stale env var cannot silently redirect `scan` away from the canonical corpus.
2. **`$VIDEO_INTEL_OUTPUT_DIR`** — env var override for users who point a cached plugin at a different corpus. Must be an absolute path.
3. **`~/.video-intel/config.yaml`** — user-level minimal config accepting `output_dir` (required) and `vector_db_dir` (optional). Extra keys are ignored with one INFO log.
4. **Hard error** naming both the env var and the user-config path.
**Smoke-test safety (learned the hard way 2026-09-01).** Because step 1 WINS, setting `VIDEO_INTEL_OUTPUT_DIR` is NOT enough to keep a smoke test off the live corpus: a `config.yaml` in the working tree overrides it silently, and a Gate-1 smoke that runs `index` or `scan` will then rebuild the REAL index. Any smoke that must not touch live data has to **park the config** (`mv config.yaml config.yaml.parked`) so the env var wins, and **assert in the harness that it is absent** before running anything - `assert not (WT / "config.yaml").exists()`. This bites specifically in a worktree, where the gitignored `config.yaml` must be copied in for normal curate work and then removed again for the smoke.
One INFO log line per invocation names the winning source (e.g. `"Config resolved from VIDEO_INTEL_OUTPUT_DIR=/foo"`).
**Curate guard:** curate commands (`scan`, `concepts`, `dedupe`, and the `--channel` branch of `mindmap` / `transcript` / `process`) require `channels:` in the resolved config. Running them with the user-level minimal config (no channels) fails fast with an actionable message. Read-only commands (`search`, `status`, `index`, `taxonomy-build`, `profile show`) do not require `channels:`, and neither does `nugget` - it writes only an additive brief under `_briefings/nuggets/` and stays exempt from config snapshots (a channel-less config must never overwrite `config.latest.yaml`, the record of the channel list that produced the corpus). `profile init` does not require `channels:` either - it stays curate-routed because it WRITES, not because it needs channels (issue #117).
### User-level install (`video-intel-search` skill anywhere)
The read-only search skill can be made available from any project via a
user-level marketplace entry in `~/.claude/settings.json`. See
[INSTALLATION.md](INSTALLATION.md#claude-code-user-level-access-the-search-skill-from-any-project)
for the step-by-step procedure (JSON to paste, OS-specific absolute paths,
env-var vs user-config choice).
> **Critical:** the marketplace key under `extraKnownMarketplaces` and the
> suffix after `@` in `enabledPlugins` MUST both be exactly `video-intel`
> (matching `.claude-plugin/plugin.json`'s `name` field). Claude Code
> normalizes the key silently - any suffix like `-local` is stripped, the
> `enabledPlugins` entry points at a marketplace name that does not exist,
> and skills never appear in other projects. If you hit "skills not
> appearing after Claude Code restart", this is the first thing to check.
Curate operations (`scan`, `process`, `concepts`, `dedupe`, and the
`--channel` branch of `mindmap` / `transcript` / `process`) still require
the plugin repo as CWD - they read `channels:` from the plugin-local
`config.yaml` which the user-level fallback does not provide.
`profile show` is deliberately reachable from the globally installed search
skill (issue #117): "why am I seeing this" is a question asked from wherever
the user happens to be working, and the command writes nothing. `profile init`
stays curate-only. The split is by **write scope, not by topic** - both
commands concern the same two files, so a future edit that moves `init` into
the search skill "to keep personalization together" would break that skill's
no-curate-writes guarantee (nugget persists only its own brief under
`_briefings/nuggets/`). Test contract: `tests/test_skill_descriptions.py::TestPersonalizationRoutingSplit`,
`tests/test_curate_guard.py::TestProfileNeedsNoChannels`.
**Shared utilities:** `scripts/gemini_common.py` — Gemini retry logic (`get_retry_delay`), client factory with httpx timeouts (`create_client`), lazy imports (`require_gemini`, `require_youtube`). Used by both scripts; kept minimal.
**Single script:** `scripts/video_intel.py` — all logic in one file, subcommands:
- `scan` — uses YouTube Data API to discover new videos per channel, then runs (issue #54) the **transcript loop FIRST**, the mindmap loop SECOND, and the concepts loop THIRD. The mindmap loop's per-video source is decided by `resolve_mindmap_source(channel_cfg, transcript_available=...)`: `auto` (default) routes to text-from-transcript when one landed on disk in Step 1 (cheaper, faster, no 10800-frame cap), else falls back to video. Channel knob `mindmap_source: auto|transcript|video|none` overrides per-channel. Legacy `auto_mindmap: none` is preserved as a separate top-level scan-skip gate. Optionally chains transcript and concept generation — **issue #173 (scan half): the `auto_concepts` loop's candidate pre-filter (`concepts_path.exists() and not args.force`) and its `process_concepts` call (`force=args.force`) both honor `--force`**, mirroring `cmd_concepts`'s existing gate; pre-fix, `scan --force` regenerated every mindmap but silently left every existing concepts.json untouched, pairing fresh mindmaps with stale concepts on the path a bulk remediation (issue #172) is most likely to use. **Under `--force` the candidate loop is additionally bounded to `touched_prefixes`** (the same window-bounded, force-aware `new_videos` set the mindmap loop just ran against, captured right before that loop) - review round 3 caught that honoring `--force` alone turned the corpus-wide `*.meta.json` glob (unbounded by `--since` or the fetch window, and reached even on a zero-new-videos scan per the `auto_concepts` carve-out in the `if not videos:` gate) into a full-corpus concepts re-extraction on every `scan --force`. Without `--force` the pre-filter is unchanged. Corpus-wide re-extraction keeps its correct home in `concepts --channel X --force`, which is where issue #172's remediation should point. **Issue #176: the `auto_concepts` loop now accumulates each video's newly-extracted concepts into the in-memory taxonomy before the next `process_concepts` call, per ADR-0010, matching `cmd_concepts`** - so two videos in one scan (or two channels in one scan) normalize against each other's freshly-minted concepts instead of the on-disk snapshot from before the scan started. See the Code Review Guardrails entry for the shared-helper contract. Supports **selective mode**: channels with `playlists` or `keywords` in config skip the date-based uploads scan and only process matching videos. `resolve_playlist_ids()` resolves human-readable names to IDs via case-insensitive contains matching. `fetch_keyword_videos()` uses `search().list()` (100 quota units/call, capped at `KEYWORD_MAX_PAGES` pages). `fetch_selective_videos()` dispatches and deduplicates. After all primary processing and the failure summary, a full `scan` (not a `--channel X` run) renders the **headline digest** (issue #113) via `render_headline_digest`: peripheral vision over `enabled:false` + `headline_digest:true` YouTube channels, metadata-only (no Gemini, no corpus artifacts), ranked by `rank_headlines` against the compiled interest model from `load_interest_model` (read-only; never persists), seen-state in `_headlines/seen.json`. See the Code Review Guardrails entry for the five invariants.
- `transcript` — calls Gemini with `response_json=True`, parses the three-task JSON response (speech + screen_content + speakers), and merges them into a fused markdown document via `merge_transcript_json()`. Resilient to malformed JSON: tries direct parse, then `isolate_json()` cleanup, then `salvage_transcript_sections()` to recover partial content. **Task-wrapper normalization** ([issue #45](https://github.com/dzivkovi/video-intel/issues/45)): both the full-parse path (`try_parse_transcript_json`) and the salvage path (`_normalize_task_wrapper` at the top of `salvage_transcript_sections`) call `_wrapper_to_envelope_dict()` to detect Pro's `[{"task": ..., "output": [...]}, ...]` malformation and rewrite it into the flat `{"transcripts": [...], ...}` envelope. The helper only fires when the parsed top-level is a list whose items name a `_KNOWN_TASK_KEYS` task; non-wrapper inputs pass through unchanged. A scoped `_strip_cyrillic_for_structure()` strips Cyrillic-token intrusions inside `_normalize_task_wrapper` only, never globally - the global pre-strip rejected by the issue would risk corrupting verbatim foreign content. Saves raw Gemini response as `.transcript.raw.txt` sidecar on failure for forensics. One bounded retry if salvage fails. Partial transcripts are written with a visible warning block and `transcript_status: "partial"` in meta.json. Accepts either `--url` (YouTube) or `--file` (local MP4) as input. For local files, uses `upload_local_video()` via Gemini Files API (48h auto-expire); by default output lands next to the source file with the filename stem as prefix. **Channel-scoped local recovery (plan rev 4):** pass `--channel <NAME>` alongside `--file` (or drop the file under `output_dir/<channel>/` and let the parent-folder inference pick it up) to route artifacts into the canonical channel folder with the same meta.json shape as scan-generated artifacts. `resolve_local_file_identity()` picks the prefix: (1) sibling `.meta.json` wins, (2) G2 dedup against canonical scan metas by `video_id`, (3) explicit `--video-id`/`--title`/`--date` flags, (4) filename stem plus `LastWriteTime`. **Issue #186: on the flags path, BOTH `--title` and `--date` together derive the artifact prefix through the scan writer's own `video_file_prefix` (`{date}-{slug}`), so curated local ingests follow the same naming convention as scanned artifacts.** One flag alone keeps the stem: a prefix built on the mtime-fallback date would change whenever the file is copied (new mtime, new prefix, duplicate artifacts on the next run). The sibling-meta and G2 prefixes are deliberately untouched - they carry already-written artifacts (renaming existing artifacts is a manual follow-up, out of scope by the issue's own statement). `--date` is validated as strict `YYYY-MM-DD` at the PARSER (`iso_date_arg`, all three subcommands) because the flag now builds a filesystem path and a `2026/08/31` would embed a separator - the parser runs before any upload, per probe-before-you-pay; the resolver keeps warn-and-fall-back-to-stem belts for library callers and for a title that slugifies to nothing. Step 1 also adopts a stem-named meta found in the TARGET channel dir (a pre-#186 ingest from outside the corpus left one there; missing it would re-bill Gemini for a duplicate ingest), the true sibling wins when both exist, and the step-1 return's `channel_dir` is the adopted meta's own directory so artifacts can never split from their meta. **The adoption is conflict-guarded (`_adoptable_channel_stem_meta`, Codex peer-pass P1)**: a same-basename meta can belong to a DIFFERENT video, and adopting it returns at step 1 before G2's video_id matching can object - so a conflicting explicit `--video-id` OR id-shaped stem OR a conflicting stored `channel` rejects the candidate with a WARNING and resolution continues to G2/fallback; an unreadable candidate is rejected the same way (adoption is an optimization, and the wrong video's meta corrupts identity). The true sibling next to the file keeps its pre-existing adopt-with-flag-overrides contract - the guard is only for the channel-dir candidate. Known residual, accepted: an id-less local file processed once WITH flags and once WITHOUT splits across two prefixes - nothing can find the derived meta without a `video_id`; the recovery is consistent flags or `--video-id`, and the 403-recipe path is safe because id-shaped stems dedupe through G2. Reviewers: the derived prefix must come from `video_file_prefix`, never a re-derived `f"{date}-{slugify(title)}"` copy - the PR #136 checker/writer rule. Test contract: `tests/test_local_mp4_recovery.py::TestPrefixFromExplicitFlags`, `::TestDateFlagValidation`, `::TestChannelDirStemMetaAdoption`, `tests/test_cmd_process.py::TestCmdProcessFileDerivedPrefix`. Canonical `video_url` and Gemini `media_uri` stay separate fields so `file_uri` never persists to disk. Both paths support `--start`/`--end` for segment clipping (parsed by `parse_time_to_seconds()`, accepts `MM:SS`, `HH:MM:SS`, or raw seconds). **Issue #248:** on the three YouTube-URL chunked call sites (scan, `transcript --url`, `process --url`) a chunk LOST to the #74 wall-clock cap (or any failure) returns `"partial (chunks lost: N of M)"`, and under `transcript_source: auto` the shared `_finish_chunked_transcript` tail replaces the WHOLE video from the caption track, keeping the Gemini partial as a `.transcript.raw.chunked-partial.txt` sidecar; under `gemini` the partial stays and the logged remedy names `--chunk-minutes 10` or `--transcript-source yt-captions`, never "re-run". A quality-only partial never triggers it. Invariants in `.claude/rules/transcript.md`.
- `mindmap` — generate a mind map for a single video. Accepts `--url` (YouTube) or `--file` (local video) as input. **Issue #54: the `--url` branch consults `resolve_mindmap_source` and reads on-disk transcript text when one exists**, calling Gemini text-only via `process_mindmap(source="transcript", ...)`. Falls back to legacy mindmap-from-video (with the issue #50 fps fallback for the 10800-frame cap) when no transcript is on disk and the channel's `mindmap_source` resolves to `auto`/`video`. The `--file` path mirrors `transcript --file` and stays on the legacy mindmap-from-video path (chunking it would defeat the one-upload guarantee that motivated `process --file`); drop a gated-video MP4 under `output_dir/<channel>/` (or pass `--channel` explicitly) to route the `.mindmap.md` + `.meta.json` into the canonical channel folder using the same identity resolver as transcript. Used by the scan 403 recovery flow: `scan` on a members-only video returns 403 and the log line prints a two-command recipe (`mindmap --file ... --channel ...`, then `transcript --file ... --channel ...`).
- `concepts` — extract and normalize concepts from existing mindmaps against a growing canonical vocabulary (thesaurus). Text-only Gemini calls reading mindmap markdown, not video.
- `process` — pipeline orchestrator. Two input modes:
- **`--url`** (YouTube): inverted ordering per issue #54 — Step 1 transcript (chunked if long, per PR #51), Step 2 mindmap-from-transcript (text-only Gemini call against the on-disk transcript), Step 3 concepts. Resolver picks mindmap source per channel; `auto` (default) uses transcript when Step 1 wrote one, else falls back to mindmap-from-video. The Step 1 transcript call is wrapped in try/except so an uncaught transcript exception still lets Step 2 run with `source="video"` fallback (preserves the "mindmap is the AI's discovery surface and must always run" invariant). The 10800-frame fps fallback is preserved on the video-fallback branch only — text input has no frame cap. **Issue #173: Step 3's `process_concepts` call passes `force=args.force`** (an isolated omission previously left `process --url --force` re-runs with stale concepts from the pre-force mindmap, silently, at exit 0). The ENTIRE step body — `mindmap_path.read_text()`, `load_taxonomy()`, and the `process_concepts` call — is wrapped in one try/except + `_record_concepts_error`, matching the `--file` path below: an uncaught exception anywhere in Step 3 has no other handler between `_cmd_process_url` and `main()`'s bare dispatch, so a preamble read that raises (a corrupt mindmap encoding, a cloud-mount `OSError`, an unparseable `taxonomy.json`) must land inside the same net as the call itself, not just wrap the call. The same fix (both the pre-filter and the `process_concepts` call) also applies to `scan`'s `auto_concepts` loop above. Test contract: `tests/test_process_force_propagation.py` (parametrized over all three steps and both `--url`/`scan` concepts sites to catch a future step or site that drops force propagation).
- **`--file`** (local MP4): UNCHANGED. Calls `upload_local_video()` once (lazy: skipped when meta.json already records all modes completed and artifacts exist on disk), threads the `file_uri` to `process_mindmap(..., media_uri=...)` and `process_transcript(..., media_uri=...)`, then runs `process_concepts(...)` inline on the now-on-disk mindmap text. Stays on legacy mindmap-from-video because chunking would multiply the upload cost, defeating the one-upload guarantee. Partial-success semantics: mindmap persists even if transcript fails; exit 0 whenever mindmap succeeded. File-expiry fallback: if either helper returns a status matching `_is_file_expiry_error_status()`, re-upload once and retry once.
The observability helper `log_usage_metadata()` logs token usage on every Gemini call through `process_mindmap`, `process_transcript`, `process_concepts` (via the `on_response` callback in `call_gemini`/`call_gemini_text`); `cmd_nugget`'s direct `generate_content` call at `:3356` is intentionally not instrumented. Accepts `--file` (required for the local path), `--url` (for the YouTube path), `--channel`, `--video-id`, `--title`, `--date`, `--start`/`--end`, `--force`, and `--prompt`.
- `taxonomy-build` — rebuild `taxonomy.json` by aggregating all per-video `concepts.json` files. This is a derived artifact, always rebuildable.
- `topics-build` - rebuild `topics.json` from the two topic-assertion sources: briefing front-matter `video_ids` keyed by the first folder under `_briefings/`, and per-video meta.json `topics` stamps written by `--topic`. Derived, byte-stable, always rebuildable, and it never touches `taxonomy.json`. `--dry-run` prints and writes nothing. Read surfaces: `status` (per-channel topics rollup), `search --topic <slug>` (with a query: filtered search; WITHOUT a query, issue #188: a pure listing of the topic's members from topics.json - no retrieval, no index), and `nugget --topic <slug>` (scopes synthesis retrieval to the topic's members). See the Code Review Guardrails entry for the invariants.
- `briefings --unseen` — catch-up briefings (issue #80). Selects corpus videos absent from every existing `_briefings/**/*.md` front-matter `video_ids` list (strict set difference via `load_seen_video_ids`, **never window-based** — so a video is never re-surfaced once it lands in any briefing), bounds them to a UTC date window (**unbounded by default as of issue #88** — `compute_catchup_window` returns `lower=date.min` when `--since` is absent, so a never-briefed video of any age is a candidate; `--since`/`--until` *narrow* back to a floor when wanted; the old 30-day `DEFAULT_RECENCY_DAYS` floor was removed because it hid old-but-never-briefed videos, the opposite of a catch-up's job) and caps to the top-N by relevance (`--limit`, default 30, `0` = no cap; uncapped videos stay unseen for the next run, so the cap is a rolling catch-up, not a drop). Ranks by concept/taxonomy overlap against the compiled interest model from `_briefings/profile.yaml` (`load_interest_model` then `compile_interest_model`, issue #115: inferred in memory from the scanned channel list + top `taxonomy.json` concepts when no file is persisted, and **never persisted by this command**: `profile init` is the only writer, and an existing file is **never overwritten**, so hand-edits are the retune path; tolerates a malformed hand-edit — list/scalar `interest_concepts`, bare-string `interest_domains`, string weights — without crashing). Writes `_briefings/<date>-catch-up-unseen.md` (suffixed if one already exists that day, so a same-day re-run never clobbers an earlier briefing's `video_ids`). Zero-score entries render without mindmap deep-links (their timestamps can mismatch the title). `--dry-run` prints the ranked unseen set and writes nothing (not even `profile.yaml`). The corpus walker `collect_corpus_videos` skips `.`- and `_`-prefixed dirs so `_briefings/` is never mistaken for a channel. No Gemini call and no `channels:` required — deterministic ranking only; the optional LLM-judgment layer is deferred per [the requirements doc](docs/brainstorms/2026-06-21-knowledge-gap-detection-requirements.md). **Temporal rendering (issue #88):** the primary list stays strictly relevance-ranked (recency is only a tiebreaker in `rank_unseen`, so old-but-high-relevance videos surface near the top, not buried); temporal structure lives in (a) a per-item **age badge** (`· age 3y`/`8mo`/`5d` via `_format_age` — a mechanical y/mo/d derivation, NOT a semantic "evergreen" judgment the score can't support) and (b) a secondary **"By year" appendix** (`_render_by_year_appendix`) that regroups the *same* `video_ids` newest-year-first. Do NOT put year/month headers inside the primary list — that fights the relevance sort. On the first run against a large corpus with no `profile.yaml` yet AND `total_unseen > limit`, `cmd_briefings` emits a one-line cold-start WARNING (freshly-inferred profile can overweight generic concepts; points at `profile show` / `profile init` + hand-edit) — it warns rather than reimposing a floor. **Topic subfolders under `_briefings/` (e.g. `_briefings/sales/`, or any other name) are a supported, code-free organizing convention** (2026-07-06) — `load_seen_video_ids` recurses (`rglob`, not `glob`), so briefings moved or written into a subfolder stay in the "seen" set. There is no per-topic config, flag, or naming convention baked into `briefings`; for BRIEFING SELECTION and SEEN-STATE folder names remain meaningless to the code and can change freely over time. **That claim stopped being universal in issue #146**: for TOPIC DERIVATION the first path segment under `_briefings/` IS the topic name, so renaming a topic folder renames the topic and re-slugs every membership on the next `topics-build` (`nuggets` is reserved and excluded). The two readings coexist on purpose - `load_seen_video_ids` still never looks at the name, and `topic_from_briefing_path` looks at nothing else. **PDF parity:** `scripts/briefing_pdf.py` re-implements the age badge (`_age`) and the By-year appendix locally (it deliberately does not import from `video_intel.py`); any change to the Markdown render's age/appendix behavior MUST update both, and `_age` must stay behaviourally identical to `_format_age` — test contract `tests/test_briefings_pdf.py::test_pdf_shows_age_badge_and_by_year_appendix`. Reviewers: any change that makes unseen-selection window-based instead of `video_ids` set-difference, that overwrites a hand-edited `profile.yaml`, that reverts `load_seen_video_ids` from `rglob` back to a non-recursive `glob`, that reintroduces a default recency floor (issue #88 removed it deliberately), or that inserts chronological headers into the primary relevance-ranked list, regresses the issue #80/#88 contract or silently un-sees subfoldered briefings — see `tests/test_briefings.py` (`test_load_seen_video_ids_recurses_into_topic_subfolders`, `test_compute_catchup_window_default_is_unbounded`, `test_render_unseen_briefing_by_year_appendix`).
- **Where the user's own preferences live, and how to change them.** The standing preference surface is two files in the CORPUS, never in the repo and never in an agent's memory: `<output_dir>/_briefings/audience.md` (hand-written prose - persona, standing pillars, current goals, what counts as signal, what counts as noise, delivery preferences) and `<output_dir>/_briefings/profile.yaml` (machine-scored interest weights that rank `briefings --unseen` and the scan headline digest). `examples/audience.md` is only the TEMPLATE that `profile init` copies; the live file is the one in the corpus. **When the user states a shift in what they care about, edit `audience.md` (and `profile.yaml` if weights should move) rather than recording it anywhere else** - a second home for the same fact is how the two drift apart. **Back up before every edit**: copy to a dated sibling (`audience.backup-<YYYY-MM-DD>.md`) in the same folder FIRST, because no command regenerates these files and `profile init` refuses to overwrite them, so a bad edit is unrecoverable. Say in the reply what changed and where the backup is.
- `profile show` / `profile init` - the personalization surface (issue #115). `show` prints the resolved interest model (source `persisted` vs `inferred`, top weighted concepts/domains, and the `output_dir`-relative paths of `_briefings/profile.yaml` + `_briefings/audience.md`) and writes **nothing**. `init` persists the inferred `profile.yaml` and scaffolds `audience.md` from `examples/audience.md`, never overwriting either file, including a partial or malformed one. No `profile edit` by decision (editing is opening the file; `show` prints the path). Neither requires `channels:`. See the "One compiled interest model" guardrail in [`.claude/rules/briefings.md`](.claude/rules/briefings.md) for the invariants.
- `search` — search corpus by concept label/alias (default) or hybrid BM25+vector (`--vector`). Concept search returns matching videos with artifact paths. Hybrid search returns ranked transcript chunks by combined keyword and semantic relevance (RRF fusion). Use this FIRST when the user asks about topics — avoids reading the entire corpus. **Stage-1 query expansion (2026-04-20, [ADR-0017](docs/adr/ADR-0017-kb-layer-strategy.md)):** hybrid mode preprocesses the query through `expand_query_via_taxonomy()`, appending creator-vocabulary siblings for any canonical label or alias in `taxonomy.json` that matches the query. The expander uses a punctuation-aware boundary (handles `C++`, `.NET`, `(MCP)`, `k3s` where stdlib `\b` fails), caps sibling additions at 12 per query to limit embedding dilution, and writes its expanded string to both the BM25 FTS call and the Voyage query embed. Pass `--no-expand` to disable and run the pre-Stage-1 baseline behavior for A/B comparison. `hybrid_search()` also accepts `return_diagnostics=True` to return `(hits, expansion_record)` — the eval harness uses this to write per-query records to `tests/evals/results/<run_tag>-expansion.jsonl`. Concept-search mode (`search_corpus()`) is intentionally untouched by Stage 1. **Concept-mode RANKING is specificity-first (issue #189):** the bag score over label+all-aliases keeps its SELECTION role unchanged (exactly the same concepts match, and the exact-vs-partial video-lookup RULE is untouched; on the PARTIAL path the five concepts that fetch videos follow the NEW order, so the returned video set can change there - intended, since `[:5]` means "the best five" and #189 redefines best; the exact path's video set is order-independent and stays byte-identical), but ORDER among equals is label-phrase FIRST (2026-09-02 taste lab, operator-delegated: a concept whose own PREFERRED LABEL carries the query phrase is the concept NAMED for the query and outranks alias-only matches), then phrase-in-one-field (via the shared `_alias_boundary_pattern` - one definition of "appears as a phrase"), then best single-field term coverage, then tightness (query terms as a share of that field's own tokens), then FOCUS (ascending `_alias_count`: among otherwise-equal claims a junk-drawer concept hoarding 400+ aliases has the weakest claim on any one of them - this is what routes a contested alias like "prompt engineering", held by BOTH a 435-alias mega-concept and the focused owner, to the focused owner), with `video_count` demoted to the LAST tiebreak. The label tier is the control that killed the naive focus-only variant: a query matching a mega-concept's OWN label ("agent configuration", "token economics") still returns it first. Measured on a 19-query adjudicated set: status quo 4/19 top-1, shipped chain 17/19, with both residuals adjudicated as eval artifacts (one where the shipped answer is BETTER than the expectation, one caused by a genuine taxonomy bug - 13 duplicate-label concept pairs exist, e.g. two ids both labeled "AI Agent Configuration"; merging those is data curation, not ranking). The tie-break counts TOKEN EQUALITY with boundary punctuation stripped (`_FIELD_TOKEN_STRIP`, conservative: no `+`/`#`/`.` so "c++"/"c#"/".net" survive) - a substring numerator over a token denominator made "prompting, engineering" a perfect, maximally tight match for "prompt engineering" (Codex peer-pass P2); the SELECTION bag keeps substring semantics on purpose. The phrase regex sees the WHITESPACE-NORMALIZED query so a pasted double space cannot silently disable only the phrase bonus - count-first rewarded genericness because the bag let each query term match a DIFFERENT alias (measured live: 20 mega-concepts at a perfect bag score buried `structured_prompting` at rank 21 of 111 for `search "prompt engineering"`; post-fix ranks on the six-query measurement set: 2/1/1/1/1/1 from 21/10/3/5/5/2, with the remaining rank-1 being a concept whose alias IS the exact phrase - the taxonomy's own claim, remediable by curation, not ranking). Non-string aliases degrade to ignored (the pre-#189 bare join raised TypeError over one bad entry). Reviewers: a diff that restores `video_count` above the specificity keys, or re-derives phrase matching instead of using `_alias_boundary_pattern`, regresses this. Test contract: `tests/test_concept_rank_specificity.py`.
- `index` — build search index (vector embeddings + FTS on title/text) from all transcripts using LanceDB + Voyage AI. Required before `search --vector`. Rebuildable at any time. **`--channel X` is INCREMENTAL as of issue #183**: it embeds only that channel, deletes only that channel's rows, appends the new ones and compacts, leaving every other channel untouched. It is the only incremental primitive the tool has (plain `index` re-embeds the entire corpus every run - there is no embedding reuse), which is what makes the re-index `dedupe` and `prune-shorts` both ask for affordable. Resilient to Voyage's per-batch token cap (120,000 tokens for `voyage-4-large`): `_embed_batch` catches the typed token-cap error and recursively halves the offending batch down to a `MIN_BATCH_SIZE = 4` floor, so adding 2-3 dense transcripts no longer fails the whole rebuild (issue #44).
- `dedupe` — find and clean up title-rotation duplicates (same `video_id`, different slug). Groups meta.json files by `video_id`; for any group with >1 meta, picks canonical **quality-first, then by latest `processed` timestamp** (tie-break on `modes_completed` size, then alphabetical prefix): a meta whose transcript tripped a severe `transcript_quality_flags` entry (issue #159, reusing `transcript_quality_flags_are_severe()` from the #157/#158 quality machinery) never outranks a clean duplicate, even a much older one, so a severe-flagged rerun can no longer beat a healthy artifact on recency alone. Within one severity bucket (both clean or both severe) the ordering is unchanged. Merges loser titles into canonical's `alt_titles` list, moves artifacts for any mode only a loser has, deletes loser siblings. Dry-run by default; pass `--apply` to mutate. After `--apply`, re-run `taxonomy-build` and `index --force` (derived artifacts are not auto-rebuilt: blast radius stays predictable). Prevention is automatic: `is_processed()` consults a per-channel `{video_id: prefix}` index before falling back to slug-based existence checks, so the same `video_id` under a rotated title is recognized as already-processed. A pre-scan pass inside `cmd_scan` calls `record_alt_title_if_rotated()` to capture ongoing rotations into existing metas' `alt_titles`.
- `prune-shorts` — find and delete YouTube Shorts that polluted the corpus before the scan-time filter existed. Walks `*.meta.json` files per channel, classifies via `is_short()` (duration < 60s OR `/shorts/<id>` HEAD redirect returns 200, with one bounded retry on transient errors and fail-safe to long-form on classification ambiguity), uses cached `duration_seconds` from meta.json when present and batches `videos.list(part='contentDetails')` for legacy metas missing the field. Dry-run by default outputs `title | duration | url | artifact_count` per Short plus per-channel summary; `--apply` deletes via the explicit `PRUNE_SHORTS_DELETION_PATTERNS` allowlist (mindmap, transcript, the `.transcript.raw*.txt` and `.mindmap.raw.txt` forensic sidecars, concepts.json, meta.json) — **NOT** the whole-prefix glob `_apply_dedupe_group` uses, because translate_video.py produces `.en.srt` and `.translate-bcs.txt` siblings that share the prefix and must survive (translate-bcs is operationally separate from curate). Mirrors dedupe's blast-radius discipline: after `--apply`, re-run `taxonomy-build` and `index --force` manually.
**Scan-time `skip_shorts` filter** (per-channel flag, default `true` as of plugin v1.11.0). `cmd_scan` calls `enrich_with_durations(youtube, video_ids)` (batched 50 per `videos.list` call) for all fetched videos so meta.json carries `duration_seconds` going forward, then drops Shorts before any Gemini call. Per-channel `skip_shorts: false` opts back in for substantive-Shorts creators. On HTTP 403 quota-exceeded mid-classification, the scan aborts that channel cleanly and continues to the next instead of silently fail-safing all videos as long-form.
All commands support `--force` to regenerate existing output files. Gemini-calling commands (scan, mindmap, transcript, concepts, process) accept `--model` / `-m` at the top level to override the config.yaml model. Precedence: CLI flag > config.yaml > `DEFAULT_MODEL` constant. `MAX_OUTPUT_TOKENS = 65536` caps Gemini output (matches `translate_video.py`). Default log level is `info` (visible progress without extra flags).
**Per-subsystem guardrails live in `.claude/rules/*.md` and load when a session READS a matching path.** The mechanism was verified live on this machine (2026-07-25: reading `scripts/burst_report.py` injected `.claude/rules/intelligence-layer.md` in full), and `tests/test_guardrail_inventory.py` keeps every guardrail resolvable so none can silently stop loading. **Those files are binding, exactly like the ones below.**
| Rule file | Loads when you touch |
|---|---|
| [`transcript.md`](.claude/rules/transcript.md) | the transcript call, chunking, quality assessment, captions failover, any transcript writer |
| [`scan-config.md`](.claude/rules/scan-config.md) | `config.yaml` reading, scan channel gates, manual `--url` resolution, dedupe/prune-shorts |
| [`search-index.md`](.claude/rules/search-index.md) | the LanceDB index, hybrid/concept search, concepts to taxonomy, topics, nugget |
| [`evals.md`](.claude/rules/evals.md) | the retrieval eval, model scorecards, model choice, the mindmap prompt |
| [`briefings.md`](.claude/rules/briefings.md) | briefings, the headline digest, the personalization profile |
| [`docs-currency.md`](.claude/rules/docs-currency.md) | README, INSTALLATION, `docs/`, the SKILL.md files |
| [`intelligence-layer.md`](.claude/rules/intelligence-layer.md) | `intel_graph.py`, `lead_lag_report.py`, `burst_report.py`, `sdsm_network.py`, `wiki_atlas.py` |
| [`translate-bcs.md`](.claude/rules/translate-bcs.md) | `translate_video.py`, its prompts, its tests (operationally separate) |
**A reviewer that does not auto-load them must be handed the matching file.** Codex reads `AGENTS.md` and `CLAUDE.md` only, never `.claude/rules/`, so a peer-review brief has to name the rule file for the subsystem under review or the pass silently loses every guardrail that governs it.
**Prompt templates:** `prompts/*.md` — self-contained, referenced by name (without extension) in `config.yaml`:
- `mindmap-knowledge` — thematic mind map with domain terminology + timestamps (default)
- `mindmap-light` — fast scan, 4-6 branches
- `mindmap-heavy` — comprehensive, 6-10 branches with resources/perspectives
- `transcript` — three-task decoupled prompt returning structured JSON
- `mindmap-from-transcript` — text-input mind map prompt (issue #54). Used when the resolver picks `source="transcript"`. Output structure matches `mindmap-knowledge` so downstream concepts extraction is unchanged.
- `concepts` — concept extraction + normalization against taxonomy, with `{{taxonomy}}` template slot
- `topic-digest` - cross-source digest of many short sources about one topic, ranked, fixed sections. A plain default meant to be overridden through `prompt_dirs`, see [`docs/reading-layer.md`](docs/reading-layer.md)
- `cliffnotes-distiller` - one long transcript to a deep-linked reference. A plain default meant to be overridden through `prompt_dirs`, see [`docs/reading-layer.md`](docs/reading-layer.md)
- `translate-bcs` — BCS subtitle translation system prompt for the **video-understanding fallback** (used by `translate_video.py` when no captions are available)
- `translate-bcs-from-srt` — BCS translation prompt for the **captions-first path** (text-in / text-out, preserves `[HH:MM:SS]` prefixes, optional `{{AUTO_GEN_NOTE}}` cleanup slot for auto-generated tracks)
`resolve_prompt_path` stays the ONE place a name becomes a path, and it now searches each `prompt_dirs:` config entry, then each path in `$VIDEO_INTEL_PROMPT_DIR`, then the bundled `prompts/` last - first file found wins, so a private copy of a shipped template overrides it without editing the repo and a checkout with neither set resolves exactly as before. `load_prompt` keeps its own exists-check and `sys.exit(1)`, with the error naming every directory searched. Entries must be absolute paths; a leading `~` is expanded (a relative one is ignored with a WARNING and is never resolved against the CWD - a prompt whose meaning depends on the shell's working directory is worse than no override, and an entry naming a file rather than a folder is dropped the same way). A directory that does not exist is allowed and skipped, so one shared config can name a folder only some machines have - but a prompt name that falls back to the bundled template while override directories are configured gets ONE log line per name per process: INFO when the name has never resolved from an override in this process (the routine case), escalating to WARNING when it previously did and now doesn't, so a typo'd filename never silently sends the shipped default to Gemini and an operator who overrides only a few templates does not get warned about every other one on every run. `PROMPT_DIRS` is populated by `load_config` alone, on every path that returns a config, and reset first so an absent key cannot inherit the previous load's value. Reviewers: a diff that re-derives `SKILL_DIR / "prompts" / f"{name}.md"` anywhere outside `resolve_prompt_path` reopens the PR #136 checker/writer-path-drift class that `tests/test_channel_knob_preflight.py::TestResolvePromptPathMatchesLoadPrompt` and `tests/test_prompt_dirs.py::TestLoadPromptAgreesWithTheResolver` exist to pin. `scripts/translate_video.py` is operationally separate and is deliberately NOT wired into this.
**Config:** `config.yaml` — channels, output directory, model, parallelism, per-channel prompt/since overrides.
- `vector_db_dir` (optional): path for the LanceDB vector index. Defaults to `output_dir / .lancedb`. Must be on a real local filesystem — cloud-synced mounts (Google Drive File Stream, OneDrive, Dropbox) do not support the atomic file operations LanceDB needs to commit its MVCC manifests. The `index` command runs a pre-flight probe (`probe_atomic_writes`) that does a throwaway LanceDB connect + create + drop round-trip against the target path; if that round-trip fails, the command aborts with an actionable diagnostic *before* any Voyage embedding call, saving the user from paying for embeddings on a write that cannot succeed. The vector index is a derived artifact (rebuildable from transcripts via `index`), so it is safe to live in a local cache directory (e.g., `~/.cache/video-intel/lancedb`) outside a cloud-synced `output_dir`. See [ADR-0016](docs/adr/ADR-0016-vector-db-path-config.md). Related but **accepted, not code-fixed** (issue #67): on the same cloud mounts, `.meta.json` reads can be stale (read-after-write), so a `scan` can occasionally re-queue an already-done video. Unlike the index, `meta.json` is part of the portable corpus and intentionally stays on the mount; a defensive re-read would be false confidence (a stale read is indistinguishable from a genuine "not done"). The hazard is bounded to wasted re-transcription since #66 (identity always stamped) and #74 (hung transcript times out) — see `docs/troubleshooting.md` ("Cloud-mount stale meta reads").
- Per-channel `enabled: false` (added 2026-04-24) skips the channel from `scan` entirely — including explicit `scan --channel <name>` invocations — but keeps the channel addressable for `mindmap --url --channel <name>`, `transcript --url --channel <name>`, `mindmap --file` / `transcript --file` / `process --file`, and `concepts --channel <name>`. Use this for non-scannable sources: Skool communities (no YouTube API metadata), Vimeo and other platforms the URL parser does not understand, members-only YouTube that 403s from Gemini, and one-off creators whose feed is mostly off-topic. The flag is opt-in (default true) and strict — overriding it on the command line is deliberately unsupported. See [`docs/solutions/integration-issues/non-scannable-sources-enabled-flag-20260424.md`](docs/solutions/integration-issues/non-scannable-sources-enabled-flag-20260424.md) for the full pattern and `tests/test_channel_enabled_flag.py` for the contract.
**Idempotency:** `is_processed()` checks for existing output files by `{date}-{slug}.{mode}.md` naming. Re-running scan safely skips already-processed videos. All commands support `--force` to regenerate.
**Output goes to** `~/video-intel/{channel_name}/` (configurable via `output_dir`), not into this repo. Master `taxonomy.json` lives at the output root.
**Concept layer:** Per-video `concepts.json` is the source of truth. `taxonomy.json` is derived (rebuilt by `taxonomy-build`). During batch extraction, new concepts accumulate in memory so each video normalizes against concepts discovered in earlier videos. See ADR-0010.
**Search internals:** Score math, pipeline mechanics, tuning levers, and empirical observations are documented in [`docs/search-internals.md`](docs/search-internals.md). Read this before modifying search behavior.
**Testing and eval framework:** [`docs/testing.md`](docs/testing.md) is the operational reference for all three suites — unit/integration in `tests/`, the free measurability audit at `tests/evals/test_instrument.py`, and the grounded-golden-dataset retrieval eval at `tests/evals/test_search_quality.py`. As of 2026-09-03 the hybrid-search eval baseline is 1/25 (Q11) on an 89,992-chunk index, measured after a full `index --force` rebuild that followed a 41-video catch-up scan; the instrument was 50/50 both before and after that rebuild. The N/25 has now held at 1/25 across three successive index states with no retrieval-logic change, which is the expected result and is the reason a corpus-growth re-index does not need its own ADR. Earlier reference points: 1/25 on the 2,360-video / 85,854-chunk index of 2026-09-02 (after the #195 remediation re-index of 10 channels), 1/25 on the pre-remediation 80,297-chunk index, and 0/25 on the same corpus before issue #190 fixed the instrument; none of these are comparable to the 2026-04-19 1/25. Any PR that touches retrieval logic must re-run **`pytest tests/evals/test_search_quality.py`** and record the new N/25 in the description — run that module by name, not the whole `tests/evals/` directory, or `test_instrument.py`'s deliberate failures land in the same summary line and the N/25 is no longer derivable. The golden dataset at `tests/evals/golden_dataset.yaml` is a frozen contract per [ADR-0017](docs/adr/ADR-0017-kb-layer-strategy.md) — edits need ADR-grade justification.
**KB-layer direction:** [`ADR-0017`](docs/adr/ADR-0017-kb-layer-strategy.md) established the original staged approach (query expansion → LightRAG → LLM Wiki). Stage 1 (query expansion) shipped 2026-04-20 and did not move the 1/25 baseline. [`ADR-0018`](docs/adr/ADR-0018-nugget-cli-cross-creator-synthesis.md) (2026-04-22) shipped the `nugget` CLI for cross-creator synthesis and **deferred Stage 2 LightRAG behind three named signals** — until any of those fires, LightRAG is not the next step. The **active forward direction** is captured in [`docs/brainstorms/2026-05-28-intelligence-layer-roadmap.md`](docs/brainstorms/2026-05-28-intelligence-layer-roadmap.md): a structured intelligence layer (DuckDB 6-node / 6-edge starter schema + bidirectional Displacement/Magnet lenses + 6-tuple stance schema in `prompts/concepts.md`), justified by the observation that the user-experienced gap is *aggregate / contrastive / polarity-flipped* queries, not better retrieval. Phase 0a shipped via PR #62; Phase 1.5 spike is tracked as [#63](https://github.com/dzivkovi/video-intel/issues/63). Cognee remains rejected. The 2026-04-16 work-notes (`work/2026-04-16/03-architecture-futures-cognee-lightrag-llm-wiki.md` and `04-knowledge-layer-options-brainstorm.md`) plus the 2026-04-28 7-agent synthesis (`work/2026-04-28/03-knowledge-discovery-synthesis-paths-forward.md`) are the historical context.
## Key Design Decisions
- Gemini is a multimodal proxy, not a competing assistant. Video understanding requires vision+audio that Claude doesn't have via API.
- The transcript prompt requests structured JSON with three parallel tasks (diarization, screen content, speaker ID). `merge_transcript_json()` fuses them by timestamp sort.
- `SKILL_DIR` is resolved from the script's own path (`Path(__file__).resolve().parent.parent`), making the skill relocatable across `~/.claude/skills/`, `~/.gemini/skills/`, or `~/.agents/skills/`.
- Lazy imports (`require_gemini()`, `require_youtube()`) in `gemini_common.py` give clear error messages when dependencies are missing instead of cryptic ImportErrors.
## Packaging, Distribution, and Release
Plugin distribution model, the release checklist, and the migration note for users on the pre-plugin layout live in the `release` skill: [`.claude/skills/release/SKILL.md`](.claude/skills/release/SKILL.md).
## Development
```bash
# Install dev dependencies
pip install -e ".[dev]"
# Run unit/integration tests
pytest tests/ -v --ignore=tests/evals
# Run tests with coverage
pytest --cov=scripts --cov-report=term-missing -v --ignore=tests/evals
# Is the ruler intact? Free - no Voyage call. Run this before reading any N/25.
pytest tests/evals/test_instrument.py -v
# Run retrieval eval (requires pip install deepeval, VOYAGE_API_KEY, built LanceDB index)
# Name the module, not the directory: the audit's deliberate failures would
# otherwise share the summary line and the N/25 stops being derivable.
pytest tests/evals/test_search_quality.py -v -s
# Lint and format
ruff format .
ruff check . --fix
```
Config in `pyproject.toml`. Run ruff before declaring any task complete.
## Workflows
This project uses the [Compound Engineering plugin](https://github.com/EveryInc/compound-engineering-plugin/) for structured workflows:
- `/workflows:work` — Execute tasks with progress tracking
- `/workflows:review` — Code review with multi-agent analysis
- `/workflows:compound` — Document solved problems (produces `docs/solutions/` entries)
Session plans are stored in `plans/` (configured via `.claude/settings.json`). Plans are session artifacts — historical, not living docs.
Solved problems are recorded in `docs/solutions/` (categories `integration-issues/`, `process-issues/`, `workflow-issues/`; each entry carries YAML frontmatter: `title`, `date`, `category`, `tags`, `components`, `severity`, `symptoms`, `root_cause`, so it is searchable by field when working in an area it already covers), following the three-bucket rule (living / historical / decision records).
After a solved, verified problem, offer once to invoke the `ce-compound` skill at the completion checkpoint only when the work produced durable project reasoning that is not readily recoverable from the final code, tests, types, comments, or existing documentation, and losing it would plausibly cause recurrence, material risk, or substantial rediscovery. Apply this counterfactual: if the learning document disappeared, would a future engineer reading the final implementation still be likely to repeat the mistake or redo substantial investigation? If not, do not offer. Completion, effort, and diff size alone are not enough. Offer at the checkpoint so a qualifying learning can ship in the PR that produced it, and only where the repository treats captured learnings as tracked, committed knowledge.
## Compounding operational recoveries (durability ladder)
When an operator (human or agent) recovers from a stuck state during a scan - a hang, a new failure mode, an undiscovered video - that recovery only compounds if it lands in a durable layer. A recovery that lives only in a chat session is re-improvised next time. After **every** such recovery, run it through one gate:
> **Could a future scan hit this same failure with no human noticing?**
> - **Yes → it must become CODE** - a runtime signal (`exception type`, status code, timeout expiry, empty output, known error substring) triggers automatic recovery. Strongest compounding; the reference implementations are the captions failover (#60), the `prompt=0` confab guard (#60), the pre-flight premiere filter (#70), and the per-transcript timeout (#74).
> - **No** (the right response needs operator judgment, or it is genuinely one-off) **→ a `docs/troubleshooting.md` row** is enough.
Sharper heuristics: *diagnosed in >10 min → at least a troubleshooting row; recurred across sessions → code.* And the decisive test for code-vs-doc: **"Would a correct automated response require knowing something only the operator knows?"** If yes, doc. If no, code. Avoid both anti-patterns - do not document (and re-improvise forever) a recovery whose signal is computable, and do not write 50 lines of code for a failure that has happened once.
`docs/troubleshooting.md` is the registry: every failure mode is a row with a **Status** of `auto (#PR)` or `manual (...)`. A `manual` row with a planned code fix is a visible debt marker; flip it to `auto` and link the PR when the fix ships. When you ship a code-layer recovery, add its review guardrail in the **same** PR - in the matching `.claude/rules/*.md` file, or in this file if it is genuinely cross-cutting - so a future change cannot silently remove it.
## Code Review Guardrails
Rules for review agents (auto-selected by `/ce-code-review`) and for anyone cutting a PR. These are the non-obvious checks - the general "does it work, does it have tests" bar is assumed.
**What lives here versus in a rule file.** This section holds the cross-cutting principles that per-subsystem guardrails cite, plus any rule whose governing session may read no matching repo file at all (a corpus Q&A session, a briefing that renders deep links, hand corpus maintenance, PR authoring, a review agent reading `docs/plans/`). Everything path-specific lives in `.claude/rules/*.md` - see the table above.
**Six principles are cited by tag rather than restated.** Each canonical statement below opens with a bracket tag. A rule file cites that tag inline instead of re-deriving why the principle holds, and keeps only the case-specific what. Grep the citations with `rg 'core:' .claude/rules`. The six tags are `writer's-path`, `real-caller`, `identity #66`, `one-definition`, `probe-before-pay` and `#124 read-guard`; `tests/test_guardrail_inventory.py` fails on a citation that names none of them.
- **Bounded retries only.** The `transcript` path tries one JSON parse, then `isolate_json()`, then `salvage_transcript_sections()`, then one bounded retry if salvage fails. Do not promote this to an unbounded loop. Partial writes plus a `.transcript.raw.txt` sidecar are the designed failure mode, not something to "fix."
- **[core: #124 read-guard] Writer-side meta reads go through `_read_meta_best_effort`, never a bare `json.loads`, and the two failure classes stay apart (issue #124).** Every path that merges into an existing `meta.json` sits inside, or feeds, an exception handler whose job is to RECORD a failure; a read raising from inside that handler masks the error it was preserving. Verified on the real handler: a corrupt meta plus a failing Gemini call used to raise `JSONDecodeError`, `UnicodeDecodeError`, or `AttributeError` and the Gemini error was lost; all five corruption shapes now record it. Four things reviewers must protect:
1. **The catch is `(ValueError, OSError)`, never `(json.JSONDecodeError, OSError)`.** `UnicodeDecodeError` subclasses `ValueError`, NOT `OSError`, and a write torn mid-multibyte-character is the normal shape of a truncated write on a corpus with Cyrillic/BCS titles. This file has been bitten by exactly that twice before (`_load_video_id_index`, `load_interest_model`); the first attempt at this very fix shipped the narrow tuple and left the headline case open.
2. **Unusable CONTENT and a failed READ are different, and conflating them trades one bug for a worse one.** Content we read but cannot use is quarantined to a `.meta.corrupt.json` sidecar and replaced. An `OSError` means the bytes may be perfectly intact, so `update_meta` (the shared SUCCESS-path writer) passes `raise_on_os_error=True` and propagates: overwriting there would destroy `alt_titles` (title-rotation history exists nowhere else) and `skip_modes`/`skip_reason` (the operator's deliberate stage suppression, issue #42). Error-path callers pass `False`, because there the alternative is destroying the error being recorded. A diff that defaults the keyword, or hands `True` to an error handler, breaks one half or the other.
3. **Read bytes, then decode+parse.** `read_text()` raises `UnicodeDecodeError` from the call that is supposed to only be able to fail at the I/O layer, folding the two classes back together.
4. **Every writer still stamps identity.** A `{}` read means the fields dict is the whole file, so a writer supplying only `{"processed": ...}` would leave an identity-less meta that `_load_video_id_index` skips - re-queueing the video for a full re-transcribe (issue #66). The concepts writer and `cmd_mark_skip` were fixed for this in the same PR.
5. **The quarantine saves the bytes the helper ALREADY read, never a re-read**, and never overwrites an existing sidecar. Re-reading opens a race where a concurrent healthy writer's fresh bytes get quarantined and then overwritten by the caller - the recovery mechanism becoming the data loss it exists to prevent.
Call sites: `update_meta`, `process_mindmap`'s `except` handler, `process_transcript`'s parse-failure writer, `_record_transcript_error`, `cmd_process --file`'s upload-failure handler and both of its identity reads, `cmd_mindmap --file`, `cmd_transcript --file`, `cmd_mark_skip`, and the mindmap `transcript_status` provenance read. Reviewers: a new `json.loads(meta_path.read_text(...))` anywhere on a writer or error path needs pushback. Reader-side consumers that WANT strictness are deliberately excluded - `_load_video_id_index` must not invent identity from a damaged file. Test contract: `tests/test_meta_best_effort.py`.
- **[core: writer's-path] A verifier must use the WRITER's path, never re-derive its own - and the test that proves it cannot be a stub test.** Whenever code is added that *checks whether an artifact exists* (an exit-code gate, an idempotency probe, a completeness sweep), it has to obtain the path from the same rule the writer used. PR #136 shipped one such check and three separate defects fell out of it, all the same mistake: the exit check built `channel_dir / f"{prefix}.concepts.json"` while `process_concepts` hardcodes `output_dir / channel_name`; and `_cmd_process_url` looked under the `video_id`-indexed prefix while `process_mindmap`, given no explicit `prefix=`, wrote under the title-computed one. Each produced a **permanent false failure on a fully successful run**, where the logged recovery ("re-run to fill the gap") can never fill it - the worst possible outcome for a guard whose entire value is being believed, because it trains the operator to ignore the signal. The two sides drift for real reasons: identity resolvers, title rotation, and `channel_dir_override` all move the writer's destination without touching the checker. So: pass an explicit `prefix=` / `channel_dir_override=` to the writer and reuse that same value in the check, or have the writer return the path it used. **The second half is not optional.** Every stub-based test hands the artifact path to the stub, so writer and checker agree by construction and cannot disagree - PR #136's suite was green at 1537 passing with all three defects live, and both reviewers who found them did so by EXECUTING a repro, not by reading the diff. A test only covers this class if it derives the writer's real destination and the checker's expectation **independently and then compares**; the shape to copy is `tests/test_transport_retry_and_partial_exit.py::TestCheckerAndWriterAgreeOnPaths`. Reviewers: any new `path.exists()` check on a per-video artifact whose path is rebuilt from parts, rather than taken from the writer, needs pushback.
- **[core: real-caller] A guard test must see what it guards, and it is falsified before it ships.** Four shapes have each let a green suite hide a live defect in this repo. **(a) A stub agrees with its own assertion**: hand the stub the same path or value the assertion checks and writer and checker cannot disagree - PR #136 was green at 1537 passing with three defects live, and both reviewers who found them EXECUTED a repro rather than reading the diff. **(b) A helper is unit-tested but never proven CALLED**: delete the call and watch the suite stay green (#203's `drop_topic_leaks`). **(c) A source walk or regex guard with no companion proving the walk finds anything** - the tautology class, which has now appeared in #182, #195, #204 four times, #205 and #213; a source COUNT standing in for caller-level coverage is the same sin (#205 item 6, #213 item 1b). **(d) `pytest.raises(SystemExit)` where the exit CODE is the claim** (#185): it is satisfied by `SystemExit(0)` exactly as happily. **The rule:** drive the real command with only the paid boundary stubbed, assert on content or ordering (a paid call NOT made) rather than on a bare exception, and before shipping neuter the guard and confirm exactly the relevant test fails.
- **[core: one-definition] A decision has ONE definition and N consumers.** Wherever two call sites need the same answer - a path, a boundary regex, a severity verdict, a channel lookup, a prompt fallback, a config reader, a topic-slug resolver, a SQL literal escape - they call one helper, and a drift guard walks the module for private copies. This is not a style preference: **every duplication found here had already drifted by the time it was found.** #152 `timestamp_utils`, #159 `transcript_quality_flags_are_severe`, #176 `accumulate_concepts_into_taxonomy`, #183 `escape_sql_string_literal`, #188 `resolve_topic_filter`, #195 `ENTRY_TIMESTAMP_PATTERN`, #203 `drop_topic_leaks`, #205 `match_configured_channel`, #210 `DEFAULT_PROMPT_NAME`, #213 `configured_channels`, #221 `resolve_prompt_path`, #115 `compile_interest_model`. Reviewers: a diff that adds a private copy of an existing helper, or re-derives its output inline, needs pushback. A helper that does not quite fit is extended, not forked.
- **Timestamp helpers live in `scripts/timestamp_utils.py`, and that module is the import-weight firebreak (issue #152).** Do not duplicate `normalize_timestamp`, `normalize_mm_ss_zero_timestamp`, `should_reinterpret_part_as_mm_ss_zero`, `timestamp_tolerance`, `parse_time_to_seconds`, or `timestamped_url` into other scripts. The last two moved here from `video_intel.py` and `intel_graph.py` respectively; both original homes keep a re-export so every existing importer and `tests/test_utils.py` / `tests/test_intel_graph.py` still resolve them from the old module. **`timestamp_utils.py` must never gain a dependency beyond the standard library.** It is what lets the standalone read-only analytics scripts (`wiki_concepts`, `wiki_atlas`, `lead_lag_report`, `lead_lag_viz`, `burst_report`) import without dragging in the curate stack: each of them imports `timestamped_url` from HERE, never from `intel_graph`, because `intel_graph` imports `video_intel`, which hard-imports `googleapiclient` at module level. Repointing one of those five back at `intel_graph` silently re-couples that script to the full curate stack - measured before #152, all five failed to import at all when `googleapiclient` was absent. Test contract: `tests/test_timestamp_utils.py::TestStandaloneScriptImportIsolation` (subprocess, heavy modules blocked via `sys.meta_path`). `_classify_and_offset_timestamp` in `scripts/video_intel.py` MUST call `normalize_timestamp` before classification — issue #58 caught the regression when chunk-3 of a 2h+ Tucker transcript produced 89 `Implausible timestamp [100:XX:XX]` warnings because PR #51 ported the classifier but skipped the normalize pre-pass. Reviewers: grep for these six names in any new diff. A definition outside `scripts/timestamp_utils.py` needs pushback; the only legitimate non-definition hits are the re-export lines in `scripts/video_intel.py` (`parse_time_to_seconds`) and `scripts/intel_graph.py` (`timestamped_url`), plus `scripts/translate_video.py`, which shares the normalization helpers and is operationally separate.
- **[core: probe-before-pay] Probe before you pay.** Any check that can fail before a paid call - a Gemini upload or text call, a Voyage embed, a YouTube quota unit - runs BEFORE it, not after. The instances, each of which cost something before it became a rule: `probe_atomic_writes()` before any Voyage embedding call in `build_search_index` (~$0.30 per failed run, [ADR-0016](docs/adr/ADR-0016-vector-db-path-config.md)); the `chunk_minutes` and file-size guards before `upload_local_video` on `process --file` (#168, #185 - a one-character config typo used to buy a full multi-minute MP4 upload); the scoped-index no-index refusal above the Voyage call (#183); the unknown topic slug before `hybrid_search` (#188); the YouTube-shape check before `get_channel_id` (#113, #205). **A test of that ordering asserts the paid call was never made, never just the exit code** - an exit-code-only assertion passes whether the guard runs before or after the spend [core: real-caller]. Reviewers: grep for `probe_atomic_writes` in any diff touching `build_search_index` or the `index` CLI, and for `upload_local_video` in any diff touching `process --file`.
- **Timestamps are data, not decoration.** Every retrieved chunk carries `timestamp_seconds`, surfaced as `&t=<seconds>` in result URLs. Changes to chunking, dedup, or rendering that drop or corrupt that field break user-visible behavior. Reviewers: grep for `timestamp_seconds` in diffs touching `hybrid_search`, `_select_hits_by_video`, or chunk rendering.
- **Skill-parity: same diff, not follow-up.** When a PR adds a new CLI subcommand or flag to `video_intel.py` or `translate_video.py`, the matching `SKILL.md` entry (natural-language routing) lives in the same PR. "I'll update the skill separately" is a regression — the plugin's skill surface drifts from its CLI surface and users can't reach the new capability through the skill.
- **Paraphrase verification uses `search --vector`, not `Grep`.** The speaker's vocabulary almost never matches a paraphrase verbatim; keyword search returns false negatives. A user prompt like "verify whether [creator] said [X]" or "is this quote real" routes to the `video-intel-search` skill's hybrid-search command. Reviewers: grep for direct corpus `Grep`/`Read` calls in any verification-shaped session - those are bugs.
- **Video id is the identity, slug is decoration.** Any code that dedups, idempotency-checks, or looks up per-video artifacts must key on `video_id` (with slug as fallback for legacy files missing meta.json). The 2026-04-22 title-rotation dedup shipped `_load_video_id_index()` and a `dedupe` subcommand precisely because slug-only checks missed A/B-tested titles. Reviewers: grep for `video_file_prefix` or `is_processed` in diffs touching scan/transcript/concepts — any new path that treats slug as identity needs pushback.
- **[core: identity #66] Every transcript meta write stamps full identity.** Issue #66: the single-shot and captions transcript writers used to persist `{processed, transcript_status, ...}` with no `video_id`, and since the transcript loop is the first writer (inverted ordering #54) it left identity-less metas that `_load_video_id_index` skips — breaking idempotency (re-transcribed every scan; a re-queued hang then froze the scan). All three transcript meta writers (single-shot success, salvage, captions failover) MUST merge `_transcript_identity_fields(video, channel_dir)` — `{video_url, video_id, channel, title, published}`, matching the chunked path's `meta_fields` shape. Reviewers: any `update_meta(..., "transcript")` call in `process_transcript`/`_try_captions_transcript` that does not spread `_transcript_identity_fields` is a regression. `repair-metas` backfills pre-existing identity-less metas from the `.transcript.md` header (Source URL → `video_id`); it must never overwrite an existing field or fabricate identity for a non-YouTube source. Test contract: `tests/test_identity_meta.py`.
- **Out of scope for cleanup flags.** `docs/plans/*.md`, `docs/solutions/*.md`, `docs/reports/**` (markdown and generated HTML), `work/**/*`, and the root `plans/` directory are living or historical session artifacts. Review agents must not flag them for deletion, rewriting, or consolidation — that's the three-bucket rule at the end of the Workflows section.
- **Mindmap is the AI's discovery surface and must always run on `process --url`.** The Step 1 transcript call in `_cmd_process_url` is wrapped in try/except so an uncaught exception still lets Step 2 mindmap run with `source="video"` fallback. Reviewers: any diff that drops or narrows that try/except (or removes the test `test_mindmap_still_runs_when_transcript_step_raises`) regresses the user's "mindmap always runs" invariant.