process-watch-report · git:20260924.2e23a38 · 2026-09-24 · sha256 10d8d5f86098c4ea
process-watch-report git:20260924.2e23a38C
Immutable. This exact content never changes and is served at /api/v1/blob/10d8d5f86098c4ea.
---
name: process-watch-report
description: Compare state/watch/* (what the daily watcher discovered) against state/ingest/* (what each downstream skill last ingested) and invoke every skill whose mapped sources have changed since its last successful run, timing every step into an append-only trace, then committing the result and following up with /upload-watch-changes to publish it — unless the run hit a major issue, in which case the manifest is left pending and the operator is asked. Use when the user says "process today's watch report", "sync data based on the watcher", "refresh everything that changed", "run the right skills for what changed", or otherwise asks to act on what the daily watcher found. Robust to multi-day gaps between orchestrator runs — never misses an intermediate-day change.
allowed-tools:
- Read
- Bash
- Skill
---
# Process watch report (orchestrator skill)
The daily watcher in `scripts/watch/` writes per-source state to `state/watch/<source>.json` (with a `lastChanged` ISO timestamp) and also produces a human-readable daily report under `data-reports/`. This orchestrator decides what to ingest by comparing watcher state against per-skill ingest markers under `state/ingest/<skill>.json` (with a `lastSuccessfulIngest` timestamp).
Why state-driven, not report-driven? The watcher's `## Changed` section in any one report is relative to the previous watcher run, not the last successful ingest. If a source changed on Monday but the orchestrator wasn't run until Wednesday, Wednesday's report only lists Wednesday's changes — Monday's would be silently missed. State files are the durable truth: `state/watch/<source>.json.lastChanged` is "when did this source last actually move?" and `state/ingest/<skill>.json.lastSuccessfulIngest` is "when did we last ingest that move?". The comparison is invariant under any number of skipped days.
The human-readable reports under `data-reports/` are still useful for narration in the final summary — but they're no longer the decision input.
## Unattended mode (the nightly routine)
When the invocation args contain **`unattended`** (the nightly "Electionsbg watch" routine
passes it), nobody is at the keyboard, and a run that stops to ask a question simply stalls
until morning. The operator has approved in advance: every recommended ingest in the plan,
**and** the publish that follows it. So in unattended mode:
- **Step 3 does not wait.** Print the plan exactly as usual, including the publish line, then
go ahead. Treat it as though the operator answered "yes, run all".
- **Tell every child skill it is unattended.** Pass this in each `Skill` call's args:
`"unattended nightly run from /process-watch-report — do not ask; take the documented
default at every prompt; skip any step that needs human judgment and report it"`. The step-14
hand-off to `/upload-watch-changes` keeps its "proceed without re-asking" phrase.
- **Human-judgment steps are skipped and reported, never guessed.** Examples: `update-polls`
operator review/accept for agencies with no extractor, the manual-paste and manual-download
sources (Step 0, TI CPI, municipal fiscal, capital-programme fetches), `enrich-open-calls`
promotion, typo-override decisions in `update-connections`, and regulator-roster re-checks.
If a skill's automated part succeeded and only a human step was skipped, it counts as
`partial`: stamp it only if that skill's own rules say the automated part is complete.
Otherwise leave it unstamped so the next run queues it again. List every skipped step under a
**Needs operator** heading in the summary.
- **A failed skill → option (a), "skip and continue", chosen automatically.** Do not halt and
ask (Data-integrity contract #2). Record the failure, leave that skill unstamped, and run the
rest of the queue. This is still a MAJOR ISSUE under step 12, so the publish is **held**:
the ingest is committed and the manifest stays pending for the operator. Unattended mode
approves ingests the operator would have approved. It does not approve publishing a corpus
the run could not vouch for.
- **Every other gate still applies unchanged:** stamp only on success, the step-11 commit by
explicit pathspec, no push, and no `bucket:sync` / `:cloud` loader outside
`/upload-watch-changes`.
- **End with a notification when one is available.** If the `PushNotification` tool is present,
send one line: skills run, ok/partial/error counts, whether the publish went out or was held
(and why), and how many "Needs operator" items there are. The step-13 summary stays the full
record.
Without `unattended` in the args, nothing here applies. The skill confirms at step 3 and halts
on failure, as before.
## Inputs
- `state/watch/<source>.json` for each watcher source — the truth about what's changed when. Always exists once the watcher has run at least once.
- `state/ingest/<skill>.json` for each tier-2 skill that has run successfully under this orchestrator. May be missing on first run after the state-driven migration — see "Bootstrap" below.
- `data-reports/<YYYY-MM-DD>.md` reports — read for narration in the final summary, not for decision-making.
## Source → skill mapping
The "Changed" section of the report contains a bulleted list. Each bullet's label maps to a downstream skill:
| Source label in the report (label prefix is enough) | Skill to invoke |
| --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `Parliament roll-call votes` | `update-rollcall` |
| `Parliament MPs` (active roster) | `parliament-scrape` (Step 5 — also re-runs the local-elections MP-link decorator via `npx tsx scripts/parsers_local/decorate_local_mp_links.ts` so any newly-seated MPs' photos surface on the local-election mayor / council rows they ran on; Step 6 — also re-runs `npx tsx scripts/preferences/rebuild_resolved.ts` so the precomputed `/candidate/*` resolution shards re-match against the refreshed index, then `bucket:sync` since those shards are gitignored; **Step 7 — reload PG: the roster/profiles/avatars are Cloud-SQL-served since persons-pg-retirement-v1 T2.2/T2.3/T2.4, so run `npm run db:load:mp-roster:pg` locally + `db:load:mp-roster:pg:cloud` to prod — regenerating `parliament/{index.json,profiles/,avatars.json}` alone reaches nothing live; those files are excluded from `bucket:sync`**) |
| `Polls —` (Тренд, Алфа Рисърч, Маркет ЛИНКС, Сова Харис, Мяра, Глобал Метрикс, Галъп, press discovery — eight `polls_*` sources share this prefix) | `update-polls` — the PRIMARY sources (docs/plans/polls-agency-watchers-v1.md decisions 1-2): each agency's own publication. `BG Wikipedia polls` below is cross-check only. `update-polls` is rewritten for this pipeline (T3.1) and runs `polls:fetch` → `polls:extract` → operator review → `polls:accept` end to end for Тренд/Алфа Рисърч. ⚠️ **The other five agencies (Маркет ЛИНКС, Сова Харис, Мяра, Глобал Метрикс, Галъп) and every press-discovery capture still lack a built extractor** — `polls:fetch` captures them fine, but `polls:extract` has no rule for them yet (§6.2, Tier 2c), so a flip there still needs an operator to read the raw capture and hand-write the draft (or wait for that agency's extractor) — see the `polls_trend` row below for the exact TR/AR-only boundary. |
| `BG Wikipedia polls` | `update-polls` |
| `BG Wikipedia governments list` | _manual edit required — see "Governments: manual edit first" below_ |
| `Сметна палата declarations — MPs` | `update-connections` |
| `Сметна палата declarations — executive (officials)` | `update-officials` (also re-runs the `update-funds` political-economy join step — `npx tsx scripts/funds/political_links.ts` — since the officials → company links are an input. **[2026-08-21] those links are `company_politicians` at `kind='official'`, not a file this skill writes** — so sequence `db:resolve:persons` → `db:load:tr:pg` first, or the join republishes the previous vintage at exit 0) |
| `Сметна палата declarations — municipal (mayors & councillors)` | `update-officials` (Step 1b — municipal ingest; same political-economy follow-up applies) |
| `data.egov.bg Commerce Registry` | `tr-daily-refresh` (`npm run tr:daily-refresh`) |
| `EC Financial Transparency System` | `db:load:ngo-funding:pg` (`npm run db:load:ngo-funding:pg` — new FTS year: download `{YEAR}_FTS_dataset_en.xlsx` into `raw_data/ngo_funding/fts/`, then load into the `ngo_funding` table; DB-only, no JSON) |
| `data.egov.bg АОП` (procurement) | `update-procurement` (also re-runs the funds political-economy join — `npx tsx scripts/funds/political_links.ts` — since АОП award totals overlay the EU-funds political flags). **Procurement is served from Postgres (Cloud SQL), NOT GCS** — the whole `data/procurement/` tree (except `roads.json` + `derived/mp_party.json`) is **excluded from `bucket:sync`** and no longer gzip-shipped, so the run's ordinary `bucket:sync` (done anyway for funds/prices/etc.) already handles procurement's two GCS stragglers; there is no `bucket:sync:all` step for procurement. **Publish procurement via Postgres:** the skill's Step 2b runs `npm run db:refresh` for LOCAL PG (`contracts` + `tenders`, rebuilt from the fresh gitignored shards — no commit/sync); for **prod**, emit the Cloud SQL loaders in Next-steps (Procedure step 9): `npm run db:load:pg:cloud && npm run db:load:tenders:pg:cloud && npm run db:load:awarder-seats:pg:cloud`. (Both tables are wired into `db:refresh`; a migrated table left out of the refresh path silently goes stale — the tenders-stale bug, fixed 2026-07-02.) **Textbook market:** the CPV-22112 concentration overlay (`/awarder/000695114`) is built from this same corpus — after the re-ingest, re-run `gen_textbook_market.ts` + `bucket:sync:paths -- education` (see the `egov_procurement` row). |
| `АОП debarred-suppliers register` | `update-procurement` (Step 5 — debarred-list refresh; debarred flags on flagged EU-funds beneficiaries refresh via the same political_links step) |
| `КЗК procurement-appeals register` (жалби по ЗОП) | `update-kzk-appeals` — **an automated orchestrator step**; invoke it with the Skill tool like any other queued skill. It drives a _headed_ Playwright browser (a desktop window pops up for ~2 min) and needs Bulgarian egress (reg.cpc.bg 403s non-BG), so it is still **never** a CI/`--all` step — but on a developer machine in BG the orchestrator runs it unattended. **⚠ Pin the local DB** — `--apply` writes Postgres via `scripts/db/lib/pg.ts`, and any `DATABASE_URL` left exported by an earlier `db:*:cloud` command is _password-less by design_ and resolves its password from `.pgpass` (the CLOUD password), so it fails `28P01` against local PG. The skill's Step 0 covers this. **⚠ Assert 2,098 outcomes survive** every `--apply` (`select count(*) from kzk_appeals where outcome is not null`) — the ~2,098 rows with a NULL `decision_act_no` were produced interactively before the merits crawler existed and remain irreplaceable; never wipe `kzk_appeals.json` or TRUNCATE the table. (Rows WITH an act number are machine-derived by `npm run kzk:rejoin` and can be rebuilt.) The merits arm is now `npm run kzk:decisions` — a SEPARATE headed crawl this skill does not run. Stamp `state/ingest/kzk_appeals.json` on success. Surfaces on `/tenders/:unp`, the `/procurement` "Recent appeals (КЗК)" tile, and the `procurementAppeals` AI tool. **Publishing is NOT `bucket:sync` and NOT `db:dump`** — `procurement/` is excluded from the sync, and `db:dump` only `pg_dump`s _outward_ to GCS. See the skill's Step 4. |
| `КЗК decisions register` (решения и определения по ЗОП) | `update-kzk-appeals` — **but its STEP 1b, not a re-run of Step 1.** The two arms are different registers and fail independently: Step 1 crawls the INTAKE register and its `COALESCE(existing, EXCLUDED)` upserts are structurally incapable of writing an outcome, so re-running it on a `kzk_decisions` flip accomplishes nothing and reports success. Step 1b is three commands, and the crawl alone changes nothing served: `npm run kzk:decisions -- --year <YYYY> --apply` → `npm run db:load:kzk-decisions:pg` → `npm run kzk:rejoin -- --apply`. Headed Playwright + Bulgarian egress, same as the intake arm. **Run `--probe` first on a new machine** — the register's markup has never been read by committed code, and the probe also reports whether another `ot` value carries the определения register (the missing authoritative source for `suspension`). Stamp `state/ingest/kzk_decisions.json` — its OWN marker, deliberately separate from `kzk_appeals`, because a shared one is how this arm sat five weeks stale while the intake arm kept reporting success. Cloud: `npm run db:load:kzk-decisions:pg:cloud` then `npm run kzk:rejoin:cloud -- --apply`. Surfaces on `/tenders/:unp`, `/procurement/appeals`, the `procurementAppeals` AI tool, and — via `upheld_ocids` — the contract Corruption Risk Index. |
| `МОН регистър на институциите` (Tier R crosswalk + Tier B на awarder geo) | Re-crawl the ЕИК→EKATTE crosswalk that geo-resolves school/kindergarten procurement buyers: `npx tsx scripts/procurement/mon_ri_crawl.ts` → `data/procurement/derived/mon_ri_eik_crosswalk.json`, then `npx tsx scripts/procurement/awarder_geo_map.ts`. That rebuilds the override map; to reach `by_settlement` it needs an ingest, and it must be the **FULL current-value chain** from `update-procurement` — a bare `procurement:ingest` recomputes `amountEur = toEur(amount)` and silently drops the post-annex fold (~€1.75bn), at exit 0, with the euro-peg canary green. **Headed Playwright** (ri.mon.bg is Cloudflare-walled — a desktop window opens ~1 min; reachable from any egress, no BG requirement); **never** a CI/`--all` step. Fires rarely — EIK↔EKATTE is stable, so only flips when schools open/close. NOT a full `update-procurement` re-ingest — just the crawl + geo rebuild. Stamp `state/ingest/mon_ri_register.json`. See [[reference_mon_ri_register]] + `scripts/procurement/mon_ri_crawl.ts`. |
| `ИСУН EU funds` (beneficiaries) | `update-funds` |
| `ИСУН EU funds` (projects) | `update-funds` (Step 2 — contract-level ingest; runs `npm run funds:ingest-projects` after the beneficiaries ingest, then `npx tsx scripts/funds/build_geo_pins.ts` to rebuild the slim per-município files under `data/funds/projects/by-muni-geo/` — top-200 contracts by money, geocoded + non-geocoded — that back the My-Area "Проекти от еврофондовете" tile: a scrollable project list plus an on-demand Leaflet map of the geocoded subset.) |
| `keep.eu Interreg (programme re-imports)` | `update-funds` (Step 3c — the Interreg corpus. keep.eu re-imports whole programmes and its index exposes no `modified`, so this watches each programme's `date_of_data_import`; the report NAMES which moved. That is diagnosis, not scoping — the crawler has no programme filter, so the action is a ~2 h `--full` walk; the names tell you a re-import HAPPENED, which the id-descending index cannot. Only 11 of 22 programmes publish the date, so "no change" means "none of those 11 moved". **Interreg is NOT in ИСУН** — it runs on Jems — so the ИСУН steps above cannot pick it up. **PG-served**: publish via `db:load:interreg:pg:cloud`, NOT `bucket:sync`.) |
| `data.egov.bg бюджет` (budget execution) | `update-budget` (a December snapshot that closes a fiscal year also re-bases the tax-policy simulator — run `npx tsx scripts/budget/run_policy_baseline.ts` after the ingest; see "Policy-baseline refresh" below) |
| `Per-ministry execution reports` (програмен бюджет) | `update-budget` |
| `Доклад за състоянието на администрацията (IISDA)` | `update-budget` (resolve new file id first — see `iisda_doklad` describe-line) + `update-administration` (service-quality reparse + context rebuild) |
| `Административен регистър — услуги (IISDA)` | `update-administration` (re-scrape the ИИСДА services register → `services_overview.json`/`services_catalog.json`, reload the `admin_services` Postgres table) |
| `Eurostat e-government (isoc_ciegi_ac)` | `update-administration` (refresh `data/administration/egov.json` — BG e-gov use vs the EU) |
| `Eurostat digital skills (isoc_sk_dskl_i21)` | `update-administration` (refresh `data/administration/digital_skills.json` — BG citizen digital skills vs the EU) |
| `Агенция "Митници" — Митническа хроника` annual reports | `update-budget` (revenue-breakdown sub-step — runs `scripts/budget/run_customs_revenue.ts`, **then `run_policy_baseline.ts`** since the per-product excise levers read this customs split; see "Revenue-breakdown ingest" below) |
| `НАП — Годишен отчет за дейността` | `update-budget` (revenue-breakdown sub-step — runs `scripts/budget/run_nap_annual.ts`, then `run_policy_baseline.ts` since the PIT shares feed the tax-policy simulator; see "Revenue-breakdown ingest" + "Policy-baseline refresh" below) |
| `Eurostat policy-baseline (BG)` (consumption + tax aggregates) | `update-budget` (policy-baseline sub-step — runs `scripts/budget/run_consumption_coicop.ts` then `scripts/budget/run_policy_baseline.ts`; see "Policy-baseline refresh" below) |
| `НОИ — месечни B1 отчети по фондове` | `update-noi` (manual download to `raw_data/budget/noi/`, then `tsx scripts/budget/noi/__write_funds.ts`) |
| `ДВ — Инвестиционна програма за общински проекти` | `update-budget` (investment-program sub-step — fetch PDF to `raw_data/budget/investment_program/{year}-annex-iii.pdf`, then `tsx scripts/budget/investment_program/__write_program.ts`) |
| `МРРБ — ИПОП (Инвестиционна програма за общински проекти) изпълнение` | `update-budget` (IPOP sub-step — `curl -o raw_data/budget/ipop/projects-{year}.csv 'https://ipop.mrrb.bg/reports_projects_export.php'`, then `tsx scripts/budget/ipop/ingest.ts` — re-fetches the daily-refreshed national execution CSV, rewrites all 264 municipality shards) |
| `Общински капиталови програми` (per-município capital lists) | `update-budget` (capital-programmes sub-step — for each (year/muni) flagged in the watcher's describe-line, fetch the source file into `raw_data/budget/capital_programs/<muni>-<year>.{xlsx,pdf}` and run `tsx scripts/budget/capital_programs/<muni>.ts --year <year>` THEN `tsx scripts/budget/capital_programs/__shrink_for_tile.ts` — the shrink step regenerates the `{muni}-tile.json` sidecar files that the dashboard hooks read; without it, the tile would still try to load the full 50-450KB file and bandwidth would balloon — see "Capital-programmes ingest" below) |
| `data.egov.bg общински бюджети` (касово изпълнение по ЕБК) | `update-budget` (municipal-execution sub-step — fully automated from the portal API, no manual download: `tsx scripts/budget/municipal_execution/ingest.ts --all` re-fetches every covered muni-year, parses the B3 ЕБК report, and rebuilds `data/budget/municipal_execution/{muni}/{year}.json` + `index.json`) |
| `Финансови показатели на общините (ЗПФ чл. 130г) — предстоящо тримесечие` | `update-budget` (municipal-fiscal sub-step) — **manual download first**, see below |
| `Сметна палата party financing` | `update-financing` |
| `Сметна палата annual-report index` | `update-financing` (annual-report year added — runs `scrape_reports.ts`) |
| `Eurostat macro` (BG) | `update-macro` |
| `EC EU budget per-MS spreadsheet` (BG receipts/contributions) | `update-macro` |
| `World Bank WGI` (BG) | `update-macro` |
| `Transparency International CPI` (BG) | `update-macro` (manual paste required first — see below) |
| `Standard Eurobarometer (institutional trust)` (`eurobarometer`) | `update-macro` (manual paste required first — a new Standard EB wave means the `EB_TRUST_*` arrays are stale; open the wave's BG national report, paste the new `{ year, value }` for trust in national government/parliament/EU into `scripts/macro/fetch_eurostat.ts`, then re-run) |
| `BNB domestic ДЦК auctions` | `update-macro` (re-scrapes debt-emissions list) |
| `Minfin government-debt bulletins (Eurobond signal, via Wayback)` | `update-macro` (Eurobond sub-step — read the latest MoF debt bulletin, hand-add any new international emission to `data/debt-emissions.json`) |
| `BNB monthly FDI flows` | `update-macro` (runs `fetch_bnb_fdi.ts` → `data/macro_fdi.json`) |
| `Minfin КФП monthly bulletins` | `update-macro` (re-scrapes fiscal-reserve via Wayback) |
| `Minfin programme-budget execution reports` (МФ via Wayback) | `update-budget` (operator backfills `raw_data/budget/exec-admin-ministerstvoto-na-finansite-<fy>.pdf` from the Wayback URL in the watcher's describe-line, then activates the manual-pdf entry in `EXECUTION_REPORTS`) |
| `МВнР programmatic execution reports` (via Wayback) | `update-budget` (when a new fiscal-year ZIP lands at `mfa.bg/upload/<id>/...програмен отчет МВнР <date>.zip`, add an `xlsx-in-zip` entry to `EXECUTION_REPORTS` with `entryName: "1100-Otchet programi <YYYY>12_MVnR.xlsx"` — the existing FY2023 entry is the template) |
| `State Budget Law promulgation` (via Wayback) | `update-budget` (the annual ЗДБ — or, for a year with no adopted budget, the interim "Закон за събирането на приходи" bridging law — landed in ДВ. Resolve its `idMat` from dv.parliament.bg and add a row to `LAW_DV_MATERIALS` (annual law) or `INTERIM_BUDGET_LAWS` (bridging law / ЗИД) in `scripts/budget/fetch_sources.ts`, then re-ingest. Best-effort/Wayback-lagged — cross-check dv.parliament.bg + minfin.bg/bg/1770) |
| `Eurostat regional` (BG) | `update-regional` |
| `НСИ regional open-data` (BG) | `update-regional` |
| `Ember Yearly Electricity Data` (generation mix) | `update-energy` (runs `scripts/energy/fetch_generation.ts` → `data/energy/generation.json`) |
| `Eurostat energy prices (BG)` (household electricity `nrg_pc_204` + gas `nrg_pc_202`) | `update-energy` (one run of `scripts/energy/fetch_prices.ts` writes both `data/energy/prices.json` + `gas_prices.json`, each BG+EU27+RO/GR/HU/HR peers) |
| `EC Weekly Oil Bulletin` (consumer fuel prices) | run `npx tsx scripts/consumption/fetch_fuel.ts` (no update-* skill; the ingest self-reports its `/data/updates` row, writes `data/fuel.json` for /consumption/fuel). |
| `Eurostat food price levels (BG)` (PLI) | `update-macro` (also re-runs `scripts/macro/fetch_food_pli.ts` → the `foodPli` block in `data/macro_peers.json` for /consumption/eu). |
| `AZ (Агенция по заетостта)` | `update-indicators` + `update-regional` |
| `МОН: ДЗИ резултати` | `update-indicators` (município ДЗИ) **+ `update-schools`** (per-school /education + /school: НВО fetch → schools index → SES context → textbook concentration) |
| `МОН: НВО 7. клас резултати` | `update-schools` (prior-attainment baseline for the value-added on /school/:id; dedupe with the ДЗИ row — usually the same day) |
| `НСИ: население по общини` | `update-indicators` |
| `НСИ: раждания, умирания и миграция по общини` | `update-indicators` |
| `НСИ: Баланс на територията` (LANDUSE annex) | `update-landuse` (operator pastes the new year's PDF URL into `LANDUSE_REPORTS` in `scripts/landuse/sources.ts`, then runs `npx tsx scripts/landuse/fetch.ts --year <year>` — re-parses the 28-oblast / 8-category land-use composition into `data/landuse/index.json` for the My-Area "Property stock" tile) |
| `ГРАО: население по постоянен и настоящ адрес` | `update-grao` |
| `CIK local-elections results bundles` | `update-local-elections` (run one `--local-ingest <cycleSlug>` per changed cycle from the watcher's describe-line — the HTML-only path that mirrors per-município pages; Cloudflare bypass via headless Playwright; idempotent on the mirror so re-runs are cheap. `_chastichen` and `_nov` (incl. single-round council re-elections) are auto-handled. A describe-line ending **`<cycle> тур2 (runoff) re-check`** means a mayoral partial's round-2 runoff window has opened (the watcher re-flags each fresh partial ~9 and ~16 days after election day) — **re-run the same `--local-ingest <cycle>`**: it's idempotent on the already-mirrored round-1 pages and pulls the round-2 (tur2) pages published since, resolving the real runoff winner (round-1 HTML marks both finalists elected). **Exact by-election turnout (+ the per-section mayor map) is backfilled automatically for EVERY race** at the tail of every `chmi*` `--local-ingest` (`ingestByElectionTurnout` reads ЦИК's «Числови данни» HTML — no OCR): район/община mayor → `bundle.protocol` + `data/<cycle>/sections/<code>.json` (the section map); **kmetstvo (village) mayor → the matching `bundle.kmetstva[]` entry**. It is **round-aware** — each race is fetched for the round that elected its winner, so the **tur2 re-check re-run backfills the runoff turnout** (a round whose pages aren't published yet 404s and is skipped, then picked up next run). It rebuilds `local_chmi_history.json` so turnout reaches the dashboard `активност`, the section map, the `/local/chmi` feed, and the `chmiEvents` AI tool. If protocols aren't published yet on election night the dashboard falls back to an _estimate_ until a later run; to refresh turnout alone without re-mirroring HTML, `npm run data -- --local-byelection-turnout <slug>`. For a re-uploaded **regular** cycle (mi2019/mi2023) whose section bundle changed, also run `--local-csv <slug>` to refresh per-station votes/turnout. Final steps — `npx tsx scripts/parsers_local/backfill_kmetstvo_ekatte.ts` refreshes the kметство → EKATTE lookup the My-Area village-mayor tile reads; `npx tsx scripts/parsers_local/decorate_local_mp_links.ts` stamps `mpId` on every candidate row whose name matches an MP so the parliament.bg portrait surfaces on mayor / council rows; `npm run data -- --local-coords` stamps GPS + building address onto the section index that powers the per-município section map — idempotent, and also re-run after a new **parliamentary** election lands fresh section coordinates; finally `npm run data -- --local-problem-sections` (run AFTER `--local-coords` — most neighborhoods key on the stamped address; the exception is blank-address махала pinned by МИР-agnostic section suffix, e.g. Филиповци) regenerates the Roma-neighborhood `problem_sections.json` behind the município **and** район drill-down Risk-votes tiles — each neighborhood carries a `rayonCode` so it also surfaces on its район page (Sofia `S2xxx` / Пловдив-Варна `<muni>-<code>`). Then regenerate the derived cross-cycle artifacts that must gain the new cycle — flag-gated steps NOT folded into `--all`, so the orchestrator has to run them after any local-cycle ingest: `npm run data -- --local-place-trends` (per-place + per-município trend shards → `data/local_place_trends/{s,r,p,m}/` — the settlement / район / Sofia-район trend tiles AND the município council-trend + mayor-timeline tiles all read these one-shard-per-place files instead of fanning out the per-cycle bundles), `npm run data -- --prevote-flows` (the parliament→council pre-vote Sankey → `data/transitions_prevote/`), and `npm run data -- --local-flows` (council→council flow → `data/transitions_local/`). Finally `bucket:sync` the regenerated `data/` trees — `data/local_place_trends/` is **gitignored** and ships ONLY via the bucket, so skipping the sync leaves every place/município trend tile 404-empty on prod.) |
| `ЦИК presidential-elections results bundles` | `update-presidential-elections` (its OWN watcher and its OWN skill, deliberately: the `cik_results` row above is coupled to `update-persons` + `db:load:person-elections:pg`, and that loader reads the **parliamentary** candidate files — a presidential flip routed through it stalls on a tree it cannot read or skips the new cycle silently. A describe-line reading **`<slug>: NEW CYCLE <year> — not ingested`** means an election happened and ЦИК published its archive: the slug has to be added to `PRESIDENTIAL_SOURCES` **with the real archive URLs checked in a browser** before the ingest can run, because ЦИК has used a different archive name in every cycle and 2006 is on a different host. **`<slug> tur2: bundle published`** is the runoff, about a week after round 1 — re-run the same `--pvr <cycle>`, which is idempotent. A line ending **`(URL guessed)`** means the cycle is not in the catalogue yet and the URL was a template, so a 404 there proves nothing. **The chain lives in that skill, not here** — copying it produced two versions that already disagreed. Follow its Steps 1–7. ⚠ **IT DOES LOAD POSTGRES NOW, and this row said it did not.** „NO POSTGRES — nothing in this family loads into a database" was true when written and stopped being true when migration 195 landed: `npm run db:load:election-national:pg:cloud` publishes `data/<cycle>/national_summary.json` — every election folder, `_pvr` included — and it is the ONLY path that corpus takes to prod, so skipping it leaves the national results a vintage behind at a 200. It is Step 6 of that skill; named here because Step 9 has to emit it, not so the chain gets a second copy. Still no `person:slugs` — the person layer does not read presidential results in v1. ⚠ `npm run sitemap` PRUNES every shard before writing, so run it only on a machine that has the `data/*_pvr` trees or it deletes `public/sitemap_presidential.xml` and exits 0.) |
| `CIK news` (if re-enabled) | _no skill yet — surface as TODO_ |
| `iisda.government.bg — Кметове на общини` | `update-municipal-contacts` (re-scrape iisda mayor pages — captures mayor + every "Заместник-кмет" email per município, surfaced as inline `mailto:` icons on MyAreaGovernmentCard) |
| `ИАОС air quality (data.egov.bg)` | `update-air-quality` (refresh PM10 + PM2.5 quarterly CSVs for the My-Area air-quality tile) |
| `Прозрачност без граници — LISI` | `update-transparency-lisi` (manual paste — 27 new oblast-capital scores into `LISI_2024` array, then rebuild `data/municipal_transparency/index.json`) |
| `ВСС — съдебна статистика` | `update-judiciary` (add the new year to `VSS_ANNUAL_TABLES`, then rerun the caseload + court-load parsers and reload the PG-served court_load) |
| `НАТО — разходи за отбрана` | `update-defense` (parse Tables 3 & 8a of the new NATO PDF into `data/defense/{gdp_share,category_split}.json`) |
| `МО — доклад за отбраната` | `update-defense` (refresh `data/defense/readiness.json` from the МО state-of-defence report) |
| `МИ — износ на оръжие` | `update-defense` (refresh `data/defense/exports.json` from the Ministry of Economy export report — euro figures, not SIPRI TIV) |
| `АСП — отчети и доклади` | `update-social` (a new АСП annual report landed — re-run `scripts/social/fetch_asp_benefits.ts` to verify `data/social/benefits.json` against it; update the curated figures if the drift guard flags any) |
| `ГИТ — доклади за дейността` | `update-social` (a new ГИТ annual report landed — re-run `scripts/social/fetch_git_inspections.ts` to verify `data/social/git_inspections.json`; update the curated figures if the drift guard flags any) |
| `Eurostat road safety (BG)` (sdg_11_40) | _no full skill — run `npx tsx scripts/security/fetch_road_safety.ts`_ (re-fetches the national road-death series into `data/security/road_safety.json` for the /sector/security "Пътна безопасност" outcome tile; a small committed file, no bucket sync, no PG). |
| `Eurostat rail (BG)` (rail_pa_total) | _no full skill — run `npx tsx scripts/transport/fetch_rail_ridership.ts`_ (re-fetches national rail passengers into `data/transport/rail_ridership.json` for the /sector/transport rail subsidy-per-passenger tile; a small committed file, no bucket sync, no PG). |
| `Eurostat tourism (BG)` (tour_occ_nim / tour_occ_ninraw) | _no full skill — run `npx tsx scripts/tourism/fetch_eurostat_tourism.ts`_ (re-fetches monthly nights-spent by month + country-of-origin into `data/tourism/visitors.json` for the /sector/tourism seasonality + origin-markets charts, read via `src/data/tourism/useTourismVisitors.ts`; a small committed file, no bucket sync, no PG). |
| `ИПИ — Местни данъци (265 общини)` | `update-local-taxes` (re-fetches the 5 ИПИ per-indicator CSVs from `265obshtini.bg/downloadCSV/{id}`, normalises município names via the alias table in `scripts/local_taxes/lib/match_obshtina.ts`, computes per-indicator national ranks, and merges into `data/local_taxes/index.json` — preserves the per-município `naredba` blocks the Tier B parsers wrote) |
| `Общински наредби за местни данъци` | `update-local-taxes` Tier B — `npx tsx scripts/local_taxes/run_naredba.ts <obshtina1>,<obshtina2>` for exactly the municípios named in the watcher's describe-line (`N naredba(s) re-uploaded: SOF00, PDV01`). Adds residential ТБО + basis flag, tourist tax, and dog tax into the same `data/local_taxes/index.json` under each município's `naredba` key, preserving the Tier A `ipi` blocks. |
| `Council resolutions + vote tallies` | `update-council-minutes` (`npm run council:scrape -- --per-councillor` — incremental scrape across every município wired in `data/council/sources.json` that isn't `phase1Defer`; uses per-município watermarks in `state/ingest/council_<obshtina>.json`; pdftotext+regex parsers for the tier-A native-text munis, Gemini Vision OCR fallback gated by `--ocr` for scanned-only protocols, and `--per-councillor` joining the named-vote blocks to the cacbg municipal roster at `data/officials/municipal/` — required, not optional. Finish with `npm run db:load:council:pg` and `npm run db:load:council:pg:cloud`.) |
Some sources map to the same skill — e.g. `update-connections` handles both `cacbg_declarations` and the BG Post postcode→EKATTE refresh — so dedupe and run it once. (Commerce Registry is no longer one of them: `egov_commerce` now maps to its own `tr-daily-refresh` step.)
Four of these skills — `update-rollcall`, `update-procurement`, `update-funds`, and `parliament-scrape` — also (re)generate the per-MP **data-diet shards** the candidate / company / official pages read instead of multi-MB aggregates. They regenerate automatically inside the ingest. For `parliament-scrape` these reach prod via the run's final `bucket:sync` (mandatory, not optional); **`update-rollcall`, `update-procurement` and `update-funds` all reach prod via `db:load:*:cloud` instead** — their shards are PG-load sources, not serving artifacts. (`update-rollcall` joined that list in json-retirement-v2 Tier 1; this sentence said `bucket:sync` until 2026-08-29.) See **"Per-MP / per-entity shard invariant"** below.
**Connections-graph freshness.** The `/connections` graph + the `/person` "Свързани лица" tile are served LIVE from the Postgres graph engine (`graph_payloads` / `person_connections`, migrations 127-129 + 084), re-derived by `db:load:graph:pg` off the person layer — the old static `connections*.json` generator is retired (connections-engine-v1 §P4.3). **[2026-08-21] The TR-namesake guard this note used to describe is gone from every arm that had it.** `mp_connected` (2026-08-20) and `pep_connected` (2026-08-21) now read the gated person layer — `company_politicians`, or the same gate re-derived live — where migration 148's `tr_name_fold_people` people-per-name fold **REFUSES** a name the Commerce Registry says covers more than one human, rather than keeping a name-only match that happens to map to a single TR company. Same purpose (a councillor stays off Софарма Трейдинг's billions), a strictly better test; the per-EIK `company-connections/` JSON that shared the old proxy was retired to migration 158 on 2026-08-16. ⚠️ Both links' staleness trigger moved with their source: `db:load:tr:pg`, which needs a resolved person layer. The graph folds the officials links `/update-officials` produces via the person layer, so when `cacbg_officials`/`cacbg_local` change **without** a same-run `cacbg_declarations` change, re-run the person chain (`db:resolve:persons` → `db:load:graph:pg`, or the `update-persons` cloud chain, which sequences it) after `/update-officials` to refresh the graph's officials layer. There is no offline connections rebuild anymore.
**Price-hub freshness.** The `price_payloads` **hub-stats** blob (the headline numbers on the `/consumption` hub tiles) folds five gap fields from OTHER skills' JSONs — `fuelGapPct` (`data/fuel.json`, `ec_oil_bulletin` → `update-fuel`), `electricityGapPct` + `gasGapPct` (`data/energy/{prices,gas_prices}.json`, `eurostat_energy_prices` → `update-energy`), `foodInflationPct` (`data/macro.json`, `eurostat` → `update-macro`) and `euFoodPli` (`data/macro_peers.json`, `eurostat_food_pli` → `update-macro`). But the blob is rebuilt ONLY by `update-prices` (source `kzp_prices`). So when any of `ec_oil_bulletin` / `eurostat_energy_prices` / `eurostat` / `eurostat_food_pli` change **without** a same-run `kzp_prices` change, those tile numbers go stale. After the owning skill runs, also run `npm run prices:payloads` (it re-folds the JSONs into the serving blobs from the current PG price data — it does NOT re-fetch the daily price ZIP), then publish to Cloud SQL in Next-steps (step 9): `DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npm run prices:payloads`. (In practice `kzp_prices` publishes ~daily so the daily `update-prices` run already refreshes them within a day; this covers the days it doesn't — a price-feed gap, or a fuel/energy/macro release landing alone.)
### Per-MP / per-entity shard invariant (data-diet)
The candidate / company / official pages stay lean by reading **per-entity shards + a tiny `index.json` manifest** instead of chamber-wide aggregates. Each shard family is regenerated **idempotently on every ingest** of its source (write-if-changed guards, so unchanged shards don't churn the bucket) and ships to prod via the orchestrator's final `bucket:sync` — they all live under `data/`, which the sync rsyncs whole. None of this needs a separate command in the per-skill recipes; it's already inside the ingest. The point of this note is the **invariant the orchestrator must not break.**
| Shard family | Regenerated by | Generator | Aggregate it replaces (frontend fallback size) |
| --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| ~~`parliament/votes/derived/per-mp/<ns>/<csvId>.json`~~ **(PG-served since json-retirement-v2 Tier 1 — NOT bucket-synced; the bucket objects were REMOVED, and `sessions/`, `derived/dissents.json` + `derived/party_pair_breaks.json` went with them)** | `update-rollcall`, every metric rebuild | `scripts/parliament/derived/per_mp_shards.ts` | n/a — `/candidate/:id` reads `/api/db/mp-similarity` + `mp-dissents` (loaded by `db:load:rollcall-derived:pg`) |
| ~~`procurement/derived/per-mp/…`, `per-eik/`, `pep-by-eik/`, `pep-by-slug/`~~ **(PG-served — NOT bucket-synced; gitignored local PG-load sources)** | `update-procurement`, every `procurement:ingest` | `scripts/procurement/cross_reference.ts` + `pep_connected.ts` | n/a — `/candidate/:id`, `/company/:eik`, `/officials/:slug` read Postgres via `/api/db` |
| ~~`funds/derived/per-mp/…`, `by-eik/`, `political-by-eik/`~~ **(PG-served — NOT bucket-synced; gitignored local PG-load sources)** | `update-funds`, every `funds:ingest` | `scripts/funds/cross_reference.ts` + `political_links.ts` | n/a — `/candidate/:id`, `/company/:eik`, `/funds/*` read Postgres via `/api/db` (loaded by `db:load:funds:pg`) |
| `<election>/candidates/by-slug/*.json` + `resolved.json` | `parliament-scrape` Step 6 (`rebuild_resolved.ts`) — **not** auto-run by any ingest; the one manual link | `scripts/preferences/save_candidate_resolved.ts` | `candidates.json` (~1 MB) + `parliament/index.json` (~950 KB) |
| `parliament/by-id/<id>.json` (one roster entry per MP) | `parliament-scrape` (auto-written every run by `lib/writeMpById.ts`; standalone backfill `build_mp_by_id.ts`) | `scripts/parliament/lib/writeMpById.ts` | `parliament/index.json` (~950 KB) |
**Why the orchestrator must care:** the frontend falls back to the aggregate whenever a shard is missing. The candidate page additionally **gates** the votes fallback on NS membership — an MP _in the selected parliament_ whose vote shard is missing pulls the **12 MB** `similarity.json`; an off-NS / former MP skips it entirely (see [[project_candidate_page_data_diet]]). So:
> **Invariant: never publish an aggregate refresh without re-publishing its shards + manifest.**
In practice this is automatic — the ingest writes shard, manifest, and aggregate in one pass, and the final `bucket:sync` pushes all three. The only ways to break it:
- Running an ingest but **skipping the publish step** (or syncing a partial subtree). The **GCS-served** family is now just the candidate-resolution shards (`<election>/candidates/by-slug/` + `parliament/by-id/`), and for those the Next-steps `bucket:sync` is **mandatory**, not optional. **Rollcall, procurement and funds are all PG-served**: their shards are load sources, so the publish is `db:load:rollcall[-derived]:pg:cloud` / `db:load:pg:cloud` / `db:load:funds:pg:cloud`, NOT `bucket:sync`.
⚠️ **[2026-08-29] The rollcall half moved, and this bullet asserted the opposite until then.** `parliament/votes/{sessions,derived/per-mp}` are excluded from every sync path AND their bucket objects are gone, so a scoped `bucket:sync:paths -- parliament/votes` pushes 0 of them by design. Do not read that zero as a missed upload — it was misread as one on 2026-08-29.
- For candidate pages specifically, refreshing `parliament/index.json` (via `parliament-scrape`) **without** Step 6's `rebuild_resolved.ts` — the resolution shards then point at a stale roster. That step is already called out in the `Parliament MPs` mapping row; this is the reminder of why it's load-bearing.
The manifest (`per-mp/index.json`) is the cheap gate the frontend reads _first_ to decide whether a shard exists at all — it's written in the same generator pass. For the candidate-resolution family a normal full-tree sync carries it; for the PG-served families the loader is what publishes it.
### Пловдив/Варна район shards (pipeline-derived — no watcher source)
The Пловдив/Варна административни районите surface as obshtina-shaped places (`/settlement/PDV22-06`, `/governance/PDV22-06`, the 4-tab place switcher) backed by per-район shards under `data/<election>/municipalities/<muni>-<code>.json` plus the район geometry (`data/maps/city_rayons/<muni>.json`) and per-election results (`data/<election>/rayon/<muni>.json`). These are **derived from the parliamentary section data**, not an external feed — so there is no watcher source and no mapping-table row. They are rebuilt by the main pipeline's `--city-rayons` step (`scripts/helpers/gen_city_rayon_data.ts`, an exported function called from `scripts/main.ts`), which is **folded into `--all`** — so `npm run prod` always refreshes them in step with the section data they derive from.
When the район data needs a refresh outside a full `npm run prod`:
- A new parliamentary election is ingested, or section votes / coords change — note the `--local-coords` step in the `CIK local-elections results bundles` recipe above already re-runs "after a new parliamentary election lands fresh section coordinates", and the район **geometry** depends on those same coords. So after that step, also run:
```bash
npm run data -- --city-rayons
```
Then **publish** — the shards are gitignored and bucket-served, so run `npm run bucket:sync:all` (otherwise `/settlement/PDV22-06` 404s its shard on prod). There is no `state/ingest` marker: it's a pipeline step, not a tier-2 skill, so nothing to stamp.
### Governments: manual edit first
`wiki_governments` is special — `data/governments.json` is hand-maintained from the BG Wikipedia governments-list page, and no ingest script exists. The watcher detects when the page is edited (new cabinet row added, end-date filled on the incumbent, coalition footnote updated) but cannot itself update the JSON.
When this source flips, surface it under a dedicated "Manual action required" block in the plan with the change signal from `state/watch/wiki_governments.json` (`detail` field and the `describe()` line, which distinguishes "+N cabinet row(s)" from "tail of page edited"):
> Manual action required: `wiki_governments` flipped — `{describe-line}`. Check the BG Wikipedia governments-list page (https://bg.wikipedia.org/wiki/Списък_на_правителствата_на_България) and edit `data/governments.json` to add the new cabinet, update the previous incumbent's `endDate` + `endReason`, or fix the coalition footnote. Then re-run the orchestrator.
The user must edit the file by hand. There is no marker to stamp; the next orchestrator run will re-surface the change until the fingerprint matches a future state (i.e. the watcher won't auto-resolve — once the user has done the edit, they can run `npx tsx scripts/stamp-ingest.ts wiki_governments --summary "manually edited data/governments.json"` to stamp).
If the user says "skip governments for this run", drop it from the plan without stamping.
### Municipal fiscal: manual download first
`municipal_fiscal_due` does not watch minfin.bg — it cannot. That site serves an
interactive Cloudflare Turnstile to every non-browser client, and the Internet Archive's
CDX API rate-limits this repo's egress. So the source watches OUR OWN coverage against the
calendar: the newest quarter in `data/budget/municipal_fiscal/index.json` versus the newest
that should have been published by now (quarter end + a 90-day lag).
**It raises a `ManualRequest`, which the watcher renders in a "Manual downloads needed"
section ABOVE `## Changed`.** Read that section first — it names the exact files and where
they go. The request is repeated on every run until the file lands, including runs where
the source is off-cadence or errored, because a missing file blocks the ingest every day
rather than only on the days МФ is due to publish.
**This sub-step has its OWN ingest marker: `state/ingest/update-budget-municipal-fiscal.json`,
never `state/ingest/update-budget.json`.** That is load-bearing rather than tidiness. Roughly
thirteen watcher sources map to `update-budget`, so a shared marker is stamped whenever ANY of
them legitimately fires — and stamping it while this quarter is still missing masks the
outstanding download for the ~90 days until the next one falls due, silently. The same failure
cost the КЗК arm five weeks, documented elsewhere in this file.
**Do not invoke the municipal-fiscal sub-step while the download is outstanding.** The ingest
reads the drop directory; run it with the workbook missing and it parses nothing new and exits
0. Report it as skipped and leave `update-budget-municipal-fiscal.json` untouched. The request
itself keeps firing regardless — it is derived from what is on disk, not from a marker — so
what a wrong stamp actually suppresses is the ORCHESTRATOR queueing the work, not the report
line.
Once the workbook is in `data/_cache/minfin_municipal_fiscal/`:
```bash
npx tsx scripts/budget/municipal_fiscal/ingest.ts # → data/budget/municipal_fiscal/*.json
npm run db:load:municipal-fiscal:pg # → the municipal_fiscal table
npm run macro:municipal-stocks # → the 3 national series in data/macro.json
```
**The THIRD is easy to forget and fails silently.** The corpus has two consumers: the
`/governance` surfaces read Postgres and move after the second command, while
`/indicators/fiscal` is bucket-served from the committed `data/macro.json` and moves only
after the third. Skip it and the national commitments tile keeps serving the previous
quarter at a 200 with every row count reconciling.
Only the SECOND is in `db:refresh` — the ingest has no npm alias and is not in the chain,
because its input is the gitignored drop directory. So a `db:refresh` re-loads whatever JSON
is committed; it never re-reads a workbook. On the cloud side the loader is standalone and
needs `db:load:place-dim:pg:cloud` first — see CLAUDE.md.
**The archive is 27 workbooks across four incompatible layouts**, and only the 2024+ ones
parse. The ingest skips the rest by design and prints why; a "column map mismatch" line is
the one to actually read, because that is the shape a shifted CURRENT release takes. See
`data/_cache/minfin_municipal_fiscal/README.md`.
### TI CPI: manual paste first
`transparency_cpi` is special. The CPI values live in a hand-curated `TI_CPI` array at `scripts/macro/fetch_eurostat.ts:572`; the watcher detects when TI publishes a new annual score, but `update-macro` itself does not re-scrape the country page. So when this source flips:
1. **Do not auto-queue `update-macro` for it yet.** Instead, surface it under a dedicated "Manual action required" block in the plan with the new score from `state/watch/transparency_cpi.json` (the `meta.year` and `meta.score` fields):
> Manual action required: `transparency_cpi` flipped to `{year} CPI = {score}/100`. Paste `{ year: {year}, value: {score} }` into the `TI_CPI` array at `scripts/macro/fetch_eurostat.ts:572`, then re-run `/update-macro`.
2. **If the user confirms they've pasted (or invokes the orchestrator again after the paste),** treat `transparency_cpi` like any other `update-macro` source and queue it. The skill will re-run, write the new point into `data/macro.json`, and the marker stamps as usual.
3. If the user says "skip CPI for this run", drop it from the queue without stamping — the next orchestrator run will surface it again.
### Capital-programmes ingest (`capital_programs`)
The 26 ingested общини (Sofia, Plovdiv, Burgas, Stara Zagora, Ruse, Varna, Pleven, Sliven, Dobrich, Asenovgrad, Shumen, Vidin, Veliko Tarnovo, Pernik, Haskovo, Gabrovo, Yambol, Kardzhali, Lovech, Dupnitsa, Velingrad, Samokov, Karlovo, Kazanlak, Kyustendil, Montana) each publish an annual капиталова програма on their own website. The watcher tracks all sources under one fingerprint; its describe-line names exactly which `<year>/<muni>` entries flipped.
Each município has its own parser script:
After any capital-programmes parser run, ALWAYS run the shrink step:
```bash
tsx scripts/budget/capital_programs/__shrink_for_tile.ts
```
This regenerates the `{muni}-tile.json` sidecar files (~5KB each) that the dashboard hooks fetch instead of the 50-450KB full files. The tile sidecars truncate `projects[]` to top-30 by amount and add a `projectCount` top-level field so the tile's "X projects" badge still shows the full count. `bySettlement[]` and `byRayon[]` are kept intact because the Sofia tile uses `byRayon.find()` to look up specific rajons — truncating would hide outside-top-N rajons.
- **XLSX/XLS**: Sofia, Burgas, Ruse, Veliko Tarnovo (sheet "Pril15"), Pernik (post-euro EUR figures), Burgas 2022, Karlovo (sheet "2025" — `Обща сума за обекта` col is the per-line total, no OCR).
- **Born-digital PDF**: Plovdiv, Stara Zagora (inside a ZIP), Asenovgrad, Shumen.
- **Rasterized PDF needing Gemini Vision OCR**: Varna, Sliven.
- **Born-digital PDF needing OCR after page-slicing** (fragmented layout): Pleven, Burgas 2024/2023.
- **HTML scrape**: Dobrich (inline tables).
- **`.doc` inside a RAR archive**: Vidin (operator workflow: unar → textutil).
Output schema is shared under `data/budget/capital_programs/{year}/{muni}.json`. Operator workflow when the source flips:
1. **Identify which (year, muni) entries flipped** from the watcher's describe-line — e.g. `"2 capital programme(s) re-uploaded: 2025/sofia, 2025/burgas"`.
2. **Fetch each flagged source** into `raw_data/budget/capital_programs/`. The URLs are catalogued in `CAPITAL_PROGRAM_URLS` in `scripts/watch/sources/capital_programs.ts`; mirror to disk with a browser UA:
```bash
curl -sL -A "Mozilla/5.0" \
-o raw_data/budget/capital_programs/<muni>-<year>.<ext> \
"<url from CAPITAL_PROGRAM_URLS>"
```
For Stara Zagora, the catalogued URL is a ZIP — extract the `pr 4 KV*.pdf` file from it. For Sofia/Burgas use the `.xlsx` extension; Plovdiv uses `.pdf`.
3. **Run the matching ingest script** per flagged município:
```bash
tsx scripts/budget/capital_programs/sofia.ts --year <year>
tsx scripts/budget/capital_programs/plovdiv.ts --year <year>
tsx scripts/budget/capital_programs/burgas.ts --year <year>
tsx scripts/budget/capital_programs/stara_zagora.ts --year <year>
tsx scripts/budget/capital_programs/ruse.ts --year <year>
tsx scripts/budget/capital_programs/dobrich.ts --year <year>
tsx scripts/budget/capital_programs/asenovgrad.ts --year <year>
tsx scripts/budget/capital_programs/veliko_tarnovo.ts --year <year>
tsx scripts/budget/capital_programs/pernik.ts --year <year>
# Haskovo: OCR via Gemini Vision first, then rollup:
tsx scripts/budget/capital_programs/haskovo_ocr.ts --year <year>
tsx scripts/budget/capital_programs/haskovo.ts --year <year>
# Gabrovo: same OCR pattern as Haskovo:
tsx scripts/budget/capital_programs/gabrovo_ocr.ts --year <year>
tsx scripts/budget/capital_programs/gabrovo.ts --year <year>
# Yambol: extract RAR (unar) or ZIP (Python CP866) first, then OCR:
tsx scripts/budget/capital_programs/yambol_ocr.ts --year <year>
tsx scripts/budget/capital_programs/yambol.ts --year <year>
# Kardzhali: same OCR pattern; URLs discovered via Google site:kardjali.bg:
tsx scripts/budget/capital_programs/kardzhali_ocr.ts --year <year>
tsx scripts/budget/capital_programs/kardzhali.ts --year <year>
# Lovech: scanned 77-page bundle; slice pages 36-42 with pypdf, then OCR:
# python3 -c "from pypdf import PdfReader, PdfWriter
# r = PdfReader('raw_data/budget/capital_programs/lovech-<year>.pdf')
# w = PdfWriter()
# for i in range(35, 42): w.add_page(r.pages[i])
# open('raw_data/budget/capital_programs/lovech-<year>-capital-pages.pdf','wb').write(w)"
tsx scripts/budget/capital_programs/lovech_ocr.ts --year <year>
tsx scripts/budget/capital_programs/lovech.ts --year <year>
# Dupnitsa: download via PHP service-endpoint (needs Referer), then OCR:
tsx scripts/budget/capital_programs/dupnitsa_ocr.ts --year <year>
tsx scripts/budget/capital_programs/dupnitsa.ts --year <year>
# Velingrad: born-digital PDF on m.velingrad.bg, then OCR:
tsx scripts/budget/capital_programs/velingrad_ocr.ts --year <year>
tsx scripts/budget/capital_programs/velingrad.ts --year <year>
# Samokov: Annex 5 PDF on samokov.bg, then OCR:
tsx scripts/budget/capital_programs/samokov_ocr.ts --year <year>
tsx scripts/budget/capital_programs/samokov.ts --year <year>
# Karlovo: Annex 7 XLSX on karlovo.bg (curl with Referer header), direct parse:
# curl -H 'Referer: https://karlovo.bg/' -o raw_data/budget/capital_programs/karlovo-<year>.xlsx \
# 'https://karlovo.bg/inc/service/service-download-file.php?identifier=<UUID>'
tsx scripts/budget/capital_programs/karlovo.ts --year <year>
# Kazanlak: Annex 4 inside the council's "Приложения" PDF (kazanlak.bg
# is Nuxt-rendered — fetch /<year-slug>/_payload.json to surface the
# file URL), then OCR. The accompanying Budget_<year>.xls is password-
# protected and unusable:
tsx scripts/budget/capital_programs/kazanlak_ocr.ts --year <year>
tsx scripts/budget/capital_programs/kazanlak.ts --year <year>
# Kyustendil: Annex 6 inside the council's "Окончателен годишен план"
# PDF (41 pages, capital programme on pp. 30-40). Operator pre-slices
# with pypdf, then runs Gemini Vision OCR:
# python3 -c "from pypdf import PdfReader, PdfWriter; r=PdfReader('IN'); w=PdfWriter(); [w.add_page(r.pages[i]) for i in range(29,40)]; open('OUT','wb').write(w.write)"
tsx scripts/budget/capital_programs/kyustendil_ocr.ts --year <year>
tsx scripts/budget/capital_programs/kyustendil.ts --year <year>
# Montana: 5-page scanned Konica Minolta "Капиталова програма" at
# /свали/бюджет/32. Parser uses page 5 only (consolidated summary;
# pages 1-4 are funding-source sub-appendices that would double-count):
tsx scripts/budget/capital_programs/montana_ocr.ts --year <year>
tsx scripts/budget/capital_programs/montana.ts --year <year>
# Vidin: unar extract + textutil convert the .doc first, then:
tsx scripts/budget/capital_programs/vidin.ts --year <year>
```
**Varna is two-step** — the source PDF is rasterized (200dpi scans), so a Gemini Vision OCR pre-step is required before the rollup parser:
```bash
# Step 1: OCR (one-shot, ~$0.30 in Gemini API credits, ~5-10 min for 71 pages)
tsx scripts/budget/capital_programs/varna_ocr.ts --year <year>
# Writes raw_data/budget/capital_programs/varna-<year>-ocr.json
# Step 2: rollup (deterministic, offline)
tsx scripts/budget/capital_programs/varna.ts --year <year>
# Reads the OCR JSON, writes data/budget/capital_programs/<year>/varna.json
```
The OCR JSON is committed to raw_data/ so the rollup is reproducible; only re-run varna_ocr.ts when the upstream PDF actually changes (the `capital_programs` watcher's fingerprint catches this). Requires GEMINI_API_KEY in `.env.local`.
**Pleven is three-step** — the source is a 63-page budget docket where capital data sits on pages 13-17 (Прил. №4 general) and 35-37 (Прил. №10А EU projects). Text is technically extractable but the layout is heavily fragmented, so we slice the capital pages into a focused PDF and OCR them:
```bash
# Step 1: slice the 8 capital pages out of the 63-page docket (pypdf required)
python3 -c "
from pypdf import PdfReader, PdfWriter
r = PdfReader('raw_data/budget/capital_programs/pleven-<year>.pdf')
w = PdfWriter()
for i in list(range(12,17)) + list(range(34,37)): w.add_page(r.pages[i])
with open('raw_data/budget/capital_programs/pleven-<year>-capital-pages.pdf','wb') as f: w.write(f)
"
# Step 2: OCR (cheap — 8 pages, ~$0.04 in Gemini credits)
tsx scripts/budget/capital_programs/pleven_ocr.ts --year <year>
# Writes raw_data/budget/capital_programs/pleven-<year>-ocr.json
# Step 3: rollup (deterministic, offline)
tsx scripts/budget/capital_programs/pleven.ts --year <year>
# Reads the OCR JSON, writes data/budget/capital_programs/<year>/pleven.json
```
Same OCR-cache convention as Varna. If a future year ships the capital appendices on different page numbers, update the python slice block above.
Each writes `data/budget/capital_programs/{year}/{muni}.json` and prints the recap total + per-район or per-village breakdown for canary verification.
4. **Adding a new year** (e.g. when 2026 programmes publish in spring): also add the new (year → urls) entry to `CAPITAL_PROGRAM_URLS` in `scripts/watch/sources/capital_programs.ts` AND update the `SOURCE_URLS` map at the top of each parser. Adding a new município follows the existing pattern: write a parser under `scripts/budget/capital_programs/<muni>.ts`, add a row to `CAPITAL_PROGRAM_URLS`, build a frontend tile (see the existing `{Sofia,Plovdiv,Burgas,StaraZagora,Ruse,Varna,Pleven}CapitalProjectsTile.tsx` for design variants — районирана / per-settlement / per-funding-source).
After running, stamp `update-budget` with a summary like `"capital_programs: refreshed sofia+burgas+ruse 2025"`.
### Revenue-breakdown ingest (`customs_revenue`, `nap_annual`)
Both these sources publish annually — `customs_revenue` (Митническа хроника) in March of T+1 and `nap_annual` (НАП Годишен отчет) also in March of T+1. They share the `update-budget` mapping but run two separate scripts that aren't part of `npm run budget:ingest` yet:
- `customs_revenue` flip → run `npx tsx scripts/budget/run_customs_revenue.ts` (writes `data/budget/revenue_breakdown/customs/<year>.json` for each year in `MITNICHESKA_HRONIKA_REPORTS`), **then `npx tsx scripts/budget/run_policy_baseline.ts`** — the tax-policy simulator's excise levers (diesel/petrol/cigarettes/spirits) read the per-product excise split from this customs file, so a new chronicle year must re-base the simulator too (see "Policy-baseline refresh").
- `nap_annual` flip → run `npx tsx scripts/budget/run_nap_annual.ts` (writes both `data/budget/revenue_breakdown/vat/<year>.json` and `data/budget/revenue_breakdown/pit/<year>.json`).
When the watcher describe-line says `"N new year(s) added"`, the operator first needs to look up the freshly-published PDF URL and add it to the relevant catalogue:
- Customs: `MITNICHESKA_HRONIKA_REPORTS` in `scripts/budget/customs_revenue.ts`. New URLs come from `customs.bg/wps/portal/agency/media-center/customs-chronicle/mh<YYYY>` (linked from the chronicle index page, but the actual PDF URL is opaque-UUID — find it via WebSearch with `site:customs.bg Mitnicheska_hronika <YYYY>`).
- НАП: `NAP_ANNUAL_REPORTS` in `scripts/budget/nap_annual.ts`. New URLs come from `nra.bg/wps/portal/nra/za-nap/osnovni-dokumenti/Godishni-otcheti-za-deynostta-na-NAP` (same opaque-UUID problem; find via `site:nra.bg "Годишен отчет НАП" <YYYY>`).
If the describe-line is `"N report(s) re-uploaded"` instead, no curation needed — just run the script with `--refresh` to bypass the cache and re-ingest.
After the script runs, stamp `update-budget` with a summary like `"customs: backfilled 2025; vat/pit: 2024 only"`.
### Policy-baseline refresh (`eurostat_policy`, and after КФП / НАП ingests)
The tax-policy simulator (/budget/simulator) reads one derived file, `data/budget/derived/policy_baseline.json`, assembled from the КФП December snapshots (now also the **expenditure** side: Персонал + Капиталови lines), the НАП annual-report PIT split, the **Митническа хроника excise split** (`revenue_breakdown/customs/<year>.json` — fuel/diesel/petrol/tobacco/alcohol lines that anchor the per-product excise levers), **five Eurostat datasets** (`nama_10_co3_p3` household consumption by COICOP, `nama_10_gdp` P31_S14 totals, `gov_10a_taxag` contribution aggregates, `earn_ses_hourly` SES decile ratios, `educ_uoe_perp01` teacher headcount), plus two **non-Eurostat local sources** (НОИ quarterly STATB pension-size bulletin, NSI open-data id=612 wages) and НОИ `funds.json` pension mass. (The gambling GGR anchor, the wine-volume base, and the EU-country excise comparator rates in `src/lib/euPolicyPresets.ts` — read by both the per-lever "like in…" picks and the whole-country quick-select profiles (`COUNTRY_PROFILES`) — are curated literature/industry constants, deliberately **no watcher**; refresh manually when a new annual rate table lands. They index up yearly (e.g. PL spirits ≈+5%/yr, FR ≈+1.75%/yr), so re-verify the excise rows against the Tax Foundation energy/cigarette tables and the EC DG TAXUD alcohol tables roughly once a year — last refreshed 2026-06-14.) Refresh recipe:
- `eurostat_policy` flip → run `npx tsx scripts/budget/run_consumption_coicop.ts` (rebuilds `data/budget/revenue_breakdown/consumption.json`) then `npx tsx scripts/budget/run_policy_baseline.ts`.
- `customs_revenue` flip → run `run_customs_revenue.ts` then `run_policy_baseline.ts` (the excise levers' diesel/petrol/cigarettes/spirits anchors come from the new chronicle year).
- `policy_baseline_local` flip → run `run_policy_baseline.ts`. If the describe-line says **"НОИ STATB{Q}/{YYYY} published"**, first bump `NOI_STATB_URL` (and the matching `STATB_QUARTER`/`STATB_YEAR` in `scripts/watch/sources/policy_baseline_local.ts`) to the new quarter, then re-run.
- `egov_budget_execution` flip that closes a fiscal year (a new December snapshot) → run `run_policy_baseline.ts` after `npm run budget:ingest` so the simulator re-bases to the newly closed year (also refreshes the personnel/capital expenditure anchors).
- `nap_annual` flip → run `run_policy_baseline.ts` after `run_nap_annual.ts` (the PIT employment/final-tax split and the МОД identity both come from the НАП file).
`run_policy_baseline.ts` is self-validating: it refuses to write if the VAT calibration factor drifts >12% min-max across years, or if the engine can't reproduce its own baseline row (slice-join guard). On failure, read the error — a drift gate usually means a statutory VAT-rate change landed and `VAT_SLICES` in `src/lib/bgTaxPolicy.ts` needs the new rate window before re-running. Smoke check: `npx tsx scripts/budget/__smoke_vat_model.ts` prints the calibration table and a few scored scenarios.
After the run, stamp `update-budget` with a summary like `"policy baseline re-based to FY2026, VAT factor 1.17"`.
### Simulator anchors: manual edits (`eu_tax_rates`, `eu_excise_rates`, `eu_alcohol_excise`, `oecd_pit_params`, `oecd_family_leave`, `nato_defence`, `ec_forecast_bg`, `nsi_edp`, `ec_vat_gap`, `imf_weo_bg`, `fiscal_council_bg`, `api_road_charges`)
The simulator's EU comparators, 5-year projection AND dynamic-mode behavioral layer ride sourced CONSTANTS in code, not pipeline JSON — automated WRITES are out (each value carries an editorial note: in-force year, caveats, calibration provenance), but five anchors have an automated CHECK (`oecd_pit_params` + `oecd_family_leave` are drift-only — they feed DERIVED levers, so they flag but cannot self-verify a value):
**Run the auto-checker FIRST** for `eu_tax_rates`, `eu_excise_rates`, `eu_alcohol_excise`, `ec_forecast_bg` and `nato_defence`:
```bash
npx tsx scripts/budget/check_policy_anchors.ts --source <id> --stamp # or --source all
```
It re-probes the upstream via the watcher source's own fingerprint() and compares against the engine constants (`EU_LEVER_PRESETS` values, `EC_FORECAST_EDITION`, `NATO_COMPENDIUM_EDITION`). Exit 0 = upstream still matches the code → the script stamps the ingest marker itself, nothing else to do (the common case: the watcher flipped on a page revision, not a rate change). Exit 1 = real drift → the DRIFT lines name the exact constants to edit; do the manual edit per the per-source notes below, re-run the checker, and let `--stamp` close it. WARN lines (reduced VAT rates, Germany's combined CIT) are not failures — the quick charts simply don't carry those values; check them manually only when the country page itself changed. On a manual edit, also bump the matching edition/pin constant (`EC_FORECAST_EDITION` in bgFiscalProjection.ts, `NATO_COMPENDIUM_EDITION` in euPolicyPresets.ts, the SK top-rate pin in the checker) in the same commit.
**НАП income tiers — manual one-off backfill (NOT watched).** The simulator's earnings distribution is cross-checked against the published НАП distribution of ДДФЛ filers by годишна данъчна основа, hand-keyed for tax year 2023 in `scripts/budget/nap_income_tiers.ts` (minfin.bg is WAF-blocked, so no watcher/scrape). When a later tax year is published as a MoF parliamentary answer (minfin.bg/bg/wreply/…), transcribe the 7-row table into `NAP_TIERS_2023` (rename to the new year), bump `NAP_TABLE_YEAR` + `NAP_YEAR_WAGE_FACTOR`, then `npx tsx scripts/budget/run_income_tiers.ts` (re-injects the `incomeTiers` block) and `npx tsx scripts/budget/__smoke_income_tiers.ts` (gates: body cumulative ±10%, tail ordering employee α > all-filer α). The load-time checksum catches a transcription error in the totals row.
On a DRIFT-driven edit:
- `eu_tax_rates` — the describe-line names the moved rates (e.g. `EE VAT 24 → 26`). Update the matching option(s) in `src/lib/euPolicyPresets.ts` (value in `apply`, label, and the note's in-force year); cross-check the PwC country page before editing. If a VAT **reduced** rate is implicated (the quick chart only carries standard rates), verify the country's "Other taxes" PwC page too. The checker also pins the whole-country quick-select profiles (`COUNTRY_PROFILES`): PL/SE standard VAT and PL/GR corporate are hard; SE corporate is warn-only (the chart shows 20.6 vs the slider's rounded 21); HU flat PIT is hard; the progressive top rates (DE/FR/IE/GR/SE/PL) are warn-only early-warnings. A DRIFT on a `country_*` pin means editing that country's `apply` block in `COUNTRY_PROFILES`, not a per-lever option. Background and source list: `docs/budget_simulator_grounding.md` ("EU country comparators").
- `eu_excise_rates` — a VALUE-level check (not edition-level): the describe-line names the moved fuel/cigarette rates (e.g. `PL diesel 391 → 398`). The checker compares EVERY encoded `exDiesel`/`exPetrol`/`exCigarettes` in `src/lib/euPolicyPresets.ts` (per-lever options AND `COUNTRY_PROFILES`) against the live Tax Foundation EU tables, so a DRIFT names the exact country+lever to edit (value in `apply`, label, and the note's in-force year). Units: diesel/petrol = TF €/L × 1000; cigarettes = TF excise per 20-pack € × 50.
- `eu_alcohol_excise` — a VALUE-level check of `exSpirits` (€/hl pure alcohol) and `exWine` (€/hl) against the EC "Taxes in Europe Database" (TEDB) v3 REST API — the authoritative per-state source (Tax Foundation's alcohol tables lag, so do NOT use them here). The describe-line names moved rates (e.g. `SE spirits 5000 → 4775`). Euro-area members are checked EXACT (HARD DRIFT past ±3 €/hl); non-euro members (SE/PL/HU/DK) come back EUR-converted at TEDB's floating rate, so a few-percent gap is WARN-only (FX vs a real change — verify). A DRIFT/large WARN means editing the country's spirits/wine `apply` value + label in `src/lib/euPolicyPresets.ts`. (TEDB also carries energy/tobacco under `EDU_ENERGY`/`EDU_TOBACCO` if `eu_excise_rates` ever needs a second opinion.)
- `oecd_pit_params` — an input-DRIFT probe (no auto-checker): the `nm` and `b2.t2` levers in `COUNTRY_PROFILES` are DERIVED from OECD Tax Database thresholds/allowances/credits (`nm` = basic allowance ÷ 12, or standard credit ÷ entry-rate; `b2.t2` = the chosen bracket threshold ÷ 12), so this flags when the upstream input moves rather than auto-verifying a value. The describe-line names the moved key (e.g. `DE|TH|L3 68480 → 69900` = Germany's 42%-rate threshold). Re-derive the affected country's `nm`/`b2.t2` in `src/lib/euPolicyPresets.ts`, confirm against the PwC country page (the OECD series lags national law ~1 year — prefer the current statutory figure), then stamp. The endpoint intermittently 500s; the probe retries, so a persistent failure is a transient, not a drift.
- `oecd_family_leave` — an input-DRIFT probe (no auto-checker): the `mat` lever (well-paid second-year months on BG's 0–12 scale) is DERIVED from OECD Family Database PF2.5 paid-leave weeks. The describe-line names the country whose maternity/parental/total paid weeks moved (e.g. `HU paid-leave 24/84/160 → …`). On a flip, re-judge that country's `mat` from the new weeks × payment rate and edit `src/lib/euPolicyPresets.ts`. Annual cadence (April reference).
- `nato_defence` — a new compendium edition (def-exp-`YYYY`-en.pdf) landed. Read Table 3 (defence as % of real GDP) and refresh the `def` options in `src/lib/euPolicyPresets.ts` (PL/LT/EE/GR/DE/IT values + estimate year in label and note). The whole-country profiles (`COUNTRY_PROFILES`) also carry a `def` value (EE/PL/HU/DE/FR/SE/GR — Ireland omits it as a near-zero outlier); refresh those from the same Table 3 in the same edit. Germany may be unreported — keep the national-budget figure with its caveat if so.
- `ec_forecast_bg` — a new EC forecast edition for Bulgaria. Update `src/lib/bgFiscalProjection.ts`: `EC_BALANCE_PCT` (balance % of GDP for the forecast years), `MACRO_PATH` (real growth, HICP, unemployment, nominal-growth proxy), and the header comment's edition reference; re-run `npx tsx scripts/budget/__smoke_fiscal_projection.ts` — the EC-anchor invariants there must be updated in the same commit, and `npx tsx scripts/budget/__smoke_behavioral.ts` (the behavioral layer reads `PROJECTION_GDP_EUR`). Also re-check the hero/projection i18n strings that name the edition (`budget_policy_hero_deficit*`, `budget_policy_proj_note`, `budget_policy_proj_macro` in both locales).
- `nsi_edp` — a new НСИ EDP notification (April/October). Update the 2025-vintage ESA anchors in `src/lib/bgFiscalProjection.ts` (`GDP_2025_EUR`, `BALANCE_2025_EUR`, `DEBT_2025_EUR` — or their successors after a re-anchor) and the header note; re-run `__smoke_fiscal_projection.ts` + `__smoke_behavioral.ts`. Also refresh the "2025 outturn" lines in `docs/budget_simulator_grounding.md`.
- `ec_vat_gap` — a new EC VAT gap edition. Re-read the BG gap (EUR level + % of VTTL) and re-verify `VAT_GAP_RESPONSE` in `src/lib/bgBehavioral.ts` (band + source string); re-run `__smoke_behavioral.ts` (the VAT-direction gate).
- `imf_weo_bg` — a new IMF WEO vintage for Bulgaria (April/October; the Article IV catalog itself is bot-blocked, this is the open proxy signal). Check IMF Bulgaria publications by hand for a new Article IV / multiplier evidence and re-verify the `MULT_*` bands in `src/lib/bgBehavioral.ts`; re-run `__smoke_behavioral.ts` (the Tier-2 magnitude gates).
- `fiscal_council_bg` — a new Фискален съвет publication (describe names the slug). If it costs a lever the simulator carries, refresh the benchmark table in the methodology article (`public/articles/2026-06-12-tax-policy-simulator-{bg,en}.md`), `docs/budget_simulator_grounding.md`, and — for dividend-style behavioral costings — the `DIV_BASE_SEMI_ELAST_PCT_PP` calibration in `src/lib/bgBehavioral.ts` (its smoke gate pins the 5→10% scenario to the ФС ceiling). Monthly "fiscal deficit % of GDP" posts usually need no edit — stamp with a "no levers affected" summary.
- `api_road_charges` — АПИ posted a newer annual road-charge total (`describe` names the fiscal year; the front-page press release is "приходите от пътни такси през <YYYY>"). Read the new combined винетки+тол лева figure off the АПИ press release, convert BGN→EUR (÷1.95583), and refresh `ROAD_CHARGES_BASE_EUR` in `src/lib/bgTaxPolicy.ts`; in the SAME edit bump `API_ROAD_CHARGE_LATEST_YEAR` in `scripts/watch/sources/fiscal_anchors.ts` (the fingerprint floor — forgetting it makes the source flap when the press release scrolls off). Re-run `npx tsx scripts/budget/__test_engine.ts`. The lever's tooltip strings (`budget_policy_tip_road_charges` in both locales) name the ~€562M base — update them if the figure moved materially.
Each is a code edit → run `npx eslint . --fix` + `npx tsc --noEmit`, then stamp completion: `npx tsx scripts/stamp-ingest.ts <source_id> --summary "EE VAT 24→26 applied"` (the stamp is what stops the orchestrator re-queueing it tomorrow).
## Source → skill mapping (canonical)
Each watcher source maps to one or more downstream skills. Multiple sources can map to the same skill (deduped at queue-build time); a single source can also fan out to several skills (`indicators_az` feeds both `update-indicators` for the municipality rate and `update-regional` for the oblast long-term-unemployment series) — queue each, then dedupe:
| Watcher source id (state/watch/<id>.json) | Mapped skill |
| ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `parliament_votes` | `update-rollcall` |
| `parliament_mps` | `parliament-scrape` |
| `wiki_polls` | `update-polls` — cross-check step ONLY, never an ingest on its own (decision 1). A flip here alone runs `polls:crosscheck` (report only) and still does the skill's Steps 0 and 5, which is what stops it re-queuing every day — it never writes into the corpus. |
| `polls_trend` | `update-polls` — Тренд's site watcher, rctrend.bg (docs/plans/polls-agency-watchers-v1.md decision 2, Tier 1 §6.1). Fully wired end to end: `polls:fetch` → `polls:extract` (a built extractor exists for TR) → operator review → `polls:accept`. |
| `polls_alpha_research` | `update-polls` — Алфа Рисърч's site watcher (custom-CMS blog listing; the site injects casino-spam links on its LISTING page, filtered by `isOwnPostLink()` before any article is fetched — decision 2). Fully wired end to end, same as `polls_trend` above (a built extractor exists for AR). |
| `polls_market_links` | `update-polls` — Маркет ЛИНКС's site watcher (custom-CMS news listing; every publication is a PDF, resolved at ingest — decision 2). `polls:fetch` captures it; ⚠️ **`polls:extract` has no built extractor for ML yet** (the aligned-row rule, §6.2/Tier 2c) — treat a flip here as a signal for an OPERATOR to read the captured PDF and hand-write the inbox draft, or wait for the extractor. |
| `polls_sova_harris` | `update-polls` — Сова Харис's site watcher (per-year WordPress categories resolved by NAME each run; exit polls excluded — decision 17). `polls:fetch` captures the bulletin page images; ⚠️ **`polls:extract` has no built extractor for SH yet** (the OCR+table rule, §6.2/Tier 2c) — same manual-draft fallback as `polls_market_links` above. |
| `polls_myara` | `update-polls` — Мяра's site watcher (fixed BG/EN electoral-category ids; every post in scope is electoral — decision 2). `polls:fetch` captures it; ⚠️ **`polls:extract` has no built extractor for MY yet** (the sentence rule, §6.2/Tier 2c — the same rule already built for TR/AR) — same manual-draft fallback as `polls_market_links` above. |
| `polls_global_metrics` | `update-polls` — Глобал Метрикс's site watcher, globalmetrics.eu (publisher of the first 2026 presidential poll — decision 2, §7). `polls:fetch` captures it; ⚠️ **`polls:extract` has no built extractor for GM yet**, AND its one publication so far is presidential — `polls:accept` refuses presidential drafts outright until Tier 4 ships. Will be inboxed once an extractor exists; will not be scored until then. |
| `polls_gallup` | `update-polls` — Галъп's TWO-ARMED watcher: the site lister (gallup-international.bg, TLS-broken since ≥2026-09-05) union'd with a Google News RSS press query. A single arm failing is recorded in `meta.armErrors` and reported from the other arm alone; only BOTH arms failing throws (decision 2, §6.1). `polls:fetch --archive <waybackUrl>` works around the TLS break; ⚠️ **`polls:extract` has no built extractor for GIB yet** (the sentence rule, §6.2/Tier 2c — the same rule already built for TR/AR) — same manual-draft fallback as `polls_market_links` above. |
| `polls_press` | `update-polls` — the press-discovery watcher for every site-less agency (Медиана, АФИС, ЦАМ, Екзакта, Барометър България, ИМП, Online Solutions) — one Google News RSS query per agency; Gallup is deliberately excluded (it owns the `polls_gallup` row above, so no poll is reported twice). A flip here maps to third-party verification (the skill's Step 2: resolve the real article URL by hand, capture with `--url`, lock as `third_party_consensus` only when a second independent outlet agrees on every figure). ⚠️ **None of these agencies has a built extractor either** — the draft needs hand-writing from the captured article, same as the other unbuilt agencies above. |
| `wiki_governments` | _no skill — manual edit of `data/governments.json` required; see "Governments: manual edit first" below_ |
| `cacbg_declarations` | `update-connections` |
| `cacbg_officials` | `update-officials` |
| `cacbg_local` | `update-officials` (Step 1b — municipal ingest) |
| `egov_commerce` | `tr-daily-refresh` (`npm run tr:daily-refresh` — refresh index, fetch new TR daily filings, reconstruct `state.sqlite`, rebuild per-EIK `company-connections/`, **then reload the local Postgres `tr_companies`/`tr_officers`/`tr_person_roles` tables via `db:load:tr:pg`** — chained into the npm script so PG never goes stale; marker `state/ingest/tr-daily-refresh.json`). **NB — this is also the NGO register refresh:** the shared ТРРЮЛНЦ feed carries сдружения/фондации/читалища; the reconstruct captures their governing bodies (`ngo_board`/`ngo_representative`/`trustee`/`verifier`), objectives + public-benefit status and member nationality, and `db:load:tr:pg` derives `entity_class`/`ngo_type`, loads `ngo_details` + `country`, (re)builds the awarder K-Index matview, and — from the fresh officers — rebuilds `ngo_board_links` (the magistrate-on-NGO-board leg via `rebuild_ngo_board_links()`) + REFRESHes the `ngo_signals` matview (migration 080; the NGO public-interest signals on `/procurement/ngos` + `/company/:eik`). No separate step or JSON — DB-only. **NB the *official* board-link leg + the official_roster it matches against are refreshed only by `db:load:ngo-board-links` — run it after `update-officials` (below), not on this daily flip.** |
| `ec_fts` | `db:load:ngo-funding:pg` (`npm run db:load:ngo-funding:pg` — a new FTS year fired: download `{YEAR}_FTS_dataset_en.xlsx` into `raw_data/ngo_funding/fts/`, then load. Rematches EU-direct funds → NGO EIK via VAT→exact-fold→fuzzy into the `ngo_funding` table; DB-only, no JSON/shards. Also picks up curated `data/ngo/budget_subsidies.json`. Marker `state/ingest/db-load-ngo-funding.json`.) **THEN regenerate the /procurement hub blob** — `ngo_funding` is one of its five inputs (it backs the `ngos` tile), and nothing else in this row's path rebuilds it: `npm run db:gen-hub-stats` → commit `data/procurement/derived/hub_stats.json` → `npm run bucket:sync:paths -- procurement/derived/hub_stats.json`. ⚠️ A green `db:check-generated` does NOT mean it is current — see the blind-spot table in Procedure step 8. |
| `bgpost_postcodes` | `update-connections` (re-runs `scripts/parliament/build_postcode_ekatte.ts` before the declarations pipeline so the postcode→EKATTE table is fresh for the HQ-resolution step) |
| `egov_procurement` | `update-procurement` **+ textbook-market refresh** — the CPV-22112 slice on `/awarder/000695114` is built from the procurement corpus, so after the АОП re-ingest re-run `npx tsx scripts/education/gen_textbook_market.ts` then `npm run bucket:sync:paths -- education` (it's GCS-served). This is the ONE lightweight step of `update-schools`, not the whole pipeline — don't queue the full `update-schools` here (that needs the МОН ДЗИ CSVs and is driven by `indicators_mon_dzi`). |
| `aop_debarred` | `update-procurement` (Step 5 — debarred-list refresh) |
| `kzk_appeals` | `update-kzk-appeals` (headed Playwright + BG egress; pins the local DB, asserts the 2,098 irreplaceable outcomes, rebuilds `kzk:summary`). Stamp the marker `kzk_appeals`, not the skill name. NOT `update-procurement` — that would enqueue the full АОП re-ingest and skip the КЗК crawl entirely. |
| `kzk_decisions` | `update-kzk-appeals` **Step 1b** (the MERITS arm) — `npm run kzk:decisions -- --year <YYYY> --apply`, then `db:load:kzk-decisions:pg` + `kzk:rejoin -- --apply`, all DB writes pinned to the local URL. Do NOT settle for a Step 1 re-run: the intake crawl's `COALESCE` upserts cannot write an outcome. Stamp the marker `kzk_decisions`, not `kzk_appeals`. |
| `eop_procurement` | `update-procurement` (Step 1b — ЦАИС ЕОП flat-договори gap-fill: `npx tsx scripts/procurement/ingest_eop.ts --self-heal --apply` — the covered-buyer gap-heal (cross-source-dedup over a ~75-day window that spans АОП's OCDS-export lag; double-count-safe via the `ingest.ts` EOP-twin eviction). Shipped 2026-07-26; replaces the old plain-gap-fill + `--only-buyers` infra-recovery two-step (SIGMA-parity P1; see Step 1b); **Step 1e also (re)applies the current-value fold** `ingest_anexi.ts` (incremental, no `--apply` flag) + `anexi_current_value.ts --apply` then `rebuild_from_cache.ts` — `amountEur` is the post-annex CURRENT value (SIGMA basis) and base re-ingest resets it, so the fold must run last; **Step 1c — buyer geo-enrichment from the same storage.eop.bg buckets:** `build_ocds_party_geo.ts` (OCDS party addresses → settlement) + `build_tender_oblast_map.ts` (tenders → oblast) + `awarder_geo_map.ts` (combines tiers → `awarder_geo_overrides.json`); then rebuild via `npm run procurement:ingest`) |
| `procurement_scope_windows` | _direct command — no skill: `npm run db:load:procurement-scopes:pg:cloud` (46 s local, **87 s on Cloud SQL** — measured 2026-08-22 on `db-perf-optimized-N-2`; the earlier "expect minutes" warning was sized on the retired db-g1-small. It still rebuilds six per-scope matviews and a plain `REFRESH` still takes an AccessExclusiveLock, so `/procurement/by-settlement` and every settlement page still block for that window). Nothing is ingested: the flip means the Jan-1 calendar rollover or a new election in `src/data/json/elections.json` added a `?pscope=` window the precomputes (119/122/123/124) do not cover yet, so that scope serves an EMPTY page at a 200 on `/procurement/by-settlement`, `/procurement/contractors`, every `/procurement/settlement/:ekatte` and the six `/api/db/procurement-*` dashboard routes. Run the LOCAL twin (`npm run db:load:procurement-scopes:pg`) too — it IS in the `db:refresh` chain, but nothing triggers a refresh on a calendar rollover, so local goes stale the same way and every `.data.test.ts` over those matviews measures the old window set. Then stamp with `npx tsx scripts/stamp-ingest.ts procurement_scope_windows --summary "…"`; nothing else clears it, so an unstamped flip re-surfaces on every run. NOT `update-procurement` — no corpus moved, and that would queue a ~90-minute АОП re-ingest for a window-set change._ |
| `mon_ri_register` | Re-crawl the МОН institution register ЕИК→EKATTE crosswalk (schools/kindergartens): `npx tsx scripts/procurement/mon_ri_crawl.ts` (**headed Playwright**, ri.mon.bg behind Cloudflare, ~1 min), then `npx tsx scripts/procurement/awarder_geo_map.ts`. That rebuilds the override map; to reach `by_settlement` it needs an ingest, and it must be the **FULL current-value chain** from `update-procurement` — a bare `procurement:ingest` recomputes `amountEur = toEur(amount)` and silently drops the post-annex fold (~€1.75bn), at exit 0, with the euro-peg canary green. This is the **Tier R** exact-EIK source that resolves the ambiguous shared-name school/kindergarten buyers. It is ALSO **Tier B**'s only liveness probe — `awarder_geo_map.ts` reads the same data.egov.bg resource directly, so an ERROR on this source means Tier B is down now, not merely that the crosswalk may drift. NOT `update-procurement` (that runs the full АОП re-ingest and skips the crawl). Stamp `state/ingest/mon_ri_register.json`. See [[reference_mon_ri_register]]. |
| `isun_eu_funds` | `update-funds` (**PG-served, not GCS** — the run publishes funds via `db:load:funds:pg` locally + `db:load:funds:pg:cloud -- --full` to prod Cloud SQL — the scope flag is required, see step 9's table; `bucket:sync` **excludes** `funds/`) |
| `isun_eu_funds_projects` | `update-funds` (Step 2 — contract-level ingest; runs `npm run funds:ingest-projects` after the beneficiaries ingest, then `npx tsx scripts/funds/build_geo_pins.ts` to rebuild the slim per-município `by-muni-geo` files — top-200 contracts; the My-Area projects tile renders the list and an on-demand map of the geocoded subset. **PG-served** — publish via `db:load:funds:pg:cloud -- --full` (the scope flag is required — see step 9), NOT `bucket:sync`.) |
| `keep_eu_interreg` | `update-funds` (Step 3c — Interreg via keep.eu. **PG-served** — publish via `npm run db:load:interreg:pg:cloud`, then `npm run db:load:graph:pg:cloud` so the fourth arm of `company_public_money` picks the new corpus up, and `db:load:tr-company-place:pg:cloud` after that since its `money_eur` is denormalized from it. NOT `bucket:sync` — `data/funds/` is excluded.) |
| `isun_procedures` | `update-open-calls` (the **OPEN** half of ИСУН, not the awarded corpus — `npm run opencalls:isun` then `npm run db:load:open-calls:pg`. **PG-served, and the cloud publish is mandatory:** `npm run db:load:open-calls:pg:cloud`; nothing runs it automatically and skipping it leaves prod on the previous vintage at a 200. `bucket:sync` **excludes** `opencalls/` — it is a PG load source, not a serving artifact. Note the fingerprint is the listing GUID SET, so "unchanged" means no procedure opened or closed — it does NOT mean nothing on an existing procedure's detail page moved. Do **not** run `enrich-open-calls` from here: extraction spends tokens per document and must not publish a figure without human sign-off.) |
| `sp2023_indicative` | `update-open-calls` (Step 2 only — `npm run opencalls:sp2023`, then load + `db:load:open-calls:pg:cloud`. The ДФЗ CAP Strategic-Plan indicative schedule: month RANGES, not deadlines, so these rows are `date_precision='indicative'` and must never render a countdown. Its money IS trustworthy (`enrichment='source'` — the XLSX has real budget/aid-rate/ceiling columns), unlike the ИСУН rows. Skip this when only `isun_procedures` was flagged; the schedule is annual.) |
| `egov_budget_execution` | `update-budget` (December snapshot closing a FY → also re-run `scripts/budget/run_policy_baseline.ts`) |
| `municipal_fiscal_due` | `update-budget` (municipal-fiscal sub-step; own marker `update-budget-municipal-fiscal`) — **MANUAL DOWNLOAD FIRST**; see "Municipal fiscal: manual download first" below. Do NOT invoke it while the workbook is missing. |
| `ministry_execution_reports` | `update-budget` |
| `iisda_doklad` | `update-budget` + `update-administration` |
| `iisda_services` | `update-administration` |
| `eurostat_egov` | `update-administration` |
| `eurostat_digital_skills` | `update-administration` |
| `asp_benefits` | `update-social` |
| `git_inspections` | `update-social` |
| `customs_revenue` | `update-budget` (revenue-breakdown sub-step — re-runs `scripts/budget/run_customs_revenue.ts`, then `scripts/budget/run_policy_baseline.ts` for the excise-lever anchors. **Also** — same March cycle — hand-refresh `data/budget/agencies/customs.json` from the customs.bg „Отчет за касовото изпълнение“ Б-3 „Годишен уточнен план“ разходи, then rerun `npm run db:gen-sector-stats` — this is the `customs` tile on /governance/sectors. See update-budget § "Agency budgets".) |
| `nap_annual` | `update-budget` (revenue-breakdown sub-step — re-runs `scripts/budget/run_nap_annual.ts`, then `scripts/budget/run_policy_baseline.ts`. **Also** hand-refresh `data/budget/agencies/nap.json` from the same report's „Отчет на разходите на НАП / Уточнен план“ table, then rerun `npm run db:gen-sector-stats` — this is the `revenue` tile on /governance/sectors. See update-budget § "Agency budgets".) |
| `customs_excise_register` | _direct command — no skill: `npm run customs:excise-register` (fetches the Агенция „Митници“ BACIS register, rewrites `data/customs/excise_register.json` + the geolocated `data/customs/excise_warehouses.json`), then `npm run db:load:excise-warehouses:pg` to reload the warehouse-map table (schema 072). Feeds the Митници pack register band + warehouse count map + `/customs/warehouses` + the `exciseRegister` AI tool. Prod: `bucket:sync` `data/customs/` (register JSON) AND `npm run db:load:excise-warehouses:pg:cloud` (map table) after._ |
| `eurostat_policy` | `update-budget` (policy-baseline sub-step — `scripts/budget/run_consumption_coicop.ts` + `scripts/budget/run_policy_baseline.ts`) |
| `policy_baseline_local` | `update-budget` (policy-baseline sub-step — `scripts/budget/run_policy_baseline.ts`; if describe says a new НОИ STATB quarter published, bump `NOI_STATB_URL` first) |
| `eu_tax_rates` | _auto-check: `npx tsx scripts/budget/check_policy_anchors.ts --source eu_tax_rates --stamp` (self-stamps on PASS); manual edit of `src/lib/euPolicyPresets.ts` only on DRIFT — see "Simulator anchors" below_ |
| `eu_excise_rates` | _auto-check: `npx tsx scripts/budget/check_policy_anchors.ts --source eu_excise_rates --stamp` (VALUE-level vs Tax Foundation; self-stamps on PASS); manual edit of the `exDiesel`/`exPetrol`/`exCigarettes` values in `src/lib/euPolicyPresets.ts` only on DRIFT — see "Simulator anchors" below_ |
| `eu_alcohol_excise` | _auto-check: `npx tsx scripts/budget/check_policy_anchors.ts --source eu_alcohol_excise --stamp` (VALUE-level vs EC TEDB; self-stamps on PASS); manual edit of the `exSpirits`/`exWine` values in `src/lib/euPolicyPresets.ts` only on DRIFT — see "Simulator anchors" below_ |
| `oecd_pit_params` | _no auto-checker (drift-only) — flags when OECD PIT thresholds/allowances/credits move; re-derive the affected `nm`/`b2.t2` in `src/lib/euPolicyPresets.ts`; see "Simulator anchors" below_ |
| `oecd_family_leave` | _no auto-checker (drift-only) — flags when OECD paid-leave weeks move; re-derive the affected `mat` in `src/lib/euPolicyPresets.ts`; see "Simulator anchors" below_ |
| `nato_defence` | _auto-check: `check_policy_anchors.ts --source nato_defence --stamp` (edition-level; self-stamps on PASS); a NEW edition needs a human read of Table 3 → def options + `NATO_COMPENDIUM_EDITION` — see "Simulator anchors" below_ |
| `ec_forecast_bg` | _auto-check: `check_policy_anchors.ts --source ec_forecast_bg --stamp` (edition-level; self-stamps on PASS); a NEW edition needs a manual update of the EC baseline + `EC_FORECAST_EDITION` in `src/lib/bgFiscalProjection.ts` — see "Simulator anchors" below_ |
| `nsi_edp` | _no skill — manual edit of `src/lib/bgFiscalProjection.ts` (ESA outturn anchors); see "Simulator anchors: manual edits" below_ |
| `ec_vat_gap` | _no skill — manual edit of `src/lib/bgBehavioral.ts` (`VAT_GAP_RESPONSE`); see "Simulator anchors: manual edits" below_ |
| `imf_weo_bg` | _no skill — manual re-verify of `src/lib/bgBehavioral.ts` (`MULT_\*` bands); see "Simulator anchors: manual edits" below\_ |
| `fiscal_council_bg` | _no skill — benchmark strings + `docs/budget_simulator_grounding.md` (+ dividend calibration in `src/lib/bgBehavioral.ts` when relevant); see "Simulator anchors: manual edits" below_ |
| `api_road_charges` | _no skill — manual edit of `ROAD_CHARGES_BASE_EUR` in `src/lib/bgTaxPolicy.ts` + the `API_ROAD_CHARGE_LATEST_YEAR` floor in `scripts/watch/sources/fiscal_anchors.ts`; see "Simulator anchors: manual edits" below_ |
| `nssi_b1` | `update-noi` (run `tsx scripts/budget/noi/__write_funds.ts --fetch` — auto-downloads the new B1 XLS files into `raw_data/budget/noi/` via a Referer-header workaround that defeats NSSI's redirect-to-homepage on GET, then rebuilds `funds.json`) |
| `nssi_yearbook` | `update-noi` (the pension statistical yearbook ZIP — `Yearbook_Pensions_{YYYY}.zip` — GETs cleanly, so auto-fetch: download into `raw_data/budget/noi/yearbooks/` and run `tsx scripts/budget/noi/__write_pensions.ts` → `data/budget/noi/pensions.json`, the /pensions view's oblast + distribution + national series. Then commit + `bucket:sync data/budget/noi/`) |
| `kfn_pensions` | `update-noi` (КФН private-pension quarterly ZIP — pillars 2 & 3. Download the ZIP from the newest `fsc.bg/.../statistics/YYYY-N/` sub-page into `raw_data/budget/kfn/`, run `tsx scripts/budget/kfn/__write_funds.ts` → `data/budget/kfn/funds.json`, the /pensions private-funds tile. Then commit + `bucket:sync data/budget/kfn/`) |
| `dv_investment_annex` | `update-budget` (investment-program sub-step — operator adds the new fiscal year to both `INVESTMENT_ANNEX_URLS` in `scripts/watch/sources/dv_investment_annex.ts` AND the `SOURCES` map in `scripts/budget/investment_program/__write_program.ts`, fetches the PDF into `raw_data/budget/investment_program/{year}-annex-iii.pdf`, then runs `scripts/budget/investment_program/__write_program.ts`) |
| `ipop_mrrb` | `update-budget` (IPOP execution sub-step — fetch new CSV with `curl -o raw_data/budget/ipop/projects-{year}.csv 'https://ipop.mrrb.bg/reports_projects_export.php'`, then `tsx scripts/budget/ipop/ingest.ts`; daily cadence — every change flips this fingerprint, but the data still moves at workdays-only pace in practice) |
| `capital_programs` | `update-budget` (capital-programmes sub-step — see "Capital-programmes ingest" below; covers Sofia, Plovdiv, Burgas, Stara Zagora, Ruse, Varna, Pleven — Varna and Pleven need a Gemini Vision OCR pre-step) |
| `smetna_palata` | `update-financing` |
| `financing_reports` | `update-financing` |
| `erik_campaign_financing` | `update-financing` (ЕРИК campaign-finance ingest — runs `npm run data -- --erik --financing`; scrapes erik.bulnao.government.bg for the latest election's donors/candidates/filings and re-parses) |
| `eurostat` | `update-macro` (also rebuild the administration page-context, whose GF01/population slices come from macro+cofog: `npm run admin:build-context`) |
| `ec_budget_per_ms` | `update-macro` |
| `worldbank_wgi` | `update-macro` |
| `transparency_cpi` | `update-macro` (gated — see "TI CPI: manual paste first" below) |
| `bnb_auctions` | `update-macro` (debt-emissions sub-step — re-runs `fetch_bnb_auctions.ts`) |
| `minfin_eurobond` | `update-macro` (Eurobond sub-step — hand-add new international Eurobond to `data/debt-emissions.json`; see below) |
| `minfin_mreports` | `update-macro` (fiscal-reserve sub-step — re-runs `fetch_fiscal_reserve.ts`) |
| `minfin_program_otchet` | `update-budget` (operator backfills the new MoF ProgOtchet PDF from Wayback into `raw_data/budget/exec-admin-ministerstvoto-na-finansite-<fy>.pdf` and activates the matching `manual-pdf` entry in `EXECUTION_REPORTS`) |
| `mfa_program_otchet` | `update-budget` (when a new MVnR fiscal year lands in Wayback, add an `xlsx-in-zip` entry to `EXECUTION_REPORTS`. The describe-line shows the latest period; the FY2023 entry in `fetch_sources.ts` is the template) |
| `budget_law` | `update-budget` (a new annual ЗДБ or interim bridging law landed in ДВ — resolve its `idMat` and add a row to `LAW_DV_MATERIALS` or `INTERIM_BUDGET_LAWS` in `scripts/budget/fetch_sources.ts`, then re-ingest. Wayback-lagged; cross-check dv.parliament.bg + minfin.bg/bg/1770). **ALSO re-verify the simulator's statutory constants** in `src/lib/bgTax.ts` + `src/lib/bgTaxPolicy.ts` — the 2026 ЗДБРБ and ЗБДОО pass as one package; the ЗБДОО sets the МОД cap, SSC/health rates, min/max pension, min self-insured income, child relief, the 2nd-year maternity benefit, plus the party-subsidy, the gambling ЗХ variable fee (`GAMBLING_GGR_FEE_RATE` in `bgTaxPolicy.ts` — set by the budget package, 20→25% in Budget 2026; its GGR _base_ `GAMBLING_GGR_EUR` is a medium-confidence НАП/industry estimate, not legislative, so it is NOT re-checked here), and the min-wage levers. Check each against the new ЗБДОО text in ДВ and the НАП "осигурителни вноски" table, update the constants, and re-run the `__smoke_*` budget tests (and `__test_engine.ts` if the gambling fee moved). As of 2026-07-28 the ЗБДОО/ЗБНЗОК halves ARE adopted (ДВ бр. 68) and the constants have been updated: МОД €2,300 from 01.08, min pension €347.51 from 01.07 (the €346.87 in circulation is the DRAFT figure), min self-insured €620.20 from 01.08, maternity €398.81 unchanged, and the +2pp pension-contribution hike was DROPPED. Mid-year steps live in the `*_SCHEDULE` tables in `bgTax.ts`, not in the scalars. The ЗДБРБ half is still pending, so whatever it sets (child relief, party subsidy, ЗХ gambling fee) remains carry-over. **ALSO run `npx tsx scripts/transport/parse_rail_subsidy.ts`** — the /sector/transport rail-subsidy tile parses the SAME cached ЗДБ HTML (БДЖ PSO + НКЖИ subsidy lines) into `data/transport/rail_subsidy.json`; commit it (no bucket sync, no PG). |
| `dv_laws` | `update-budget` (a law from the budget package was promulgated in ДВ — the describe-line names which: `ЗДБРБ`, `ЗБДОО`, `ЗБНЗОК` or `удължителен`. This is the AUTHORITATIVE promulgation signal; `budget_law` is a Wayback-lagged minfin proxy that only ever sees the ЗДБРБ half. Resolve the `idMat` on dv.parliament.bg — the RSS carries none — and add the row to `LAW_DV_MATERIALS` (annual ЗДБРБ) / `AMENDMENT_DV_MATERIALS` (its ЗИД) / `INTERIM_BUDGET_LAWS` (bridging law) in `scripts/budget/fetch_sources.ts`, then re-ingest. For a `ЗБДОО` or `ЗБНЗОК` there is no appropriation table to parse — the action is to **re-verify the simulator's statutory constants** in `src/lib/bgTax.ts` + `src/lib/bgTaxPolicy.ts` against the new ДВ text, exactly as the `budget_law` row describes. A `непълен пакет <year>: липсва(т) X` note (also surfaced by `describe` as `непълен бюджетен пакет за <year> г.`) means the annual package (ЗДБРБ + ЗБДОО + ЗБНЗОК) is only PARTLY promulgated — e.g. ДВ бр. 68 от 2026 carried ЗБДОО + ЗБНЗОК but not the ЗДБРБ. Ingest the members that landed, but do **NOT** record the budget package as ingested and do **NOT** treat appropriation-dependent artifacts (the чл. 1 framework, издръжка, judiciary, Приложение № 3, the simulator's ЗДБРБ-set constants) as current — the watcher flips again when the missing member lands, and only then is the package complete. A `неинспектирани броеве: N` note means ДВ issue N published while the watcher was down and has rolled out of the one-issue RSS window — open that брой manually on dv.parliament.bg and check its official section for a budget law.) |
| `eurostat_regional` | `update-regional` |
| `nsi_regional` | `update-regional` (NSI JSON-stat open-data — FDI id=629, museum visits id=844, hospital beds id=1206, crude death rate id=1139 — merged via `fetch_nsi.ts`) |
| `indicators_az` | `update-indicators` + `update-regional` |
| `indicators_mon_dzi` | `update-indicators` (município-grain ДЗИ) **+ `update-schools`** (the per-school /education + /school pipeline: `fetch_nvo.ts` → `build_index.ts` → `gen_school_context.ts` → `gen_textbook_market.ts`; reuses the per-school МОН ДЗИ CSVs in `raw_data/indicators/mon/`, plus fetches 7th-grade НВО for value-added). Queue both, dedupe. The textbook-market blob also refreshes when the procurement corpus changes (see `data.egov.bg АОП`). |
| `indicators_mon_nvo` | `update-schools` (7th-grade НВО is the prior-attainment baseline for the value-added on `/school/:id`; `update-schools` re-pulls all НВО years via `fetch_nvo.ts`). Usually fires the same day as `indicators_mon_dzi` — **dedupe** to one `update-schools` run. |
| `indicators_nsi_pop` | `update-indicators` |
| `indicators_nsi_vital` | `update-indicators` |
| `nsi_landuse` | `update-landuse` (operator pastes the new year's PDF URL into `LANDUSE_REPORTS` in `scripts/landuse/sources.ts`, then runs `npx tsx scripts/landuse/fetch.ts --year <year>`; ingests the 28-oblast land-use composition published annually as NSI's Баланс на територията press-release annex) |
| `grao` | `update-grao` |
| `cik` (if re-enabled) | _no skill yet — surface as TODO_ |
| `iisda_mayors` | `update-municipal-contacts` (re-runs `npx tsx scripts/officials/municipal_contacts/scrape_iisda.ts` — scans the governing_body ID range 4400..4950, caches HTML to `raw_data/officials/iisda_mayors/`, walks every `<li class="level-1">` block to extract mayor + every Заместник-кмет (role, name, email), rewrites `data/officials/municipal_contacts/index.json` with `{email, mayor, iisda_id, officials[]}` per município) |
| `iaos_air_quality` | `update-air-quality` (re-runs `npx tsx scripts/air/build_index.ts` — fetches the latest PM10/PM2.5 quarterly CSVs from data.egov.bg, parses station names → município codes, rewrites `data/air/index.json`) |
| `ti_bg_lisi` | `update-transparency-lisi` (gated — see "TI-BG LISI: manual paste first" below. lisi.transparency.bg is a heavy SPA; operator copies the new year's 27 município scores into the `LISI_2024`-style array in `scripts/transparency/build_lisi.ts`, then runs `npx tsx scripts/transparency/build_lisi.ts`) |
| `vss_court_statistics` | `update-judiciary` (the ВСС published new annual court statistics. FIRST add the new year's PDF URL to `VSS_ANNUAL_TABLES` in `scripts/judiciary/sources.ts` — the filenames are not uniform across years — THEN run `npx tsx scripts/judiciary/__write_caseload.ts` (Приложение № 1 → `data/judiciary/caseload.json`) AND `npx tsx scripts/judiciary/__write_court_load.ts` (Приложение № 2, per-court натовареност → `data/judiciary/court_load.json`; run it AFTER caseload — it reconciles against it), both with Σ-reconciliation asserts. `court_load.json` is PG-served, so also `npm run db:load:court-load:pg` then `npm run db:load:court-load:pg:cloud`. Commit both JSONs + `bucket:sync data/judiciary/`.) |
| `ivss_declarations` | `update-judiciary` (the ИВСС register gained a year, or a non-compliance list changed. Run `npx tsx scripts/judiciary/__write_declarations.ts` — it discovers years from the register index, no curated map needed — then `npx tsx scripts/judiciary/__write_magistrate_holdings.ts` for the declared-companies/financials artifact `data/judiciary/magistrate_holdings.json`, which is PG-served, so also `npm run db:load:magistrates:pg` then `npm run db:load:magistrates:pg:cloud`. ⚠️ **BOTH must be followed by `npm run db:load:magistrate-filing-assets:pg` and `npm run db:load:magistrate-filing-assets:pg:cloud`** — the magistrates loader TRUNCATEs `magistrate` and takes the per-filing declaration metadata plus `magistrate.real_estate_count_parsed` with it, so the property count goes blank on every magistrate card until this repair runs. The mechanism, the measured counts and the verification queries live in ONE place — `update-judiciary`, under its Cloud SQL publish block — rather than being restated here. READ THE DIFF before committing — the register is fetched over plain HTTP from a bare IP, so a changed name is a claim about a private individual. **`declarations.json` is PG-served since persons-pg-retirement-v1 T2.6** — `/judiciary` + the `judiciaryDeclarations` AI tool read `/api/db/judiciary-declarations` (`judiciary_payloads`, migration 109), and the file is excluded from `bucket:sync` — so after `__write_declarations.ts` also run `npm run db:load:judiciary-payloads:pg` (local) + `db:load:judiciary-payloads:pg:cloud` (prod), else the live register stays stale. Commit + `bucket:sync data/judiciary/` (still ships caseload/court_load, not declarations.json).) |
| `nato_defexp` | `update-defense` (a new NATO "Defence Expenditure of NATO Countries" edition landed. Download the latest `def-exp-*-en.pdf`, `pdftotext -layout`, take Bulgaria's row from Table 3 (%GDP) and Table 8a (equipment/personnel) into `data/defense/gdp_share.json` + `category_split.json`. Read the TABLES, not the vector charts. Commit + `bucket:sync data/defense/`.) |
| `mod_defense_report` | `update-defense` (the МО published a new "Доклад за състоянието на отбраната". Parse the personnel-vacancy and reserve-fill figures into `data/defense/readiness.json` (FlateDecode PDF → `pdftotext -layout`). Commit + `bucket:sync data/defense/`.) |
| `moe_arms_exports` | `update-defense` (a new Ministry of Economy arms-export report landed. Refresh `data/defense/exports.json` with the euro values by year — NOT SIPRI TIV, which excludes ammunition and undercounts. Commit + `bucket:sync data/defense/`.) |
| `eurostat_road_safety` | run `npx tsx scripts/security/fetch_road_safety.ts` (Eurostat `sdg_11_40` republished the road-death series → refresh `data/security/road_safety.json`, the /sector/security road-safety outcome tile. Small committed file — commit it; no bucket sync, no PG, no separate skill.) |
| `eurostat_rail` | run `npx tsx scripts/transport/fetch_rail_ridership.ts` (Eurostat `rail_pa_total` republished the rail-passengers series → refresh `data/transport/rail_ridership.json`, the denominator of the /sector/transport subsidy-per-passenger tile. Small committed file — commit it; no bucket sync, no PG, no separate skill.) |
| `eurostat_tourism` | run `npx tsx scripts/tourism/fetch_eurostat_tourism.ts` (Eurostat `tour_occ_nim` / `tour_occ_ninraw` republished the nights-spent series → refresh `data/tourism/visitors.json`, the /sector/tourism seasonality + origin-markets charts. Small committed file — commit it; no bucket sync, no PG, no separate skill.) |
| `eurostat_env` | run `npx tsx scripts/environment/fetch_waste.ts` (Eurostat `cei_wm011` / `env_wasmun` / `env_bio4` republished → refresh `data/environment/waste.json`, the /sector/environment recycling-vs-target + protected-area tiles. Small committed file — commit it; `bucket:sync data/environment/` for prod; no PG, no separate skill.) |
| `nfc_film_register` | `update-culture` (the НФЦ re-uploaded a film register or added a new year. Bring local Postgres up FIRST (`npm run db:pg:up`), then run `npx tsx scripts/culture/ingest.ts` — it re-downloads changed .xls files, parses both format families and rewrites `data/culture/films.json` + `overview.json` with Σ-reconciliation asserts. The ingest also re-links the top producers to their company EIK from `tr_companies` before writing; if Postgres is down it prints `⚠ producer→EIK linking SKIPPED`, carries the previous EIKs forward, and you must repair with `npx tsx scripts/culture/enrich_producers.ts`. Verify no `eik` was dropped: `git diff -U0 data/culture/overview.json \| grep '^-.*"eik"'` must print nothing. Commit + `bucket:sync data/culture/`.) |
| `mc_dki_register` | `update-culture` (МК added, renamed or removed a държавен културен институт on one of its three ДКИ listing pages. Bring local Postgres up FIRST (`npm run db:pg:up` — the ingest resolves every institute to an EIK against `contracts` ∪ `tenders` and REFUSES to write if the corpus is empty), then `npm run culture:dki` for a dry run and `npm run culture:dki -- --apply` to rewrite `data/culture/dki_register.json`. Then `npx vitest run scripts/culture/dki/` — the reconciliation gate FAILS when МК lists a ДКИ our roll-up does not carry and nobody has ruled on it; add the EIK to `DKI_DISAGREEMENTS` in `scripts/culture/dki/reconcile.ts` with a reason, or move it in `src/lib/kulturaReferenceData.ts`. Moving one changes the sector headline €, so it is a human call. Commit + `bucket:sync data/culture/`.) |
| `cprs_register` | `update-procurement` (ЦПРС — Камарата на строителите re-published the builder register. Re-run **`npm run cprs:ingest -- --apply --refresh`** — a 1,620-query cartesian crawl, 30 области × 54 licence classes, ~42 s. ⚠️ **`--refresh` is not optional on a flip**: every cell is cached under `raw_data/procurement/cprs/` so a run interrupted by a timeout can resume, and without the flag every cached cell is reused — the run re-parses the PREVIOUS crawl in ~6 s, fetches nothing from register.ksb.bg and republishes the old register at exit 0 (measured 2026-09-08: 106,508 licences / 8,379 firms without the flag against 107,034 / 8,414 with it). The ingest now prints a loud EVERY-CELL-CAME-FROM-THE-CACHE line and holds the artifact's `fetchedAt` at the cache vintage when that happens — read either as the step not having run. Resume survives the flag: after a timeout re-run the SAME command, since cells fetched within the last hour are still skipped. Then `npm run db:load:cprs:pg` and **`npm run db:load:cprs:pg:cloud`**. Without the cloud load the eligibility check on every works contract silently keeps the previous vintage at a 200. Gate: `npx vitest run scripts/db/tests/cprs.data.test.ts`.) |
| `ted_bg` | `update-procurement` (TED published new Bulgarian notices. Re-run `npm run ted:ingest -- --apply` (⚠️ **dry-run by default** — without `--apply` it prints the full notice/buyer summary and exits **0** having written nothing, so the loader that follows silently republishes the previous vintage; 429-aware, honours `Retry-After`, concurrency 1), then `npm run db:load:ted:pg` and **`npm run db:load:ted:pg:cloud`**. ⚠️ Empty years are DROPPED, never stored as zero — a stored zero plots as „Bulgaria published nothing above the EU threshold that year", and 2015 genuinely returns 0 because the v3 index ramps. `data/procurement/ted.json` is gitignored (73 MB), so the cloud load is the only way the corpus reaches prod.) |
| `adfi_inspections` | `update-procurement` (АДФИ published new financial-inspection reports. Re-run `npm run adfi:ingest -- --apply` (⚠️ **dry-run by default** — without `--apply` it prints the counts, writes no `data/procurement/adfi.json` and exits **0**, so the loader that follows silently republishes the previous vintage), then `npm run db:load:adfi:pg` and **`npm run db:load:adfi:pg:cloud`**. ⚠️ This dataset is an ACCUSATION — a row says a named public body was inspected — so the name fold lives once in `adfiNameFold` and the gate imports it rather than restating it in SQL; a second copy once flagged 9 correct matches as false accusations.) |
| `aop_experts` | `update-procurement` (the АОП external-experts register под чл. 232а, ал. 2 ЗОП changed — and note the EXPECTED state of this source is „unchanged, forever": all 88 entries expired, the newest validity ended 2023-01-01 and nobody has been added since 2020-01-01. A fire here most likely means the register REOPENED, which invalidates the historical framing in migration 174, its data test and CLAUDE.md. Re-run `npm run aop:experts`, then `npm run db:load:aop-experts:pg` and **`npm run db:load:aop-experts:pg:cloud`**, and re-read the „is HISTORICAL" gate before publishing anything in the present tense.) |
| `ncf_grant_results` | `update-culture` (the НФК published a new grant класиране. FIRST add the new PDF URL to `NCF_RESULTS` in `scripts/culture/ncf_grants.ts` — the results are scattered across news posts, no clean index — THEN run `npx tsx scripts/culture/ncf_grants.ts`, which parses the ranking (applied vs funded → success rate) into `data/culture/grants.json`. Requires `pdftotext` (poppler). Commit + `bucket:sync data/culture/`.) |
| `nfc_commissions` | `update-culture` (the НФЦ published a new artistic-commission appointment order — a new ~6-month mandate. Download the newest "Назначаване съставите на НХК…" PDF from the nfc.bg заповеди page, read it with `pdftotext -layout`, and update the mandate window + 21 members in `scripts/culture/write_commissions.ts`; re-run it → `data/culture/commissions.json` (self-verifies 7 members + 1 chair per commission). Commit + `bucket:sync data/culture/`.) |
| `isun_clean_delivery` | `update-funds` (a listing pager moved — ~25 more EU projects closed clean. ⚠️ THE EXPORT IS OPERATOR-DOWNLOADED: the F5 WAF refuses automated downloads intermittently and blocks the `GetProgrammes` XHR outright, even from the page's own origin. Open both listings, click Експорт → Excel, and save as `data/_cache/isun_clean_delivery/contracts__ALL.xlsx` and `beneficiaries__ALL.xlsx`. Then `npm run funds:clean-delivery`, `npm run db:load:clean-delivery:pg` and **`npm run db:load:clean-delivery:pg:cloud`**. ⚠️ Do NOT partition the export per programme „to avoid a cap" — there is none; each listing's own pager confirms the export is complete, and the 4× gap between the two reports is two different populations. ⚠️ Absence from this register is NOT a financial correction — never derive one by subtracting it from `fund_projects`.) |
| `ipi_local_taxes` | `update-local-taxes` (re-runs `npx tsx scripts/local_taxes/build_index.ts` — fetches the 5 indicator CSVs from `265obshtini.bg/downloadCSV/{id}` covering all 265 общини × property/transfer/vehicle/patent-retail/patent-taxi taxes 2021–2025, normalises município names to obshtina codes, computes per-indicator national ranks ascending by rate, and merges into `data/local_taxes/index.json` — preserves the per-município `naredba` blocks the Tier B parsers wrote). Idempotent; safe to re-run. |
| `municipal_naredba` | `update-local-taxes` Tier B — `npx tsx scripts/local_taxes/run_naredba.ts <comma-list-from-describe-line>` runs only the parsers for the municípios the watcher flagged. Each parser fetches its mapped naredba sources (FEES PDF cached under `raw_data/local_taxes/naredba/<slug>.pdf`; obshtini.bg JSON cached as `<slug>.json`), extracts text via the shared pdftotext wrapper or the `lib/fetch_obshtini_bg.ts` JSON-API bridge, and writes `{tboResidential: {basis, rate?, note?}, touristTax?, dogTax?, propertyTaxIndividuals?: {rate, year}}` into the per-município shard `data/local_taxes/<obshtina>.json`. Per-município watermark at `state/ingest/local_taxes_<obshtina>.json` short-circuits unchanged sources via a SHA-256 of the source bytes. Coverage: SOF00 (FEES iisda + TAX obshtini.bg; property tax 1.875‰), PDV22 (FEES + TAX obshtini.bg; 1.8‰), VAR06 (FEES varna.bg + TAX obshtini.bg; 2.0‰), BGS04 (FEES DOCX + TAX legacy .doc via `lib/fetch_doc.ts` textutil shell-out; 1.75‰ pinned), RAZ26 (3‰), SFO39 (Samokov, 2.5‰), SZR22 (Мъглиж, 3.5‰), DOB03 (Балчик, 2.5‰), BLG33 (Петрич, 3‰), SZR31 (Стара Загора, 1.7‰) — the obshtini.bg-platform municípios use the `createObshtiniBgNaredbaParser` factory; Sofia/Varna/Burgas are bespoke (multi-platform fetch paths). Multi-source parsers populate `secondaryUrls` so the watcher fingerprints every upstream document. Add municípios by writing a new parser under `scripts/local_taxes/parsers/<obshtina>.ts`. |
| `kzp_prices` | `update-prices` (re-runs `npm run prices` — fetches the newly-published daily ZIP(s) from `kolkostruva.bg/opendata_files/<date>.zip`, loads ~1.4M store rows into **Postgres** as an SCD-2 delta (`price_facts` + `price_current` + `price_grid_days`), then rebuilds the canonical product catalogue, the `price_payloads` serving blobs and `data/prices/product_slugs.json`. **Postgres-only** — there is no `data/prices/{index,ranking,chains,settlement}.json` any more (migration 048). Idempotent. **Self-reports its `/data/updates` row**: the ingest calls `appendDataChange` when it loads a day, so DO NOT run `scripts/append-data-change.ts` for it (see the changelog note in step 5). One-off historical replay is the flag-gated `npm run prices -- --backfill --from 2026-01-02` operator step — never in the watcher.) |
| `ec_oil_bulletin` | run `npx tsx scripts/consumption/fetch_fuel.ts` (no update-* skill — the ingest **self-reports** its `/data/updates` row via `appendDataChange`, so skip the generic append; writes `data/fuel.json` for the /consumption/fuel tile). |
| `eurostat_food_pli` | `update-macro` (the macro refresh re-runs `scripts/macro/fetch_food_pli.ts` → the `foodPli` block in `data/macro_peers.json` for /consumption/eu). |
| `nzok_hospital_bmp` | `update-nzok` (re-runs `npm run data:nzok -- --hospitals` to refresh the static JSON, THEN — critically — `npm run db:load:nzok-hospital:pg` + `npm run db:load:nzok-hospital:pg:cloud` to reload the `nzok_hospital_payments` Postgres table, which is what actually serves the live pack + `/company/:eik` tiles. The `:cloud` loader IS the publish step; `npm run db:dump:cloud` afterwards only snapshots Cloud SQL to GCS for `db:restore` and is optional. Skipping the PG reload leaves both tiles stale on the old month. THEN, because a new monthly file can add/rename facilities, rebuild the hospital geo crosswalk: `npm run db:load:nzok-hospital-map:pg` + `npm run db:load:nzok-hospital-map:pg:cloud` (migration 075, `nzok_hospital_geo` — the marker map at the top of `/awarder/121858220`; must run AFTER both `nzok-hospital` and `awarder-seats`). Auto-fetch; then commit + `bucket:sync data/budget/nzok/`.) |
| `nzok_drug_quarterly` | `update-nzok` (re-runs `npm run data:nzok -- --drugs` — fetches the newest "Брутни разходи … по INN" XLS from `nhif.bg/bg/medicine_food/quarter-payments/{year}`, aggregates to top-25 INN + ATC groups (BGN→EUR) + a full-year-vs-full-year `growth` block (fastest-rising/falling/newly-reimbursed molecules from the two latest annual files), rewrites `data/budget/nzok/drug_reimbursement.json`, THEN — critically — `npm run db:load:nzok-drug-quarterly:pg` + `npm run db:load:nzok-drug-quarterly:pg:cloud` to reload the `nzok_drug_quarterly` Postgres table (migration 066), which is what actually serves the `nzok-drug-quarterly`/`-by-inn` routes (the INN-reimbursement tiles on `/awarder/121858220`). Skipping the PG reload leaves those tiles stale. Commit + `bucket:sync`.) |
| `nzok_execution_b1` | `update-nzok` (re-runs `npm run data:nzok -- --execution` — fetches the newest `B1_{YYYY}_{MM}_5600.xls` from `nhif.bg/bg/nzok/financial_report/quarter`, reads the ЕБК section totals, rewrites `data/budget/nzok/execution.json` + `execution_history.json` (every B1 month the page lists, for the plan-vs-actual pace chart). Commit + `bucket:sync`.) |
| `nzok_activities` | `update-nzok` (re-runs `npm run data:nzok -- --activities` — fetches the 12 monthly "Брой отчетени дейности по КП/АПр/КПр и брой ЗОЛ" XLSX from `nhif.bg/bg/hospitalcare-report/activities/{year}`, aggregates to the annual `(facility, procedure)` matrix + national monthly trend in `data/budget/nzok/activities.json`, THEN — critically — `npm run db:load:nzok-activities:pg` + `npm run db:load:nzok-activities:pg:cloud` to reload the `nzok_activities` Postgres table (migrations 053 + 059 pathway-tree/spend/case-mix serving fns), which serves the activity tile + cases-per-bed outlier + pathway tree on `/awarder/121858220` and the case-mix strip on `/company/:eik`. The `:cloud` loader IS the publish step. Commit the JSON.) |
| `nzok_nrd_tariffs` | `update-nzok` (a new НРД/amendment PDF landed on nhif.bg/bg/nrd — its чл. 368/369/370(б…) tables may re-set the КП/АПр/КПр prices. Re-run `tsx scripts/nzok/write_pathway_tariffs.ts --annex "<new-pdf-url>" --dump --nrd-year <year>` (`--bgn` pre-2026; the parse needs a human pass — codes may be re-tabled under new чл. numbers), then `npm run db:load:nzok-tariffs:pg` then `npm run db:load:nzok-tariffs:pg:cloud`. The tariffs JSON is gitignored — nothing to commit.) |
| `nzok_drug_unit_prices` | `update-nzok` (re-runs `npm run data:nzok -- --drug-prices` — fetches the newest `Справка 5_ПЛС2_MM.YYYY.xls` from `nhif.bg/bg/nzok/medicine/5`, rewrites `data/budget/nzok/drug_unit_prices.json` (gitignored), THEN — critically — `npm run db:load:nzok-drug-prices:pg` then `npm run db:load:nzok-drug-prices:pg:cloud` to reload the drug-price tables + apply migrations 052/054/**060**, which serve the `/molecule/:inn` + pack pages, the drug-risk board AND the **drug-savings leaderboard** (`nzok_drug_savings_overview`) on `/awarder/121858220`. The `:cloud` loader IS the publish step. JSON is gitignored — no commit, no `bucket:sync`.) |
| `mh_eeof_quarterly` | `update-nzok` (re-runs `npm run data:nzok -- --eeof` — fetches the newest МЗ ЕЕОФ quarterly XLSX, rewrites `data/budget/nzok/hospital_financials.json` (gitignored), THEN — critically — `npm run db:load:nzok-financials:pg` then `npm run db:load:nzok-financials:pg:cloud` to reload `nzok_hospital_financials` + apply migrations 051/**056**/**058**, which serve the ЕЕОФ financials strip, the **report-card + decile fan** and the **reporting-coverage** strip on `/company/:eik`. The `:cloud` loader IS the publish step. JSON is gitignored — no commit, no `bucket:sync`.) |
| `egov_commerce` → private-hospital ГФО revenue (**periodic / manual — NOT the daily flip**) | `update-nzok` → `npm run data:nzok -- --revenue` — recovers private-hospital annual revenue from the filed ГФО in the Търговски регистър (Gemini OCR of each ОПР) and rebuilds `public_private.json` (the public-vs-private band). Upstream is the same `egov_commerce` feed as `tr-daily-refresh`, but the ГФО filings are **annual/late** and the OCR is **expensive**, so DO NOT auto-run this on the daily `egov_commerce` flip — run it only when refreshing private-hospital financials or after the Sept ГФО filing season (like `--crosswalk` / `--pathway-tariffs`, no daily trigger). Resumable (only fills empty cells). Needs `GEMINI_API_KEY` + local PG; then commit + `bucket:sync data/budget/nzok/`. Separate marker `state/ingest/update-nzok-revenue.json`. |
| `America for Bulgaria Fdn grants` (**periodic / manual — NOT watched**) | `npm run ngo:abf-fetch` → then `npm run db:load:ngo-funding:pg` then `npm run db:load:ngo-funding:pg:cloud` (). ABF's Project Database (`us4bg.org/our-projects`) is Cloudflare + Elementor-AJAX walled, so it can't be fingerprinted/cron'd — `ngo:abf-fetch` drives a **headed** Playwright (set `form_fields[pagesize]=3000` → all ~1286 projects in one request) and writes the committed snapshot `data/ngo/abf/projects.json`. Grantee names are English → resolved to EIK via the curated `data/ngo/abf_aliases.json` in the funding loader (feeds the `foreign_funded` signal). Run periodically (a few times a year, when ABF publishes new grants); needs a display for the headed browser. Then commit `data/ngo/abf/projects.json` + `bucket:sync` the regenerated `data/ngo/ai_summary.json`. |
| `dfz_subsidies` | `update-agri` (ДФЗ published or re-uploaded a `data.egov.bg` org-56 subsidy year → refresh the whole `/subsidies` pack. Run `npm run agri:seu` first — headless Playwright pulls the current rolling FY2024/2025 from the СЕУ register `seu.dfz.bg` (which egov doesn't publish yet; one Actions ▸ Download ▸ CSV per year → `raw_data/agri/seu_<yr>.csv`), THEN `npm run agri:ingest` (egov org-56 CSVs + СЕУ years → the `agri_subsidies` + `agri_payloads` Postgres tables **directly**; PG-only, no static JSON, so no `bucket:sync`). **To publish:** `npm run db:load:agri:pg:cloud` **then** `npm run db:load:agri-hub-stats:pg:cloud` — the second is NOT optional and is not a refinement of the first: `agri:ingest` refreshes migration 162 only, while `db:load:agri-hub-stats:pg` is the sole applier of 163 (`agri_political_link`, `agri_cross_programme`), and those are DbDataTable resources with no degrade, so a cloud database that never ran it serves **500** on `/subsidies/political` and `/subsidies/cross-programme` rather than an empty page. (`db:dump:cloud` does NOT publish — it `pg_dump`s Cloud SQL _outward_ to GCS.) Only then do the live `/subsidies` + `/company/:eik` tiles update. СЕУ has no EIK column — the ingest recovers it by exact name-match to the 2015–2023 egov entities (recurring recipients relink; new entrants stay name-only). The СЕУ rolling window is NOT daily-watched (Playwright-heavy, updates continuously), so `agri:seu` runs on every `update-agri`; consider a monthly scheduled run for СЕУ freshness. See `[[project_agri_subsidies_pack]]`.) |
| `OFAC SDN — Bulgaria-linked designations` (`ofac_sanctions`) | `update-persons` (a Bulgaria-linked SDN row was added/removed/amended → an operator VERIFIES it at sanctionssearch.ofac.treas.gov and curates `data/person/sanctions.json`, attaching ONLY via a stable `mpId` — a name-ambiguous designee stays `resolved:false`, never publicly attached — then `npm run db:resolve:persons`. See the `update-persons` skill's defamation rule.) |
| `Комисия по досиетата — решения` (`comdos_ds`) | `update-persons` (comdos.bg published new решения → an operator reviews the published решения and curates `data/person/ds.json`, attaching ONLY via a stable `mpId` AND an exact birth-date match — a name-ambiguous / birth-date-mismatched affiliation stays `resolved:false`, never publicly attached — then `npm run db:resolve:persons`. See the `update-persons` skill's ДС defamation rule.) |
| `person_link_overrides` | `update-persons` (the committed audited identity-decision registry changed → run the full person resolve/projection chain; no publisher fetch or manual curation step is needed) |
| _person-layer re-derivation (downstream of ANY people source)_ | `update-persons` — the `person_*` tables are a pure re-derivation of the people datasets, so whenever a source that FEEDS them changed (`ivss_declarations` → `update-judiciary`, `cacbg_officials`/`cacbg_local` → `update-officials`, `egov_commerce` → `tr-daily-refresh`, `cik_results` → election ingest, `erik_campaign_financing` → `update-financing`, `parliament_mps` → `parliament-scrape`, and the curated `ofac_sanctions`/`comdos_ds`/`regulator_rosters` → `update-persons` itself), run `update-persons` AFTER that source's own skill so the `/person` profiles + merged `/candidate` block + `personProfile`/`personConnections`/`person_elections` tools don't go stale. That is BOTH `npm run db:resolve:persons` (~10s, idempotent, rebuilds identity/roles/connections) AND `npm run db:load:person-elections:pg` (rebuilds `candidate_person`/`person_election_stats` — the merged electoral block + header party badge); always run the person-elections load AFTER the resolve. On a `cik_results` (new/re-parsed election) flip the person-elections reload is the one that actually refreshes the candidate data — don't skip it. These write ONLY local Postgres; see step 9 for the Cloud SQL publish. |
| `Регулатори / независими органи — състави` (`regulator_rosters`) | `update-persons` (the Constitutional Court composition page changed — the flagship review trigger → an operator re-checks the current rosters of ALL independent bodies against their official pages and curates `data/person/regulators.json`, attaching each seat ONLY via a stable `mpId` OR a name confirmed globally-unique (`namesake_risk<=1`) — a name-ambiguous member stays `resolved:false`, never attached — then `npm run db:resolve:persons`. See the `update-persons` skill's accuracy rule.) |
| `Регистър на физическите лица в несъстоятелност (РНФЛ)` (`rnfl_insolvency`) | _no skill — nothing to ingest, and no `update-*` skill exists for this source; do not go looking for one. Surface under **"Manual action required"**, NOT `## Skipped`: an operator reads `docs/plans/rnfl-insolvency-v1.md` §T2 and DECIDES, then stamps with `npx tsx scripts/stamp-ingest.ts rnfl_insolvency --summary "…"` — nothing else clears it, and since the expected answer is "not yet, still gated" for months, an unstamped flip re-surfaces on every run for ever. Trigger = `/statistic-rnfl` serving a REAL statistics page; **a 200 alone is not enough** — the watcher compares the body against `/home-rnfl`, because a 302 back to the landing page also answers 200, so check `meta.statsBytes` / `meta.statsHash` before acting. The OTHER T2 trigger the plan names — a bulk export appearing on data.egov.bg — is **watched by nothing** (`egov_commerce` is pinned to the single TR daily-filings dataset UUID and would not see a new РНФЛ dataset under the same org), so that half needs a human to look. The `дело №1` probe is ADVISORY: it can only fire if file numbers are sequential from 1, which is unverified, so its silence must NEVER be read as "still empty". No table, loader or `recent_updates` row exists for this source. The plan refuses enumeration by file number (§2) and any probe carrying a personal identifier (§4)._ |
| `council_minutes` | `update-council-minutes` (re-runs `npm run council:scrape -- --per-councillor` — walks every município wired in `data/council/sources.json` that isn't `phase1Defer`, downloads new protokol/decision PDFs/DOCX/HTML since the per-município watermark, extracts aggregate `{for, against, abstain}` tallies + `adopted/rejected/returned/unknown` result + Решение № titles via `lib/tally.ts`, and merges into the slim `data/council/index.json` + per-município votes shards under `data/council/votes/<obshtina>.json` (feeding the "Как гласуваха в съвета" My-Area tile) + per-resolution shards under `data/council/{obshtina}/{YYYY}/`. `--per-councillor` is NOT optional on this path: without it no scrape emits `tally.perCouncillor` at all, which is what froze the named-vote half of the corpus from 2026-05-29 to 2026-08-16 with nothing reporting it. `--ocr` stays opt-in (~$1.85/session for Sofia's mojibake full-session protokols — see `lib/pdf_chunk_ocr.ts`). **Then publish to Postgres: `npm run db:load:council:pg` locally and `npm run db:load:council:pg:cloud` for prod — nothing runs the cloud half automatically, and skipping it leaves prod on the previous vintage at a 200.**) |
### The home dashboard — ONE generation step, at the END of the run
⚠️ **`/` IS NOT MAPPED TO A SOURCE, AND IT MUST NOT BE.** Nine families feed the home
dashboard — prices, macro, parliament, elections, council, open calls, budget, domestic debt
and the destination hub-stats blobs — so a per-source mapping would run the generators up to
nine times in one orchestrator run, each time folding whichever siblings had not yet finished.
The rule is therefore **once per run, after every selected skill has completed**, not once
after each source.
It is **Final post-step 7b** in the Procedure below, run unconditionally. `db:refresh` carries
the four generators at the end of its own chain, so a full refresh has already built them — but
it runs neither the health check nor the publish, so 7b is still required.
```bash
npm run db:gen-home-hub-stats # the four pulse figures + tile metrics — folds the sibling hubs
npm run db:gen-home-flyover # the flyover band's geometry, money layers and flow matrix
npm run home:flyover-posters # …and the nine stills that MUST match it
npm run db:gen-home-price-events # the basket + promotions, the ONE step that reads Postgres
npm run db:gen-home-feed # the „what changed" rows
npm run home:health # exits non-zero on missing/corrupt/unbuilt/unavailable/stale
```
⚠️ **PUBLISHING IS NOT THIS STEP'S JOB.** This orchestrator runs no sync of its own — it records
the touched subtrees in `state/upload/pending.json` and, at the end of a run with no major issue,
invokes `/upload-watch-changes` (Procedure step 14), which is what actually pushes them. Add
`home` there like any other subtree, and note that it needs `npm run bucket:gz` after the sync.
`npm run home:publish` is that pair as one command and exists for an operator working by hand.
⚠️ **THE ORDER IS NOT COSMETIC, AND `home:publish` IS TWO COMMANDS FOR A REASON.**
- `db:gen-home-hub-stats` is a FOLD of `governance/hub_stats.json`,
`procurement/derived/hub_stats.json` and `data/macro.json`, so it must follow the sibling
generators; run earlier it publishes the previous vintage of whichever had not yet run, and
`/` then disagrees with the page one click away.
- ⚠️ **The posters are NOT bucket-served.** `home:flyover-posters` writes into `public/`, which
reaches production only via `npm run deploy` — Vite copies `public/` into `dist/` at build
time — while `bucket:sync` ships `data/` and nothing else. So an orchestrator run publishes
the new `data/home/flyover.json` immediately and leaves production serving the PREVIOUS
posters, at which point the moving band and its own reduced-motion / Save-Data / prerendered
fallback show two different vintages of the map, at a 200, with the local gate green because
both halves are current there. Flag a pending hosting deploy in the run summary whenever
this step wrote files.
- `home:flyover-posters` must follow `db:gen-home-flyover` and is NOT optional. The posters
are committed files rendered from the artifact by the same engine, and
`flyover_posters.test.ts` pins the pair by the artifact's own `computedAt` — a regenerated
corpus whose posters were not re-rendered is a red gate, not a stale picture. `db:refresh`
carries the pair for the same reason. ⚠️ The comparison is CONTENT, never mtime: git does
not preserve modification times, so „is the poster newer than the artifact" has no answer on
a fresh clone or in CI.
- `db:gen-home-flyover` must FOLLOW `db:gen-home-hub-stats`: its caption headline is that
file's procurement figure, read from disk rather than recomputed, precisely so the moving
band and the `/procurement` tile one screen below cannot show two different euro totals. Run
the other way round it quotes the previous vintage and they disagree by whatever the reload
moved, with every count reconciling. `ORDER_PAIRS` holds it for the `db:refresh` chain.
- `db:gen-home-price-events` must precede `db:gen-home-feed`: the feed's price adapter reads
the committed `data/home/price_events.json` rather than Postgres (which is what lets the feed
build on a fresh clone), so run the other way round it folds the PREVIOUS vintage's basket
move and promotions with every count reconciling. `refresh_coverage.test.ts`'s `ORDER_PAIRS`
holds this for the `db:refresh` chain; nothing holds it here but this paragraph.
- **`bucket:gz` after the sync, always.** `gsutil rsync -j json` sets a TRANSPORT encoding only;
the stored objects come back `identity`, so a sync alone REVERTS the gzip a previous
`bucket:gz` established. `/` is the entry page and these two blobs are on its critical path.
**A generator failure leaves the previous public artifact in place and the run reports red.**
All THREE generators build in memory, validate, write a temporary sibling and rename, and all
three REFUSE rather than write an empty artifact — `hub_stats.ts` when no source answered or no
vintage could be normalised, `price_events.ts` when the price corpus is absent or empty, and
`feed.ts` when no adapter produced anything, no source declared a vintage, or nothing survives
the window. Do not "fix" a red `home:health` by publishing anyway: a half-written file on the
entry page is worse than a stale one.
**`npm run home:health` makes the check `db:check-generated` structurally cannot.** That one
compares LOCAL BYTES against the BUCKET, so when a generator is never re-run at all disk and
bucket are both stale, they AGREE, and it prints OK. `home:health` has two arms that do not
collapse into each other:
- **absolute** — the newest OBSERVATION any family reported, against the wall clock. ⚠️ Measured
on `observedAt` and NOT on `computedAt`: both artifacts date themselves by their newest source
vintage, so `hub_stats` being weeks back is its ordinary state (quarterly macro, sibling hub
blobs) and flagging that would be a false positive on every run. A crawl clock that has not
moved in a week means nobody has looked.
- **relative** — each family against its declared cadence (`STALE_AFTER_DAYS` in
`scripts/db/gen_home/events/adapters.ts`), computed live from the code's current declaration
rather than from the flag frozen in the artifact.
`--public` fetches the two objects a browser reads, compares their content AND checks
`x-goog-stored-content-encoding` — the header is the only thing that can see a missing
`bucket:gz`. A family with no cadence (`elections` — the last two are 539 days apart) is
reported and never flagged.
## Performance trace — EVERY step is timed
This orchestrator is the longest-running thing in the repo: a single run can invoke a
dozen ingest skills and then a `db:refresh` that is a **75-link `&&` chain**. Until 2026-08-29
none of it was measured, so "the run took four hours" was the only fact available and it
names nothing to fix.
So every step this skill takes is timed and appended to a durable trace:
```
state/perf/process-watch-report.jsonl ← this skill
state/perf/upload-watch-changes.jsonl ← /upload-watch-changes (the publish half)
```
**APPEND-ONLY, and committed.** One JSONL row per step, one `session` per orchestrator
invocation. The point is the HISTORY, not this run: a step is only diagnosable against its
own past durations, so the file is never truncated and never gitignored (a per-machine log
would be empty on the machine that later does the optimisation work — which is why the
extension is `.jsonl` and not `.log`, the latter being covered by `.gitignore`).
### The three commands
```bash
# once, at the very top of the run — every step below carries this id.
# ⚠️ PASTE THE PRINTED ID into each later command. `$S` below is shorthand: a shell
# variable does NOT survive between tool calls, and `--session ""` files rows under a
# blank session that no rollup can group.
npm run -s perf:step -- session
# a shell command: measured exactly, exit code propagated (so `&&` still works)
npm run -s perf:step -- run --run process-watch-report --session "$S" \
--step "myarea:alerts" --phase derive -- npm run myarea:alerts
# an `&&`-chain package script: ONE ROW PER LINK, plus a whole-chain total
npm run -s perf:chain -- db:refresh --session "$S" --phase ingest
# something this process did not run — a Skill invocation, a manual wait
date -u +%FT%TZ # before — read the stamp; it does not survive the tool call
# … Skill(update-procurement) …
npm run -s perf:step -- record --run process-watch-report --session "$S" \
--step "update-procurement" --phase ingest --started 2026-08-29T05:12:33Z --status ok \
--notes "3 210 new contracts"
# read the trace back (also step 10 below)
npm run -s perf:report -- --run process-watch-report --last 5
```
Four rules, each of which is a way the trace goes quietly WRONG rather than absent — and a
wrong duration is worse than a missing one, because it sends the optimisation work to the
wrong step:
- ⚠️ **Never wrap a chain in `perf:step run`.** `npm run db:refresh` under `perf:step` yields
ONE number for 75 links, which answers "was it slow" and nothing about WHERE — and "where"
is the entire question when the chain is an hour long. Use `perf:chain`, which reads the
chain text out of `package.json` (so it cannot drift from what `npm run` would have done),
runs each link through `sh -c` in order, and aborts on the first non-zero exit exactly as
`&&` does. Same applies to `tr:daily-refresh` (7 links).
- ⚠️ **`--step` labels are compared ACROSS sessions, so keep them deterministic.** No dates,
no counts, no row totals in the label — those go in `--notes`. A label that changes per run
makes every run look like a first run and the per-step history never accumulates.
- ⚠️ **Record a SKIPPED step rather than omitting it.** `--status skipped` with a `--notes`
reason. An unmeasured step and a fast step are indistinguishable in the trace, and a step
skipped four runs in five is exactly the one whose real cost gets forgotten.
- **A failed step is still a step.** `perf:step run` records the non-zero exit and then
propagates it; do not swallow the row when the orchestrator halts. Aborted runs are the
most interesting rows in the file.
`phase` is the coarse grouping — `plan` | `ingest` | `derive` | `verify` | `publish` |
`commit`. Use it consistently; it is what lets a later analysis say "the derive block is
40% of the run" without re-classifying every step by hand.
### Reading the trace
`npm run perf:report` prints two views, and the second is the worklist:
- **Last N sessions** — total work and that session's five slowest steps.
- **Cost by step, ranked by TOTAL across all sessions.** Ranked by total rather than by
median on purpose: a 4 s step that runs sixty times costs more than a 90 s step that runs
once, and the cheapest win is usually the one that repeats.
⚠️ A session's total is the **sum of its steps' own durations**, never `end − start`. The
gap between two steps is the agent thinking, or the operator answering a prompt, or a manual
download — real wall-clock, but not work this pipeline can attribute or optimise. Reporting
it as "the run took X" would credit the pipeline with time it never spent.
## Procedure
### Step 0 — read "Manual downloads needed" first
The daily report may open with a `## Manual downloads needed` section. It sits above
`## Changed` because a missing input blocks its ingest entirely — there is nothing to run
even when the upstream moved.
Surface every entry to the user before invoking anything, and **do not invoke a skill whose
input is still missing**. Report it as skipped and leave that skill's marker untouched:
these skills mostly skip-and-warn on an absent input and exit 0, so running one stamps
`lastSuccessfulIngest` over the gap and the orchestrator stops queueing the work. The
request itself is disk-derived and keeps firing — what a wrong stamp suppresses is the
ACTION, not the report line, which is why this goes unnoticed.
**A source whose skill has a shared marker needs its own.** `state/ingest/update-budget.json`
backs ~13 sources, so it is stamped whenever any of them fires; a sub-step waiting on a manual
download must key on its own marker or the wait is masked by unrelated activity.
A source can appear in BOTH this section and `## Changed` — they answer different questions
("someone must fetch something" vs "an upstream moved"). The manual entry wins: fetch
first, then re-run the orchestrator.
1. **Open the trace, then enumerate state.** The FIRST command of the run mints the session
id every later step carries — do it before anything else, because a step recorded under a
fresh id is a step that vanishes from this run's rollup:
```bash
npm run -s perf:step -- session
```
⚠️⚠️ **SUBSTITUTE THE LITERAL ID INTO EVERY LATER COMMAND.** Each Bash tool call is a fresh
shell, so `S=$(…)` set in one call is EMPTY in the next and `--session ""` files the row under
a blank session no rollup can group. `$S` in the examples means "paste what this printed"
(`--session 20260829T051233Z-a1b2`). Mint it ONCE — a second `session` call starts a second
run as far as every report is concerned.
Lost it? Read it back rather than minting a new one:
`tail -1 state/perf/process-watch-report.jsonl | jq -r .session`.
Then inspect both state directories via Bash. Each watcher source has `lastChanged`; each ingested skill has `lastSuccessfulIngest`.
```bash
# Watcher state — one per source
for f in state/watch/*.json; do
jq -r --arg name "$f" '"\($name) lastChanged=\(.lastChanged // "?")"' "$f"
done
# Ingest markers — one per skill (may be empty on first run)
for f in state/ingest/*.json; do
[ "$f" = "state/ingest/.gitkeep" ] && continue
jq -r --arg name "$f" '"\($name) skill=\(.skill) lastSuccessfulIngest=\(.lastSuccessfulIngest)"' "$f"
done
```
**Performance note (measured 2026-07-16):** the two `for`-loops above spawn one `jq` per file (~169 processes, ~1.4 s wall). A single batched pass per directory is ~35–80× faster (~0.02 s) and folds the enumerate→map→queue step (2) into one join — reach for it on a large state dir:
```bash
# Watcher state as one object: {id: lastChanged}
jq -n 'reduce inputs as $x ({}; . + {($x|input_filename|sub(".*/";"")|sub(".json$";"")): ($x.lastChanged)})' state/watch/*.json
# Ingest markers as one object: {skill: lastSuccessfulIngest}
jq -n 'reduce inputs as $x ({}; . + {($x|input_filename|sub(".*/";"")|sub(".json$";"")): ($x.lastSuccessfulIngest)})' state/ingest/*.json
```
The `input_filename`-keyed reduce is the load-bearing trick — one `jq -n … inputs` reads every file in a single process instead of `for f … jq "$f"`.
2. **Build the work queue.** For each watcher source:
- Look up its mapped skill in the table above. If no mapping, log it as "skipped — no handler" and move on.
- Read the mapped skill's `state/ingest/<skill>.json`:
- **Missing marker (skill never stamped)** → queue the skill AND flag "first-run bootstrap"; see "Bootstrap" below before invoking.
- **`lastChanged > lastSuccessfulIngest`** → queue the skill.
- **`lastChanged <= lastSuccessfulIngest`** → skip (already ingested this change).
Dedupe the queue: if `cacbg_declarations` and `bgpost_postcodes` both flag `update-connections`, queue it once. Preserve the watcher-source-order so the user sees a stable plan.
3. **Confirm the plan with the user before doing destructive work.** Print which sources triggered which skills and the estimated work. Example:
> Plan for orchestrator run:
>
> - `update-macro` ← `eurostat` (changed 2026-05-09, last ingest 2026-05-07) — ~30s
> - `update-financing` ← `smetna_palata` (changed 2026-05-11, never ingested) — first run, ~10s
> - then `/upload-watch-changes` publishes the result (bucket + Cloud SQL) — withheld
> automatically if anything above fails; say "hold the publish" to keep it manual.
> Proceed?
Wait for user confirmation (or proceed automatically if they already said "go" / "run all" / "yes proceed", or the args contain `unattended` — see "Unattended mode"). If the queue is empty, print "Nothing to ingest — every changed source has already been processed since its last change" and stop.
**The "yes" here covers the publish.** Step 14 invokes `/upload-watch-changes` on the
strength of this confirmation, and that skill does not re-ask when the orchestrator hands it
a clean run — so the plan MUST name the publish, every time. A "hold the publish" / "don't
upload" answer is remembered for step 12's gate: the manifest is still written, only the
invocation is withheld.
4. **Invoke each skill in sequence — and time each one.** Use the `Skill` tool, one skill at a
time. Don't parallelise — they can conflict on `data/` writes. Capture each invocation's actual
stdout (counts, file paths, status) for the final summary. Do NOT paraphrase as "done" — quote
specifics.
A `Skill` call is not a shell command, so it cannot be wrapped by `perf:step run`. Bracket it
instead — capture the clock immediately BEFORE the invocation and record immediately after:
```bash
date -u +%FT%TZ # ← run this, READ the output, then invoke the Skill
```
⚠️ Same rule as the session id: `T0=$(date +%s)` would be gone by the next tool call. Read the
printed stamp and paste it literally (`--started 2026-08-29T05:12:33Z`); `--started` accepts an
ISO stamp or epoch seconds. If the start time is lost entirely, use `--seconds <n>` with your
own estimate and say in `--notes` that it is estimated — an honest estimate beats a hole, and a
hole reads as "fast" for ever.
```bash
npm run -s perf:step -- record --run process-watch-report --session "$S" \
--step "update-procurement" --phase ingest --started "$T0" --status ok \
--notes "3 210 new contracts, 41 new tenders"
```
⚠️ **`--step` is the SKILL NAME and nothing else** — no date, no count. It is the key the
per-step history joins on across sessions; a label carrying today's numbers makes every run
look like a first run. The counts belong in `--notes`, which is free text.
A skill that ran and found nothing still gets a row (`--status ok`, notes `"no new data"`); a
skill that was queued and then NOT run gets `--status skipped` with the reason. Omitting either
makes an unmeasured step indistinguishable from a fast one.
Before the next invocation, run `git diff --stat data/` to capture what physically changed on disk vs. what the skill claims. The diff is truth; skill output is narration.
**A skill whose body is an `&&`-chain is worth expanding.** `update-procurement` ends in
`db:refresh` (75 links) and `tr-daily-refresh` is itself 7 links. When a skill's own procedure
tells you to run one of those, run it as `npm run -s perf:chain -- <script> --session "$S"`
instead of `npm run <script>`: identical semantics, one trace row per link. That is where the
hours are, and a single row for the whole chain names nothing to fix.
5. **Stamp success.** After each skill completes without error, run:
```bash
npx tsx scripts/stamp-ingest.ts <skill-name> --summary "<one-line recap>"
```
This writes `state/ingest/<skill>.json` with `lastSuccessfulIngest = now`. If the skill threw, do NOT stamp — the orchestrator's halt-on-error rule below applies.
The summary should reflect what was actually ingested (e.g. `"2 new sessions through 2026-05-10"` for rollcall, `"15 years tracked, 0 net change"` for financing). It lives in the marker file and shows up in `git log -p state/ingest/`.
**Then append a row to the public data-changes log — but only when data actually changed.** The `/data-changes` SPA page is for readers, not auditors; it should list substantive refreshes, not bootstrap stamps or fetchedAt-only churn. Use `git diff --stat data/` (the truth) as the gate:
```bash
# If the skill actually wrote new/changed bytes under data/, append.
# If the diff is empty (or only metadata files moved), skip.
if [ -n "$(git diff --stat data/)" ]; then
npx tsx scripts/append-data-change.ts <skill-name> \
--summary "<same one-line recap>" \
--source "<upstream label, e.g. 'Eurostat macro (BG)'>"
fi
```
Use the same `--summary` text as the stamp — keeping the two in sync means `git log -p state/ingest/` and the public page tell the same story. The script is also defensive: it auto-skips when the summary matches no-op patterns (`bootstrap:`, `unchanged`, `no data changes`, `only fetchedAt diff`, `timestamp-only diff`, `no run`), so a stray call won't pollute the page — but `git diff --stat` is the cleaner upstream gate.
**PG-only ingests that self-report — skip the generic append for these.** A Postgres-only skill writes nothing (or only a tiny artifact like `product_slugs.json`) under `data/`, so `git diff --stat data/` is empty and this gate would silently drop it — e.g. `update-prices` was the single most-frequent `/data/updates` entry, and going Postgres-only would have taken it to zero. Those skills instead call `appendDataChange` from inside the ingest, based on the rows they actually loaded. **Do NOT run `scripts/append-data-change.ts` for them** — the ingest already did. Currently self-reporting: `update-prices` (`scripts/prices/ingest.ts`), `update-persons` (`resolve_persons.ts`), `update-nzok` (`load_nzok_hospital_pg.ts`) and `update-fuel` (`fetch_fuel.ts`).
⚠️ **A self-reporting ingest runs TWICE per publish cycle, and it used to report twice.** The `:cloud` twin is the same script with `DATABASE_URL` pointed at the Cloud SQL proxy — `prices:ingest:cloud` is literally `npm run prices`, `db:load:agri:pg:cloud` is literally `npm run db:load:agri:pg` — so the code that ingested the data locally runs again to publish it. `dedupeSameDay` keys on the calendar date, and the pair straddles UTC midnight often enough that ONE refresh was published as two days of activity three times: 2026-07-28, 2026-08-28 and 2026-08-31 (the last caught by hand, the first two shipped and were removed by the fix). Two things now stop it, both central so a new self-reporter inherits them:
- `appendDataChange` **refuses any append made while pointed at the serving database** — a publish is not a data change. Nothing to add per script.
- prefer **`dedupeKey`** over `dedupeSameDay`: a stable identity for the DATA the row describes (for prices, the loaded price day), deduped across the whole log with no time window. Reach for `dedupeSameDay` only when the row legitimately recurs every run with no data day to key on (`resolve_persons` re-derives the whole person layer daily).
Three tempting alternatives that do NOT work, all measured against the committed log: a wider dedupe **window** (24/48 h) swallows genuine consecutive-day refreshes, and prices publishes daily; the wall-clock **date** is the bug itself; and **`summary` equality** misses it whenever the cloud database was further behind and loaded a different number of archives ("+2 daily archives" vs "+1") — which is exactly the shape of the 2026-07-28 pair. Gate: `scripts/lib/data-changes.test.ts`.
(`update-agri` is also PG-only and does NOT yet self-report — its `/data/updates` history is a known gap; add the same `appendDataChange` call to `scripts/agri/ingest.ts` when convenient, with a `dedupeKey` naming the financial year or corpus vintage it loaded. It needs no other change: the publish guard already covers `db:load:agri:pg:cloud`.)
6. **Final post-step: rebuild the My-Area alerts feed.** After every queued skill has run (or the queue was empty), unconditionally run:
```bash
npm run -s perf:step -- run --run process-watch-report --session "$S" \
--step "myarea:alerts" --phase derive -- npm run myarea:alerts
npm run -s perf:step -- run --run process-watch-report --session "$S" \
--step "myarea:alerts:cloud" --phase publish -- npm run myarea:alerts:cloud
```
⚠️ **BOTH lines, and the second is not optional.** Since json-retirement-v2 Tier 4b the feed is stored in Postgres (`myarea_alerts`, migration 184) rather than written as 290 files under `data/myarea/alerts/` — so a local-only run leaves **prod serving the previous vintage at a 200**, with `git diff --stat data/` empty and nothing red anywhere. This is the same class as the C1 rollcall gap: the local half looks like a complete run. The builder composes the events in TypeScript (they carry bilingual prose) and only the storage moved; 184's header has the reasoning.
The script materialises the per-município "Последна активност" feed (the `myarea_alerts` table) from already-ingested data — council resolutions, procurement (now tagged **announced / awarded / annex** via the rollups' `tag`), EU-fund contracts **plus snapshot-diff new/modified projects** (from `data/funds/projects/changes/`, dated by the real detection day), capital programmes, local elections, plenary keyword hits. It's cheap (<5 s) and its output is a function of whatever's now in `data/`, so running it last keeps the feed in sync with whatever the orchestrator just changed. Because `update-funds` writes the EU `changes/` artifact _inside_ its own run (before this post-step), the new/modified EU events are already on disk when this runs. Don't stamp this as a separate skill — it's a derived rebuild, not an upstream ingest.
**Specifically do NOT append for:**
- Bootstrap stamps (option (a) from "Bootstrap" above — `"bootstrap: marker seeded, no run"`).
- Skills that ran but produced no on-disk data diff (e.g. macro where every series is current).
- Watcher-fingerprint flips that were chrome-only (financing's `subsidii` page churn, etc.).
**Re-seed (rarely):** if you ever need to rebuild `data/data-changes.json` from current `state/ingest/*.json` markers, run `npx tsx scripts/seed-data-changes.ts`. The seeder applies the same no-op filter, so it's safe to re-run.
7. **Final post-step: refresh the data map.** After the alerts feed, unconditionally run:
```bash
npm run -s perf:step -- run --run process-watch-report --session "$S" \
--step "data:map" --phase derive -- npm run data:map
```
This rebuilds `data/data_map.json` — the `/data` map manifest — baking the latest per-source freshness from `state/watch` and reflecting the watched-sources registry. Like the alerts feed it's a derived rebuild (a function of what's now in `data/` + the registry), not an upstream ingest, so don't stamp it as a skill. The write is **churn-free**: if nothing but the `generatedAt` stamp would change, the file is left untouched, so a quiet day produces no diff. The manifest lives under `data/`, so it ships to the live SPA via the same `bucket:sync` below — no Firebase site deploy is needed to refresh the map.
7b. **Final post-step: rebuild the HOME dashboard's artifacts.** After the data map and
**before** the artifact check below (which verifies these reached the bucket), unconditionally
run the four generators and the health check:
```bash
npm run -s perf:step -- run --run process-watch-report --session "$S" \
--step "db:gen-home-hub-stats" --phase derive -- npm run db:gen-home-hub-stats
npm run -s perf:step -- run --run process-watch-report --session "$S" \
--step "db:gen-home-flyover" --phase derive -- npm run db:gen-home-flyover
npm run -s perf:step -- run --run process-watch-report --session "$S" \
--step "home:flyover-posters" --phase derive -- npm run home:flyover-posters
npm run -s perf:step -- run --run process-watch-report --session "$S" \
--step "db:gen-home-price-events" --phase derive -- npm run db:gen-home-price-events
npm run -s perf:step -- run --run process-watch-report --session "$S" \
--step "db:gen-home-feed" --phase derive -- npm run db:gen-home-feed
npm run -s perf:step -- run --run process-watch-report --session "$S" \
--step "home:health" --phase verify -- npm run home:health
```
Then **record `home` in `state/upload/pending.json`** alongside the other touched subtrees, so
`/upload-watch-changes` publishes it. ⚠️ Note in the manifest that the home subtree needs
`npm run bucket:gz` after its sync — `/` is the entry page and a `bucket:sync:paths` reverts
the stored gzip on both blobs. `npm run home:publish` is that pair as one command, for an
operator working by hand.
Unconditional, and ONCE per run: nine source families feed the home feed, so a per-source
mapping would run these up to nine times, each fold catching whichever siblings had not yet
finished. The full reasoning, the ordering constraints and what to do when `home:health` goes
red are in „The home dashboard — ONE generation step, at the END of the run" above and in
[`docs/home-dashboard-operations.md`](../../../docs/home-dashboard-operations.md).
⚠️ **`home:health` red means DO NOT publish** — a half-written file on the entry page is
worse than a stale one. It exits non-zero on a missing, corrupt, unbuilt, unavailable, stale,
unpublished or drifted artifact and names which.
8. **Final post-step: verify the `db:refresh`-generated artifacts actually reached the bucket.** After the data map, unconditionally run:
```bash
npm run -s perf:step -- run --run process-watch-report --session "$S" \
--step "db:check-generated" --phase verify --ok-exit 1 -- npm run db:check-generated
```
**NINE** COMMITTED artifacts are bucket-served static GCS blobs that a hub reads. EIGHT are regenerated from Postgres by `db:refresh` — `procurement/derived/hub_stats.json`, `procurement/derived/sector_stats.json`, `culture/derived/hub_stats.json`, `governance/hub_stats.json`, `governance/declarations_hub_stats.json` and the three the HOME dashboard reads (`home/hub_stats.json`, `home/feed.json`, and `home/price_events.json` — an intermediate the feed generator reads, published for inspectability rather than for a browser) — and their registry is `REFRESH_GENERATORS` in `scripts/db/refresh_coverage.ts`. ⚠️ Do not read this count from prose: it said SIX for a while after the home three landed. `REFRESH_GENERATORS` is the authority and this command reads it. The ninth, `parliament/votes/derived/hub_stats.json`, is written by `rebuildDerived` from in-memory objects and published by that script's own `--upload` list, so `REFRESH_GENERATORS` structurally cannot hold it; its registry is `UPLOAD_PUBLISHED_ARTIFACTS` in the same file. This command reads BOTH, compares each local file's BYTES against the live bucket object, and prints the exact `bucket:sync:paths` line for any that differ. **It never uploads** — emit that line in Next-steps like every other production command (see "What this skill does NOT do"). Exit 1 means something is unpublished.
⚠️ **THE PUBLISH TRIGGER IS NOT THE OWNING SKILL'S TRIGGER — that decoupling is the whole reason this step exists**, and reasoning about which skill "owns" a path is what failed. `db:gen-culture-hub-stats` reads contracts, tenders, fund_projects, agri_subsidies, person_role and interreg_partners, so it moves when **`db:refresh`** runs (i.e. under `update-procurement`) — while `update-culture`, the skill that owns `data/culture/` and names its sync, is woken only by `nfc_film_register` / `ncf_grant_results` / `nfc_commissions` / `mc_dki_register`. The skill holding the PATH is never woken by the thing that changes the CONTENT, so no per-skill instruction can close this. Run the check unconditionally rather than trying to work out whether it applies.
Both misses this was written for, measured 2026-08-21:
- `culture/derived/hub_stats.json` — **404 for two days.** Committed 2026-08-19, generator wired into `db:refresh` 2026-08-18, and nothing had ever uploaded it. The hub degrades a 404 to „no figure" by design (`src/data/culture/hubStats.ts`), so `/culture` rendered its tiles without numbers at a **200** with nothing red anywhere.
- `governance/declarations_hub_stats.json` — **4 days stale, across a SCHEMA change.** The 2026-08-20 rework moved `companies`/`companyMps` → `organisations`/`organisationPeople`, so the deployed bundle was reading keys the served blob did not carry. `update-connections` Step 6 regenerates it and Step 7 commits it, and that skill has **no sync step at all** — so it was stale by construction on every run.
The two that were fine (`procurement/derived/*`) are exactly the two `update-procurement` names explicitly, which is the tell: this was never a hard command to run, it was an unowned one. ⚠️ "Fine" means PUBLISHED, on 2026-08-21, and nothing more — `procurement/derived/hub_stats.json` was found on 2026-08-27 to have been carrying stale NUMBERS the whole time, which this check cannot see. The two statements are about different axes and different dates; see the blind-spot table below.
A third, measured 2026-08-27, is why the check now reads two registries:
- `parliament/votes/derived/hub_stats.json` — **16 days stale, across a SCHEMA change**, and INVISIBLE to this step until then. The 2026-08-25 „the head ranks the groups" commit added `topGroups` / `otherGroups` / `otherMembers`; the bucket copy was from 11 August and carried none of them for any of the nine parliaments. Its only publisher is `rebuildDerived --upload`, which runs on a roll-call INGEST — and the change was a code commit, so nothing ever ran it. Here the publish trigger is not even the owning SCRIPT's trigger. ⚠️ It COMPOUNDS: `gen_governance/hub_stats.ts` folds this file **from disk**, so a stale bucket copy makes `/governance` and `/parliament` — one click apart — disagree about the same parliament while both are green locally.
A full `npm run bucket:sync` would also have caught all four — `culture/` and `governance/` are not in its `-x` exclusions — but nobody runs the ~30-minute full-tree sync day to day; the orchestrator's real path is the scoped `bucket:sync:paths`, whose argument list is assembled per skill. That is the gap.
⚠️⚠️ **THIS CHECK IS BLIND TO THE OTHER HALF OF THE CLASS, AND THE TWO FAILURES LOOK NOTHING ALIKE.** `db:check-generated` compares LOCAL BYTES against the BUCKET. It answers "was the artifact published?" — never "is the artifact CURRENT?". So when a generator is never re-run at all, disk and bucket are both stale, they **agree**, and this step prints **OK**.
Measured 2026-08-27 on `procurement/derived/hub_stats.json`: it published `appeals` 7,998 against a live 8,007 (a КЗК ingest four days earlier) and `connected` short across 15 scopes (a `db:load:tr:pg`), and this check reported it **OK** the whole time — correctly, because both copies carried the same wrong numbers. The two halves fail in opposite directions and only one of them is visible here:
| failure | disk vs bucket | caught by |
| --- | --- | --- |
| regenerated, never uploaded | DIFFER | `db:check-generated` (this step) |
| never regenerated | AGREE | ✅ `scripts/db/tests/procurement_hub_stats.data.test.ts` — disk vs CORPUS |
| just uploaded, edge still caching | DIFFER (falsely) | nothing — verify by hash, see below |
⚠️ **It also reports a FALSE `STALE` immediately after a successful sync**, because it compares
over HTTP and does not cache-bust. Measured 2026-08-27: right after `bucket:sync:paths` reported
all 9 paths synced, this step still called `culture/derived/hub_stats.json` and
`parliament/votes/derived/hub_stats.json` stale, both quoting a `served` date from the previous
day. `gsutil ls -L` showed both objects with an update time minutes old and an **md5 identical
to local** — they were published. Before re-syncing on this step's say-so, check the OBJECT
rather than the response:
```bash
gsutil ls -L gs://data-electionsbg-com/<path> | grep -E "Update time|Hash"
openssl dgst -md5 -binary data/<path> | base64
```
⚠️ **The other five artifacts DO each have a corpus-freshness gate already** — an earlier draft of this block said they had none, which was wrong and would have sent someone to rebuild what exists. `culture_hub_figures.data.test.ts` fails with *"The corpus moved under the artifact — re-run db:gen-culture-hub-stats"*; `sector_stats.data.test.ts`, `governance_hub_stats.data.test.ts` and `declarations_hub_stats.data.test.ts` reconcile theirs against live sums; and `hub_stats_pg.data.test.ts` compares the parliament blob against `readHubStatsFromPg()`. `procurement/derived/hub_stats.json` was simply the ONE of the six that had no such gate — which is exactly why it is the one that went stale — and Tier 1 closed it, so the set is now complete. **The rule that follows: when a skill this run moved one of a generator's INPUTS, re-run that generator, commit, and sync — do not infer from a green `db:check-generated` that the artifact is current.** For the procurement blob the inputs are `contracts`, `awarder_seats`, `tenders`, `kzk_appeals`, `ngo_funding` (its `RELATIONS` list), i.e. `update-procurement`, `update-kzk-appeals` (whose Step 3b now does it) and the `ec_fts` NGO-funding row all move it.
8b. **Final post-step: verify Cloud SQL is running the same schema objects as local.** After the artifact check, unconditionally run:
```bash
npm run -s perf:step -- run --run process-watch-report --session "$S" \
--step "db:check-cloud" --phase verify --ok-exit 1 -- npm run db:check-cloud
```
`--ok-exit 1` because both verifiers exit 1 when they FIND something — that is the
finding, not a crash, and recording it as a trace failure every run would make the
failure count meaningless. The exit code still reaches you; read the output.
It diffs every public FUNCTION body, VIEW/matview definition and relation between local Postgres and the Cloud SQL proxy, and reports only objects some `schema/pg/*.sql` file still CREATEs — so local scratch tables are classified as scratch and need no allowlist. Read-only; it prints the `apply_functions.ts` line and exits 1.
⚠️ **THIS IS THE ONE CLASS STEP 9 STRUCTURALLY CANNOT SEE, which is why it is a separate unconditional step rather than a row in that table.** Step 9 keys its `:cloud` commands on WHICH SKILL RAN, i.e. on a watcher flip in an upstream SOURCE. A serving function, view or index carries no data, so no `db:load:*` ships it and `deploy:db` ships `functions/` code — a different thing from a Postgres function. A migration edited by a **code commit** flips no watcher, so no row of step 9's table ever fires, and prod keeps running the old body indefinitely with every row count reconciling.
Measured 2026-08-27, on a serving database every other check called healthy — five objects behind, none of them a 500:
- `budget_hub_stats()` + its cache (156) — **eight fields missing**, all eight read by `budgetHubFigures.ts`, so `/budget`'s KPI band silently fell back to the projected basis and its evidence aside rendered not at all.
- `budget_admin_list()` (155) — missing the `node_id` tiebreak. 155's own comment says 156's evidence aside ranks on the same expression and must break ties the same way, so **these two must ship together** or the head and `/budget/ministries` disagree about who is fifth.
- `council_councillor_by_slug()` (161) — **absent entirely**; the route degrades to `null`, so the councillor voting record simply never appeared on `/person`.
- `person_by_slug()` (082) — missing `linkBasis` on the NGO arm: 5,670 of 5,727 board seats rendered with no basis mark beneath a companies list that marked every row of its own.
- `mp_tr_roles()` (150) — pre-correlation body, 7,745 buffers against 867.
⚠️ **Do NOT paste the emitted command blind.** Several of these files open with `DROP MATERIALIZED VIEW` and rebuild WITH DATA in one transaction (156), so applying them blocks that matview's readers — off-peak only — and where a LOADER is the documented path because it also REFRESHes, use the loader (156 → `npm run db:load:budget-hub:pg:cloud`). Order matters: a `LANGUAGE sql` body is validated at CREATE, so a file reading another file's object must follow it.
9. **Resolve the Cloud SQL publish set — and RECORD it, do not run it.** Most skills write static JSON under `data/` and ship via `bucket:sync` — those need **no** Cloud SQL step. But a few datasets are ALSO served live from Postgres (local Docker **and** Cloud SQL). Each PG-backed skill reloads the LOCAL Postgres tables inside its own run (procurement/tenders/awarder-seats via `update-procurement`'s `db:refresh`; TR + NGO register via `tr-daily-refresh`'s chained `db:load:tr:pg`; NGO funding via `db:load:ngo-funding:pg`; EU funds via `db:load:funds:pg`). Cloud SQL is a **production** target, so no command in this skill's own procedure touches it — the loaders below are RECORDED here and RUN by `/upload-watch-changes`, which step 14 invokes after a run with no major issue.
**Since 2026-08-29 the resolved set is WRITTEN DOWN rather than printed.** Append each command to the upload manifest as you determine it (step 12 finishes the file; `write` merges, so calling it repeatedly is how the list is built):
```bash
npx tsx scripts/upload_manifest.ts write --session "$S" \
--cloud "npm run db:load:pg:cloud" \
--cloud "npm run db:load:tenders:pg:cloud" \
--cloud "npm run db:load:awarder-seats:pg:cloud"
```
⚠️ **ORDER IS LOAD-BEARING and the manifest preserves it verbatim.** `--cloud` is repeatable and the list is de-duplicated first-seen-order — never re-sorted — because a loader that reads another's output must follow it (facts before derived on roll-call; `db:load:tr-company-place:pg:cloud` after `db:resolve:persons:cloud`; the whole person-layer block below). Emit them in the order you would have run them.
⚠️ **A PG-served tree is a `--cloud` entry, NEVER a `--paths` entry.** `isExcluded` in `scripts/bucket_sync_paths.ts` refuses every PG-served and every retired tree — 20+ branches today and growing every time a family moves to Postgres, so read the list from the guard (or from `bucket:sync:paths:dry`, which prints its per-path reason) rather than from any restatement. Putting one in `--paths` produces a refusal at publish time, and worse, reads as "the data was shipped" when nothing was.
The table below is still the source of what to emit; `/upload-watch-changes` is what runs it.
| PG-backed skill that ran this session | Cloud SQL sync command(s) to emit |
| ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `update-procurement` (АОП contracts / tenders / debarred / EOP / awarder-seats) | `npm run db:load:pg:cloud && npm run db:load:tenders:pg:cloud && npm run db:load:awarder-seats:pg:cloud`. **ALSO** re-publish the three contract-DERIVED crosswalk maps `db:refresh` rebuilds from the fresh corpus (they go stale on cloud whenever contracts change — this is why `transport_project_link` drifted): `npm run db:load:transport-project-map:pg:cloud && npm run db:load:water-operator-map:pg:cloud && npm run db:load:mvr-directorate-map:pg:cloud`. **ALSO — and it is NOT a fourth member of that set — `npm run db:load:transport-facility-map:pg:cloud`.** It never reads `contracts`: it maps the curated `TRANSPORT_ENTITIES` constant through `awarder_seats`, so its trigger here is the awarder-seats reload above, and unlike the three its row count SHOULD match local exactly. A mismatch there is a defect, not the expected asymmetry the parenthetical below describes. (Their row counts are legitimately NOT equal local↔cloud — each is derived from that DB's own `contracts`/`awarder_seats`, and cloud's corpus is fuller.) **ALSO** `npm run db:load:persons-browse:pg:cloud && npm run db:load:graph:pg:cloud` — the `/persons` money column AND the `/connections` graph both sum this contracts corpus (via `person_browse_table` / `company_public_money`), so both go stale on prod when contracts change; run graph AFTER persons-browse (it reads its facets). **ALSO** `npm run db:load:tr-company-place:pg:cloud` — it denormalizes `company_public_money`, so the place-companies tile ranks and counts the previous corpus otherwise. **ALSO** `npm run db:load:annexes:pg:cloud` — MANDATORY, not optional. It resolves against the `contracts` table, so ANY contracts reload stales it and the per-annex breakdown plus the чл.116 ал.2-vs-ал.3 labelling drift on prod. The sharpest case is an eviction: the cross-source reconcile ORPHANS the evicted row's `procurement_annexes` entries (16 across 9 contract keys on the 2026-08-04 run) and only this loader re-resolves them — but a run with zero eligible evictions still needs it. **ALSO** `npm run db:load:employer-links:pg:cloud` — a pure derivation over `declaration.filed_institution` × `contracts.awarder_name`, so it must follow BOTH corpora: `db:load:pg:cloud` (this row) and declarations **phase 1** (`db:load:declarations:pg:cloud` WITHOUT `--resolve` — phase 2 only fills `person_id`). It is the sole applier of 165+168, and `/api/db/awarder-officers` has no `.catch`, so on a cloud database that never ran it the route **500s** while the UI renders nothing. **ALSO** `npm run db:load:grant-links:pg:cloud` — same shape but NOT the same prerequisites: it needs `contracts` and **`tenders`**, and reads no `declaration` at all (its own header says `after db:load:tenders:pg AND db:load:pg`). So a TENDERS reload is also a grant-links trigger, and the preflight only catches an EMPTY `tenders`, never a stale one. It derives `grant_contract_link` from the ПИИ code already sitting in `tenders.subject` and `contracts.title`. ⚠️ **`fund_projects` IS a third input, and this row said the opposite until 2026-08-21** ("the coverage denominator, not an input — do not infer a funds-side trigger from it"). Since the buyer-corroboration fix, `fund_projects.beneficiary_eik` decides each link's `buyer_basis` and therefore its `confidence`: only `code_and_buyer` may be rendered as "this grant paid for this contract", and that verdict is a JOIN against the beneficiary. So **`update-funds` is a grant-links trigger too**, and against an EMPTY `fund_projects` every link comes out `buyer_basis='unknown'` with NOTHING citable — the loader warns, but the table loads and every row count reconciles. ⚠️ It has **no serving reader yet** — the surfaces are `culture-investigative-v1.md` T1.6c, unbuilt — so skipping it breaks nothing a user can see TODAY. It is emitted anyway because it is contract-derived and therefore goes stale like the rest: wiring it now costs a clause, and leaving it out means prod is stale from the day T1.6c ships, which is exactly when nobody will think to check. |
| `tr-daily-refresh` (TR + NGO register — `egov_commerce`) | `npm run db:load:tr:pg:cloud && npm run db:load:graph:pg:cloud && npm run db:load:tr-company-place:pg:cloud && npm run db:refresh:risk:cloud` — ⚠️ **`db:load:tr-company-place:pg:cloud` must come after `db:resolve:persons:cloud`, not just after the graph.** Its `person_link_n` column is denormalized from `person_role`, and it is what `place_mp_companies()` (151) FILTERS on, so a stale one does not skew `/settlement/:id/companies`, it empties it at a 200. On this flip the resolve runs under `update-persons` (below), so re-run tr-company-place after that skill rather than leaving it here — `db:load:tr:pg` TRUNCATE+reloads `company_politicians`, the graph's procurement-edge source, so the `/connections` graph + the `/person` "Свързани лица" tile go stale on prod after a standalone TR publish unless the graph re-derives too. `tr_company_place` (133) is the third: it is rebuilt from `tr_companies` AND denormalizes `company_politicians`, so without it every governance page's "фирми, регистрирани тук" tile keeps the previous registry vintage. **`db:refresh:risk` is the fourth and is NOT optional** — `load_tr_pg` refreshes `procurement_risk_indexes_cache` but NOT `contract_risk_cache`, and both score from the same views off `company_politicians`, so a TR load alone leaves the `mpConnected` bit stale on the per-contract index while the browser payload has moved: the contract page and the browser risk column then disagree, which is precisely what `refresh_risk.ts`'s header warns about. Measured 2026-08-05 on local: a TR reload (`company_politicians` 511→521) broke `risk_parity.data.test.ts` on `mpConnected`, and `db:refresh:risk` was what fixed it. **`person_role` is a FIFTH consequence with no loader at all** — `company_politicians` feeds Bridge A inside `resolve_persons.ts:509`, which mints `source='tr'` roles, and NOTHING in this row rebuilds them: only `db:resolve:persons` does. Same measurement: the TR load left `person_role` at 175,261 tr roles and only the re-resolve took it to 175,264. The effect is small per daily flip (single digits) and the resolve is ~5 min, so it need not ride every flip — but say so in Next-steps, and let it ride the next `update-persons` run rather than assuming the graph reload covered it. It does not. |
| `db:load:ngo-funding` (`ec_fts`) | `npm run db:load:ngo-funding:pg:cloud`
| `db:load:ngo-board-links` (`cacbg_officials`/`cacbg_local` → officials leg) | `npm run db:load:ngo-board-links:cloud` |
| `update-funds` (EU funds — `isun_eu_funds*`) | `npm run db:load:funds:pg:cloud -- --full` **then** `npm run db:load:funds-fit:pg:cloud`. ⚠️ **The scope flag is REQUIRED** — without it the loader exits 1 in 0 s ("Refusing to guess the scope of a Cloud SQL load") and HALTS the chain: use `--full` when `fund_beneficiaries`/`fund_projects` moved (an ИСУН re-ingest; minutes of work re-reading ~128k shard files — measured 250 s on Cloud SQL 2026-08-21. Both tables are stage-merged since then, so neither RELOAD blocks a reader; a `--full` run does still end with a non-concurrent REFRESH of `dual_corpus_rankings_cache` (no unique index, so `CONCURRENTLY` is impossible) which can block that one matview's readers up to the 2 s `lock_timeout` — the route catches the 55P03 and falls back to the live function, so no `/api/db` route 500s), `--payloads-only` when only the precomputed payloads did. `--payloads-only` after a re-ingest is the silent-wrong direction — it republishes payloads over an unchanged beneficiary table and reconciles while under-reporting the money. The second command is not optional and is not cosmetic: it is the sole applier of migrations **143, 144 AND 145**, so it publishes the `/funds/procedure/:code` base-rate card, the band-0 wire + band-2 news rail, and the `/funds` hub stat cache — none of which any other loader ships. Run it even when `fund_fit` itself looks unchanged, because a FUNCTION-body change in 144 reaches prod no other way. ⚠️ Do NOT re-run it after `db:load:interreg:pg:cloud` to "close the cycle": the Interreg loader REFRESHes `funds_hub_stats_cache` itself (CONCURRENTLY), and funds-fit APPLIES 145, which is DROP+CREATE — so re-running it there would blank the `/funds` hub until the rebuild finished, which is precisely why the Interreg loader refreshes instead of applying. |
| `update-open-calls` (open calls — `isun_procedures` / `sp2023_indicative`) | `npm run db:load:open-calls:pg:cloud` |
| `update-rollcall` (roll-call votes — `parliament_votes`) | `npm run db:load:rollcall:pg:cloud && npm run db:load:rollcall-derived:pg:cloud` — ⚠️ **facts BEFORE derived, always** (the second reads `vote_*`/`mp_seat` the first loads). There is NO bucket half for rollcall any more (json-retirement-v2 Tier 1 retired `sessions/` and `derived/per-mp/`, objects removed), so these two commands ARE the publish: `vote_item`/`vote_cast`/`mp_seat` (134) + `mp_attendance`/`party_cohesion`/`mp_dissent`/`mp_similarity` (135) are **Cloud SQL-served** and back the LIVE routes `/api/db/vote-day-summary`, `contested-votes`, `mp-similarity`, `mp-dissents` (hooks `useVoteDaySummary`/`useContestedVotes`/`useMpSimilarity`/`useMpDissents`), so a new voting session that only ran `bucket:sync` leaves those four tiles on the previous vintage at a 200. `update-rollcall/SKILL.md` (Step 5b) documents the pair; **nothing runs them for you.** Re-run after EVERY `update-rollcall` that added a session. |
| `update-prices` (КЗП retail prices — `kzp_prices`) | `npm run prices:ingest:cloud` — the daily production publish: fetches the new day(s), loads the SCD-2 delta into Cloud SQL, and rebuilds the `price_payloads` serving blobs behind `/prices` + the `/consumption` hub tiles. **Postgres-only, no `bucket:sync`.** The narrower `npm run prices:payloads:cloud` REBUILDS the serving blobs WITHOUT an ingest (≈4 min; applies 048, seeds `price_last_seen`) — reach for it when the payload BUILDER changed but the corpus did not (a new blob field/kind, or re-folding another skill's JSON per the "Price-hub freshness" note above), **NOT** for a new price day. ⚠️ When the chain pages WIDEN, ship the frontend bundle (`npm run deploy`) BEFORE the payloads — an older bundle renders a widened payload wrong (CLAUDE.md's prices section). ⚠️ A day that trips the **>20% feed-drop guard** is HELD, not loaded — a chain gone silent (F50/F55) — because loading it would DELETE that chain's prices from `price_current`; that is a feed problem to resolve upstream, not something to force through here. |
| `update-schools` (МОН matura/НВО/context — `indicators_mon_dzi`) | `npm run db:load:schools:pg:cloud` (loads `schools`/`school_scores`/`school_context` + all three `school_payloads` kinds — the `directory` payload `/education` + `/school/:id` read, the slim `risk` payload the МОН pack's SchoolRiskTile reads, and the per-place `place` blobs keyed by oblast/obshtina code that the `/governance/region/:oblast` education section reads — one loader writes all three; skip it and that section serves the previous matura vintage at a 200). **ALSO GCS**: `data/schools/index.json` (My-Area tile) + `data/education/*.json` (textbook-market tile, incl. the per-year slices for the scope pill) stay GCS-served, so emit `npm run bucket:sync:paths -- schools education` too. If only the textbook market changed (АОП, no ДЗИ), the bucket sync alone suffices — no Cloud SQL load. |
| `update-persons` (person-identity layer — ANY people source, incl. `cik_results`/`ivss_declarations`/`cacbg_officials`/`cacbg_local`/`egov_commerce`/`erik_campaign_financing`/`parliament_mps`/`ofac_sanctions`/`comdos_ds`/`regulator_rosters`) | **The full ordered chain below — NOT just the resolve.** See "Person layer: the whole cloud chain" right after this table. The `person_*`/`candidate_person`/`person_election_stats` tables are **Postgres-only** (no serving JSON, no `bucket:sync`) — Cloud SQL is the ONLY publish target, so this MUST be emitted whenever the local re-derivation ran, or `/person` + the merged `/candidate` + `personProfile`/`personConnections`/`person_elections` go stale on prod. Reads the SAME PG upstreams (`magistrate`/`official_roster`/`tr_person_roles`/`contracts`) from the cloud proxy, so those must already be loaded there (they ride procurement/tr/magistrate cloud loads above). The resolve self-bootstraps a fresh Cloud SQL (085 before 082). Also needs a `npm run deploy` (functions) once so `db_routes.js` serves the `person-profile`/`candidate-person`/`person-elections` routes. |
| `update-administration` (ИИСДА services register — `iisda_services`) | `npm run db:load:admin-services:pg:cloud` (loads the `admin_services` table + the `services_overview.json` aggregate behind `/sector/administration`). The other administration artifacts (egov / digital-skills / service-quality / context) are static JSON under `data/administration/` → GCS via `bucket:sync data/administration/`, no Cloud SQL. |
| `update-budget` (state budget + the municipal side — `egov_budget_execution`, `ministry_execution_reports`, `budget_law`, `dv_investment_annex`, `minfin_program_otchet`, `municipal_fiscal_due`, `capital_programs`, `ipop_mrrb`, `egov_municipal_execution`) | `npm run db:load:place-dim:pg:cloud && npm run db:load:municipal-fiscal:pg:cloud && npm run db:load:budget:pg:cloud && npm run db:load:budget-muni:pg:cloud && npm run db:load:budget-hub:pg:cloud` — **in that order.** `place-dim` is a hard prerequisite of 149 (its serving functions JOIN it and a `LANGUAGE sql` body is validated at CREATE, so the file 42P01s and `exec()`'s single transaction leaves NO `municipal_fiscal` table at all); `budget-hub` must be LAST because 156's matview reads eleven relations produced by the four loaders before it (its twelfth, `budget_peer_band`, it fills itself). `budget:pg` and `budget-muni:pg` are different corpora — the STATE budget (152 КФП, 153 admin/programme/COFOG) versus the MUNICIPAL side (154 transfers, execution, capital + ИПОП) — and neither fills the other's tables. Until 2026-08-20 `update-budget/SKILL.md` contained zero `db:load` and zero `:cloud` strings, so the `/budget` module had no written publish path of its own — only `place-dim` was named elsewhere, which is why four exemptions lifted rather than five. |
| `update-judiciary` (ВСС court statistics / ИВСС register — `vss_court_statistics`, `ivss_declarations`) | `npm run db:load:court-load:pg:cloud && npm run db:load:magistrates:pg:cloud && npm run db:load:magistrate-filing-assets:pg:cloud && npm run db:load:judicial-bodies:pg:cloud`, plus `npm run db:load:judiciary-payloads:pg:cloud` when `__write_declarations.ts` ran. **The one FORCED edge is `magistrates` → `magistrate-filing-assets`, and it is the edge that gets dropped**: the magistrates loader TRUNCATEs `magistrate`, which blanks the per-filing declaration metadata and the parsed property count on every magistrate card until the asset loader re-runs. `judicial-bodies` is indifferent to that pair but must follow `court-load` AND `magistrates` (it folds `magistrate.court` ∪ `court_load.name`), so it could equally sit before the asset loader. The mechanism, the measured counts and the verification queries are NOT restated here — see `update-judiciary`'s Cloud SQL publish block, which is their one home. ⚠️ `magistrate-filing-assets` reads the gitignored `raw_data/judiciary/filing_cache.json`, so on a machine that never ran the crawl it skips-and-warns at exit 0 — a safe no-op, but not a repair. If `update-persons` runs in the same session its `db:resolve:persons:cloud` must follow `judicial-bodies` — see the person-layer chain below. |
| `update-kzk-appeals` (`kzk_appeals`) | `DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts 042_kzk_appeals.sql` then the same-URL `npx tsx scripts/procurement/kzk_appeals.ts --year <YYYY> --apply`. There is no `db:load:kzk:pg:cloud` — the crawl **is** the loader, so publishing means re-crawling against the cloud URL. The DDL is idempotent; apply it every time. |
**Cross-check the publish set with the resolver.** The table above is
hand-maintained; to compute the minimal, dependency-ordered `:cloud` commands for
the base datasets that actually changed, run:
```bash
npm run deploy:resolve -- contracts tenders # name each changed base table
```
It prints the ordered `npm run …:cloud` lines (via the chain-aware resolver,
`scripts/db/lib/deployResolver.ts` + the v2-c registry), routes the committed-artifact
generators to a `bucket:sync` note, and flags any change whose base load is a manual
crawl. Use it as a check that the hand-emitted list is complete and correctly ordered —
it is not yet the source of truth (that is v2-e's remaining wiring), but a loader it
emits that you did not is a gap, and vice-versa.
Each `db:load:*:cloud` wrapper points `DATABASE_URL` at the Cloud SQL proxy (`127.0.0.1:5434`, password read from `.pgpass`) and delegates to the base load — which reads the same fresh `data/` artifacts, reloads its table (`TRUNCATE`+COPY for most; a stage merge for the loaders that have been migrated, e.g. both funds tables), AND rebuilds the dependent matviews / `awarder_risk_grade_scoped` **on cloud** — so the served data stays self-consistent. It reads the on-disk artifacts, not local Postgres, so it's correct regardless of local PG state.
### Person layer: the whole cloud chain (emit ALL of it, in THIS order)
The `update-persons` row above is the one place where emitting "the loader for
the thing that changed" is not enough. `db:resolve:persons:cloud` READS four
inputs and DROPS one matview, so four loaders must run BEFORE it and three
AFTER. Three of the four fail **silently**, by publishing blanks rather than
erroring; the fourth, `tr-name-fold-people`, is the one that throws — which is
the good failure, but it aborts a ~29 min cloud resolve, so load it up front
rather than discovering it at the end. Emit the whole block; do not reconstruct it from memory
or from CLAUDE.md piecemeal (that is exactly how `place-dim` got dropped from
the emitted chain on 2026-07-29):
```bash
# BEFORE the resolve — it reads all four. Three of them fail SILENTLY: an empty or stale
# place-dim / judicial-bodies / declarations-phase-1 publishes blanks, not an error.
# tr-name-fold-people is the exception — the resolve throws rather than under-publish.
npm run db:load:place-dim:pg:cloud # 117 — mir/obshtina labels on ~76.5k roles
npm run db:load:judicial-bodies:pg:cloud # 116 — the court on ~2,700 magistrate roles
npm run db:load:tr-name-fold-people:pg:cloud # Bridge B's evidence table. The resolve
# THROWS on an empty one (naming this
# command), so it is a hard prerequisite
# rather than a follow-up — and it aborts
# the ~29 min cloud resolve, not a 5 min one.
npm run db:load:declarations:pg:cloud # phase 1 — rewrites subject_ref AND is what the
# resolve reads for the register gold key
npm run db:resolve:persons:cloud # 7m 35s measured 2026-08-29 on
# db-perf-optimized-N-2 (61,743 filings).
# Earlier figures on this line — "~5 min"
# (2026-08-05) then 1733 s / 28.9 min
# (2026-08-11, db-g1-small) — are both
# superseded; the second over-stated by ~4x
# and was used to argue against running it.
# AFTER the resolve — 115 drops person_role.place, taking the municipal roster matview with it
npm run db:load:declarations:pg:cloud -- --resolve # re-applies 102, rebuilds that matview
npm run db:load:employer-links:pg:cloud # 165+168 — a pure derivation over
# declaration.filed_institution × contracts.awarder_name.
# A declarations-only flip still needs it: 168 joins
# `declaration` live, so only NEW employer spellings go
# missing — silent, not a 500.
npm run db:load:official-candidate-links:pg:cloud # re-decorates + REFRESHes it
npm run db:load:person-elections:pg:cloud
npm run db:load:persons-browse:pg:cloud # folds all of the above + contracts
npm run db:load:person-search:pg:cloud # the combined-search index (reads persons-browse)
npm run db:load:graph:pg:cloud # LAST — the /connections graph (reads persons-browse facets + tr + procurement)
```
**Why phase 1 moved ABOVE the resolve on 2026-08-11.** It used to sit below it, with a
comment reading "BOTH declaration phases, phase 1 FIRST" — which meant first *of the two
phases*, not first *of the chain*. CLAUDE.md has always said the cloud order is phase 1 →
resolve → `--resolve`, and this block contradicted it. The cost is specific:
`registerIdByRef()` reads `declaration` for the Сметна палата gold key, so a resolve that
runs before phase 1 keys against the PREVIOUS filing set. Measured on the 2026-08-11
deploy, which published a 1,644-filing MP backfill: the cloud resolve reported **778
mentions aliased to an MP id where local reported 929**, and prod ended at 132 MP↔declarant
split pairs against local's 124 — the entire benefit of the backfill, absent, with every
row count on both sides reconciling exactly (49,627 filings, 5,575 mp-tier, 935 refs).
Re-running resolve → `--resolve` → the five downstream loaders fixed it, at the cost of a
second ~65-minute pass.
**It is invisible unless phase 1 CHANGES something.** On a deploy where the declaration
corpus is unmoved, the resolve reads the same table either way, which is why the wrong
order survived. Note the OTHER symptom CLAUDE.md predicts for this misordering — officials
postings publishing undated — did NOT appear, because prod already held the older filings
and so already had their `subject_ref`; only the 1,644 new ones were unseen. Do not use
"office dates look fine" as evidence the order was right.
What each omission costs, all of them green-locally / blank-on-prod:
| Omitted | Symptom on prod |
|---|---|
| `place-dim` | ~76.5k `mir`/`obshtina` roles publish `placeLabel: null`; the ~2.7k judicial roles keep theirs, so it reads as partial and is easy to miss |
| `judicial-bodies` | ~2,700 magistrate roles publish with no court |
| `declarations` phase 1 (running only `--resolve`) | A roster re-slug leaves `declaration.subject_ref` stale, so it joins to nothing: that filing keeps a NULL `person_id` and the person drops off `/person` and out of the "с декларация" facet **while every row count still reconciles**. `--resolve` does print `N/total still NULL`, so it is not literally silent — but a single-digit N against ~48k reads as ordinary residue, which is exactly how it was missed on 2026-07-31 (1 of 47,983). Re-run `persons-browse` after, since `has_declaration` reads `person_id`. |
| `declarations --resolve` | `declaration.person_id` all NULL → `has_declaration=false` for every person; the "с декларация" filter matches nobody and its KPI reads 0%, while net worth still renders. **Shipped to prod once.** |
| `official-candidate-links` | municipal roster matview left dropped → `/governance` + officials search degrade to an empty list without erroring |
| `persons-browse` | `/persons` serves the previous vintage; its money column silently disagrees with `/procurement/contracts` |
`place-dim` and `judicial-bodies` are cheap and idempotent, so emit them
unconditionally rather than trying to decide whether their inputs moved —
deciding wrong is invisible, re-running is ~seconds. `place-dim` stays cheap
on a no-op run only because it fingerprints the table either side of the
rewrite and skips its downstream refresh when nothing changed; when the
dimension GENUINELY moves it also rebuilds the per-scope precomputes that
store place labels (119 + 123), which is minutes on Cloud SQL, not seconds.
It says which of the two it did. **Verify before the resolve, not after:**
`select count(*) from place_dim` and `from judicial_body_alias` must both be
non-zero on the target.
**Prereq:** the Cloud SQL proxy must be running on `127.0.0.1:5434` and `.pgpass` must hold the proxy line — if the proxy is down, just list the commands for the user to run once it's up. (See the memory note "db:dump local vs Cloud SQL" for the proxy/`.pgpass`/`host.docker.internal` mechanics.)
**`db:dump` / `db:dump:cloud` is NOT a publish step.** `scripts/db/dump.ts` is a snapshotter: it `pg_dump`s whatever `DATABASE_URL` points at and uploads the dump to GCS + updates the committed lockfile, so a fresh clone can `db:restore` instead of re-ingesting. Pointed at the proxy (`db:dump:cloud`) it dumps Cloud SQL **outward**. It writes nothing into cloud and cannot create a missing table. Never list it _instead of_ a `db:load:*:cloud`; list it _after_ one only if a restore point is wanted.
**Datasets with no `db:load:*:cloud` wrapper** publish by re-running their writer against the proxy URL, because the ingest itself is the loader: `update-kzk-appeals` (`apply_functions.ts 042_kzk_appeals.sql`, then the crawl), which needs `DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg` prefixed.
**`update-agri` used to be in that list and is not** — it has TWO wrappers, and both must run:
```bash
npm run db:load:agri:pg:cloud
npm run db:load:agri-hub-stats:pg:cloud
```
The second is the sole applier of migration 163 (`agri_political_link`, `agri_cross_programme`); `agri:ingest` refreshes 162 only. Both are DbDataTable resources with no `missingMigration` degrade, so omitting it leaves `/subsidies/political` and `/subsidies/cross-programme` answering **500**, not an empty page.
**Opt-in / periodic (no daily watcher — do NOT auto-emit):** `db:load:nzok-tariffs:pg` / `db:load:nzok-tariffs:pg:cloud` (the НРД clinical-pathway tariffs, `nzok_pathway_tariffs`) rides `update-nzok --pathway-tariffs`, which is BG-egress + manual (like `--revenue` on line 477) and not in the default set — publish it only when that opt-in step is actually re-run. Same for the one-off `company_founded` procurement-risk backfill (`scripts/procurement/fetch_company_founded.ts`, written directly against the target `DATABASE_URL`, no loader wrapper) — cloud currently holds only a stub, so the `newFirmWinner` flag is dormant on prod until someone runs the backfill against the proxy. ⚠️ Runtime is **pace-dependent, not a fixed figure** (~14h per 10k EIKs while the source answers; far longer once it throttles), it **requires 033's `http_status`/`attempts` columns to have been applied to that DB first** (the script preflights and exits 2 otherwise), and a run that hits a hard block trips a circuit breaker and exits 1 rather than grinding — resume it from a different egress. A failed fetch is never written, so the run is always safe to re-run.
For a whole-DB reconcile instead of per-dataset (e.g. several PG-backed skills ran, or disaster recovery): `npm run db:sync:cloud -- --yes` — pg_dump local → pg_restore cloud, destructive (`--clean`), so local must be the source of truth first. Note this includes the unregenerable КЗК tier-2 outcome rows, so run `update-kzk-appeals` locally before syncing.
10. **Close the trace and read it back.** After the publish set is resolved, print this session's
timings — it is part of the deliverable, not a debugging aside:
```bash
npm run -s perf:report -- --run process-watch-report --last 3
```
Two things to actually look at rather than paste:
- **The session's slowest steps.** If a step is materially slower than its own median in the
"cost by step" table, say so in the summary. That comparison is the whole reason the trace
is append-only — a duration means nothing on its own.
- **Steps with no row.** Compare the row count against what you ran. A step you forgot to
record is a hole that reads as "fast" forever.
Nothing here fails the run. A slow step is a finding for later, not a reason to stop.
11. **Commit the run — automatically, by explicit pathspec.** A run's whole output is files on
disk: refreshed artifacts under `data/`, the ingest markers, the public change log, the data
map, this run's trace, and the upload manifest. Leaving them uncommitted is what lets a later
session's `git add -A` sweep them into an unrelated commit under a misleading message.
So commit at the end of every run, without asking. This is the ONE place this repo's
"commit only when asked" default is pre-authorised, because the operator asked for it here.
```bash
# 1. See what moved — and ONLY in the trees this orchestrator writes.
git status --porcelain -- data state/watch state/ingest state/perf state/upload public
```
⚠️⚠️ **NEVER `git add -A`, and never a bare `git commit`.** A bare `git commit` ships the
WHOLE INDEX, and in this repo the index is almost never clean at session start — a concurrent
process runs `git add -A` mid-session and other sessions leave work staged. Commit by
pathspec, which builds from HEAD plus exactly those paths and leaves every other index entry
untouched and still staged:
```bash
git commit -m "watch: ingest <source labels> (<one-line recap>)" \
-- data/... state/ingest/... state/perf/process-watch-report.jsonl state/upload/pending.json
```
Four rules, each of which has already gone wrong here at least once:
- **Name every path; sweep none.** Expand the `--` list from the `git status` above. If a
file in those trees is shown as already STAGED (a first-column `M`/`A`), it is not yours —
leave it out and say so in the summary.
- **Nothing outside those five trees.** Source changes, test changes, skill edits and
`news/**` belong to whoever was editing them. This orchestrator is data-only.
- **`git commit -- <path>` reads the WORKING TREE, not the index.** You cannot partially
commit a file another session is also editing; check `git diff -- <path>` first, and an
EMPTY diff on a file you just wrote means somebody else already committed it.
- **No `Co-Authored-By` trailer.** Plain message body, end.
Commit even when only markers moved (a no-data run still advances `state/ingest/*` and the
trace) — that is a real, meaningful commit. The only run that commits nothing is one where
`git status` on those five trees is empty.
Capture the sha for the manifest: `COMMIT=$(git rev-parse --short HEAD)`.
12. **Finish the manifest, then decide whether the run earned its publish.** The commit is the
end of the INGEST half. Nothing has reached production yet: the bucket still serves the
previous vintage of every file this run wrote, and Cloud SQL still holds the previous vintage
of every PG-backed dataset.
Finish the manifest — the bucket subtrees this run touched, the skills that ran, the commit:
```bash
npx tsx scripts/upload_manifest.ts write --session "$S" --commit "$COMMIT" \
--skills update-procurement,update-funds \
--paths myarea,budget,data_map.json,data-changes.json \
--notes "<one line: what a reader would see change>"
npx tsx scripts/upload_manifest.ts show
```
`--paths` are `bucket:sync:paths` arguments — the subtrees under `data/` that actually moved,
derived from step 11's `git status`, never a guess. `data_map.json` and `data-changes.json` are
almost always among them. PG-served trees do NOT go here (see step 9).
Then apply the gate. **The publish follows automatically (step 14) unless the run hit a
MAJOR ISSUE**, which means any of:
- a queued skill ended in `error` — including one the operator chose to skip past under the
Data-integrity contract's "(a) skip and continue"; the corpus that skill owns is unstamped
and unverified, and half a publish beside it looks like a whole one;
- a post-step exited non-zero: `myarea:alerts` / `myarea:alerts:cloud` (6), `data:map` (7),
any home generator or **`home:health`** (7b — its own rule says red means DO NOT publish);
- a `perf:chain` (`db:refresh`, `tr-daily-refresh`) aborted part-way — the corpus a loader
failed to update is unverified, and `test:data` at the end of the chain never ran;
- the commit at step 11 failed, or a path this run wrote was already staged by another
session and had to be left out — the manifest would then name a commit that does not
carry what it claims;
- the manifest is malformed — a `--paths` entry the sync guard refuses, or a PG-backed skill
ran and its `--cloud` set could not be resolved (the proxy being DOWN is not this: the
publish skill checks that itself);
- the operator said "hold the publish" / "don't upload" at step 3, or at any point since.
Not a major issue, and not a reason to hold: a skill that ran and found nothing; a `partial`
status (anomalies go in the summary's Notes); a skill skipped for a still-missing manual
download (step 0); the watcher's own `## Errors` section; a `STALE` from `db:check-generated`
(the upload is what fixes it); schema drift from `db:check-cloud` (surfaced, never applied).
When the gate HOLDS, say so plainly in the summary's Next steps and stop there:
> Ingest committed as `<sha>`. Nothing is live yet — the bucket and Cloud SQL are both still
> on the previous vintage. **Not publishing**: <the major issue, one line>. The manifest
> (N bucket path(s), M cloud command(s)) stays pending — run `/upload-watch-changes` once
> that is resolved.
⚠️ **Never run `bucket:sync` or a `:cloud` loader from this skill.** Whether the publish
happens now (step 14) or later (the operator), it happens through `/upload-watch-changes`
reading this manifest — the one path that records what was published and clears the debt.
(Step 6's `myarea:alerts:cloud` is the standing exception: a derived feed that lives only in
Postgres, documented there.)
⚠️ **A pending manifest is a DEBT, and it survives across runs on purpose.** When the publish
is held — by the gate or by the operator — the file stays on disk and the NEXT orchestrator
run merges into it rather than replacing it, so the first run's subtrees cannot be lost. Read it at the top of every run
(`npx tsx scripts/upload_manifest.ts show`) and say so in the plan when one is already
pending — "yesterday's ingest is still unpublished" is a fact the operator needs before
deciding what to run today.
## Bootstrap (first orchestrator run after this migration)
When `state/ingest/<skill>.json` is missing for a queued skill, you have two paths — ASK the user which:
**(a) Treat current state as the baseline (recommended for established repos)**
Stamp the marker to `now` without actually running the skill. This says "everything up to this moment is considered ingested; future changes will trigger ingest normally". Use when the user knows the existing data is up to date (typical case — the repo has already had ingests via earlier workflows).
```bash
npx tsx scripts/stamp-ingest.ts <skill-name> --summary "bootstrap: marker seeded, no run"
```
**(b) Actually run the skill (true backfill)**
For a clean clone, an explicit backfill, or when the user is unsure whether existing data is current. The skill runs, ingests anything new, then the marker is stamped on success.
Default to asking unless the user said "bootstrap markers" or "run all" or similar upfront.
13. **Final summary (REQUIRED).** Once all skills have run, print a structured per-skill recap. This is the deliverable — never collapse it into "all done" or a single paragraph. Format:
```markdown
# Watch report ingest — YYYY-MM-DD
## <skill-name> · <upstream label from report>
- **What ran**: one line describing the action.
- **Captured**: bullet list of concrete things (counts, file paths, new entries with dates/ids).
- **Files changed**: `git diff --stat` output for the relevant subdir, or "no changes".
- **Status**: ok | not_implemented | partial | error
- **Notes**: anomalies surfaced by the skill (e.g. unresolved MP ids, missing canary, retries).
## <skill-name> · <upstream label>
...
## Skipped (no automated handler)
- **<source label>**: short reason + suggested next step.
## Timings
| step | this run | median | notes |
| --- | --- | --- | --- |
| <step> | <duration> | <its own median> | slower/faster than usual, or blank |
Total work: <sum of step durations> across <N> steps. Slowest: <step> (<duration>).
## Next steps
- **Committed**: `<sha>` — <N> files. (Or: "nothing to commit" when the five trees were clean.)
- **Publishing now**: `/upload-watch-changes` follows this summary — <N> bucket path(s),
<M> cloud command(s). (Or, when step 12's gate held: **Not publishing** — <the major
issue>; the manifest stays pending, run `/upload-watch-changes` once resolved. Or:
"nothing to publish" when no manifest was written.)
- Whether to `git push` or hold
```
**The Timings table is REQUIRED and is not the same thing as the perf report.** Paste the
handful of rows that matter — the slowest steps, plus any step materially off its own median —
with the comparison, not the raw numbers. "db:refresh 1h 12m (median 47m)" is the sentence;
"db:refresh 1h 12m" is a number nobody can act on. Take both columns from
`npm run perf:report`; do not recompute them by hand.
**For a "no changes detected" run** (skill executed but found nothing new — e.g. `update-rollcall` walking past the last known stenogram id and finding no new sessions), explicitly write `**Status**: ok — no data changes detected` and `**Files changed**: none`. Do NOT silently omit the skill from the summary — the user must see that it ran and found nothing.
**For a "partial" status** (skill produced some output but also surfaced anomalies — unresolved MP ids, mismatched names, etc.), name each anomaly in the **Notes** field. The data was written; the warning is what the user needs to act on.
**For an "error" status** (skill threw), copy the error message verbatim and stop the orchestrator — don't proceed to the next skill until the user decides how to handle.
**Quote concrete numbers from skill stdout.** If `update-rollcall` printed `+ 2026-05-09 (id 11124): 11 item(s), 2640 rows · 37 unresolved id(s) → sessions/2026-05-09.json`, that's the kind of detail that belongs in **Captured**. If `update-macro` printed `Loading gdpGrowth (eurostat)... 84 points (latest 2025 Q4)` for 22 indicators, summarise: "22 indicators refreshed; latest period 2025 Q4 (quarterly), 2025 (annual)" — but if any indicator's count changed, name it.
### Publishing is a SEPARATE skill — invoked from here, never re-implemented here
After committing, `data/` lives in two places: the git repo (history, audit) and
`gs://data-electionsbg-com` (what the live SPA fetches). The bucket is the one users see, and
Cloud SQL is the one `/api/db` serves. **Neither is touched by this skill's own commands.**
Both are published by **`/upload-watch-changes`**, which reads the manifest step 12 wrote and
which step 14 invokes after a run with no major issue — so this skill's job is to have named
the right `--paths` and `--cloud` entries and to have judged the gate honestly, not to print
sync commands. The mechanics (scoped vs whole-tree sync, the refused PG-served trees,
deletions, gzip transport encoding, the Cloud SQL proxy) live in that skill's own procedure,
where they are executed rather than quoted.
Two things stay this skill's responsibility, because only it knows them:
- **`--paths` is derived from step 11's `git status`, never guessed.** A subtree a skill wrote
and the manifest omits is a subtree that stays stale on the bucket at a 200, with the commit
looking complete. This is the single most common publish miss in this repo's history.
- **When `Files changed` is "none" for every skill and no PG-backed skill ran, write no
manifest at all** — an empty manifest is a debt the next run has to reason about for nothing.
Say "nothing to publish" in Next steps instead.
Code/UI changes are out of scope for this orchestrator (`npm run deploy` deploys the Firebase
bundle and is not needed for pure-data refresh).
14. **Follow up with `/upload-watch-changes` — unless step 12's gate held.** This is the LAST
thing the run does, after the summary above has been printed: the ingest summary is this
skill's deliverable and must not wait behind a publish that can take tens of minutes.
Invoke it with the `Skill` tool, and tell it where it came from:
```
Skill(upload-watch-changes, args: "invoked by /process-watch-report after a clean run —
ingest session <S>, commit <sha>; proceed without re-asking")
```
Three things about the hand-off:
- **The step-3 confirmation is the authorisation.** The plan named the publish and the
operator said yes to it; the publish skill's own "Proceed?" at its step 0 is answered by
that, which is why the args say so. Without that phrase it asks again and waits — the
right behaviour when a human invokes it cold, and the wrong one at the tail of a run the
operator already approved.
- **It is a separate run with its own trace and its own commit.** It mints its own perf
session under its own run name, commits `state/perf/upload-watch-changes.jsonl` +
`state/upload/history.jsonl` + the manifest's deletion itself, and prints its own summary.
Do not re-summarise its output here, do not record its steps under THIS run's session, and
do not commit on its behalf — step 11 already committed everything this skill wrote.
- **A publish failure is that skill's to report, and it leaves the manifest pending.** If a
loader fails or the proxy is down, the publish skill halts, keeps `pending.json`, and says
which entries went live; the next orchestrator run reads the debt at step 1. Do not retry
the publish from here.
When the gate HELD, do not invoke it — the summary already said why, and what to run.
## Examples
### Nothing to ingest
Every watcher source's `lastChanged` is older than or equal to its mapped skill's `lastSuccessfulIngest`. Response:
> Nothing to ingest — all 7 sources are at or behind their last successful ingest. Watcher last ran <UTC>.
### Single change
```
state/watch/wiki_polls.json: lastChanged = 2026-05-11T01:30:00Z
state/ingest/update-polls.json: lastSuccessfulIngest = 2026-05-04T01:30:00Z
```
Queue: `update-polls`.
### Multiple changes, dedupe
```
state/watch/cacbg_declarations.json: lastChanged = 2026-05-10T...
state/watch/egov_commerce.json: lastChanged = 2026-05-11T...
state/watch/eurostat.json: lastChanged = 2026-05-09T...
state/ingest/update-connections.json: lastSuccessfulIngest = 2026-05-01T...
state/ingest/tr-daily-refresh.json: lastSuccessfulIngest = 2026-05-01T...
state/ingest/update-macro.json: lastSuccessfulIngest = 2026-05-05T...
```
Queue (`egov_commerce` → `tr-daily-refresh`; `cacbg_declarations` → `update-connections`; `eurostat` → `update-macro`):
1. `tr-daily-refresh`
2. `update-connections`
3. `update-macro`
### Multi-day gap (the case option B fixes)
You haven't run the orchestrator for 4 days. During that window:
- Tuesday: `eurostat` flipped → `eurostat.lastChanged = Tue`
- Wednesday: `wiki_polls` flipped → `wiki_polls.lastChanged = Wed`
- Friday: `parliament_votes` flipped → `parliament_votes.lastChanged = Fri`
Each downstream skill's `lastSuccessfulIngest` is still Monday's value. The orchestrator queues all three (`update-macro`, `update-polls`, `update-rollcall`) — none are missed, even though the latest report file only mentions Friday's change.
### Unmapped source
```
state/watch/cik.json: lastChanged = ... (changed)
mapping: no skill yet
```
Surface in the final summary's `## Skipped` section: "CIK news & decisions changed but has no automated handler — manual investigation needed."
Response: "1 changed source but no automated handler — `CIK news & decisions`. Manual investigation needed. Nothing to invoke."
### Worked example: full summary for a typical 2-skill run
Today's report flagged Eurostat releases + a Сметна палата page change. Plan: `update-macro` then `update-financing`. After both run, the orchestrator's final output looks like this:
```markdown
# Watch report ingest — 2026-05-11
## update-macro · Eurostat macro (BG): 13 datasets
- **What ran**: re-fetched all 22 macro indicators (13 Eurostat + 3 World Bank + 6 curated).
- **Captured**:
- 7 series with new upstream releases since last commit: sts_trtu_m, sts_inpr_q, ilc_di12, ilc_li02, namq_10_a10, ei_bssi_m_r2, prc_hpi_q.
- All 22 series passed the absolute floor + 10% regression check.
- Latest periods: gdpGrowth 2025 Q4, inflation 2026 Q1, wgiRuleOfLaw 2024.
- **Files changed**:
```
data/macro.json | 12 ++++++++----
````
- **Status**: ok
- **Notes**: none.
## update-financing · Сметна палата party financing
- **What ran**: re-scraped bulnao.government.bg/bg/kontrol-partii/.
- **Captured**:
- otcheti section: 15 years (2011-2025), status=ok, all gfopp URLs intact.
- subsidii section: status=not_implemented (skipped as expected).
- **Files changed**: none (year list identical to prior commit).
- **Status**: ok — no data changes detected (the watcher fingerprint flipped, but the structured year list is unchanged; the upstream page change was in chrome).
- **Notes**: subsidii still needs a Playwright-based ingest if the user wants the audit-reports list captured.
## Skipped (no automated handler)
_(none)_
## Timings
| step | this run | median | notes |
| --- | --- | --- | --- |
| update-macro | 2m 41s | 2m 12s | +22% — 7 upstream releases vs the usual 1-2 |
| update-financing | 18s | 17s | |
| myarea:alerts | 41s | 39s | |
| data:map | 3s | 3s | |
| db:check-generated | 11s | 11s | |
| db:check-cloud | 34s | 33s | |
Total work: 4m 28s across 6 steps. Slowest: `update-macro` (2m 41s).
## Next steps
- **Committed**: `a1b2c3d` — 4 files (`data/macro.json`, `data/data-changes.json`,
`data/data_map.json`, `state/ingest/update-macro.json`), plus the trace.
```bash
git commit -m "macro: refresh through 2026 Q1 (7 new Eurostat releases)" \
-- data/macro.json data/data-changes.json data/data_map.json \
state/ingest/update-macro.json state/ingest/update-financing.json \
state/perf/process-watch-report.jsonl state/upload/pending.json
````
- **Publishing now**: `/upload-watch-changes` follows this summary — 3 bucket path(s)
(`macro.json`, `data_map.json`, `data-changes.json`), 0 cloud command(s). Neither
`update-macro` nor `update-financing` is a Postgres-backed dataset, so the manifest
carries no `:cloud` entries; had a PG-backed skill run (`update-procurement`,
`tr-daily-refresh`, `update-funds`), step 9's commands would be in it.
- No `git push` needed yet — user decides.
````
The summary is mandatory whether or not data changed. A "no changes detected" run still gets a section per executed skill so the user can see what was checked.
## Data-integrity contract (CRITICAL)
This orchestrator MUST NOT claim success it didn't earn. Specifically:
1. **Trust downstream skills to fail loud.** Every tier-2 ingest skill is built to throw rather than write empty/partial data when upstream restructures. If a Skill invocation returns an error (or its terminal output contains "Error:", a stack trace, or otherwise signals failure), treat that source as **failed**, not "completed with warnings".
2. **Halt on first failure by default** (in unattended mode, take (a) automatically — see "Unattended mode"). When a downstream skill fails, STOP the orchestration. Do not proceed to the next mapped skill. Report which skill failed and what error, then ask the user whether to (a) skip and continue with the rest, (b) abort entirely, or (c) investigate.
3. **Never paraphrase ingest output as "done".** Read the actual stdout. Quote the relevant success/failure marker. Examples:
- `update-rollcall` success looks like `+ YYYY-MM-DD (id N): K item(s), R rows · U unresolved id(s) → sessions/<date>.json` for each session, plus a final summary.
- `update-financing` success looks like a per-section recap (`· <section>: N years` or `· <section>: not_implemented`).
- If you don't see those lines, the skill did not write data for that target — say so.
4. **Don't hallucinate counts.** When reporting back, use exact numbers from the skill's actual stdout. If a skill says "found 0 new sessions, nothing to ingest" that is a legitimate result for weekends/recess — report it as zero, do not invent a number.
5. **Surface `not_implemented` separately.** Some skills (like `/update-financing`'s subsidii section) intentionally report `not_implemented` for parts they can't yet handle. Pass that through to the user verbatim — don't fold it into "success".
6. **Errors section of the latest report.** The watcher's own `## Errors` section in the most recent report file lists upstream-fetch failures. Surface those to the user but **do not auto-retry them via this orchestrator** — the watcher will re-probe them on its next run. Manual investigation only.
7. **Only stamp `state/ingest/<skill>.json` on success.** If the skill threw, do NOT stamp. The next orchestrator run will re-detect the source as still-needing-ingest and re-queue the skill. This is the self-healing property — a transient failure isn't masked.
8. **Manual skill invocations don't stamp.** If the user runs `/update-rollcall` directly (outside the orchestrator), no marker is written. The orchestrator's next run will see `source.lastChanged > skill.lastSuccessfulIngest` and re-queue the skill. Since every tier-2 skill is idempotent on no-op input (rollcall walker finds no new sessions, financing scraper writes the same 15 years, etc.), this is wasteful at most — never wrong. The user can manually stamp after a direct run: `npx tsx scripts/stamp-ingest.ts <skill-name>`.
9. **The publish follow-up inherits every halt above.** Step 14 invokes `/upload-watch-changes` only for a run with no major issue (step 12's list). A run that halted on a failed skill, skipped past one, lost a post-step, or could not commit leaves the manifest pending and ASKS — it never publishes a corpus whose ingest it could not vouch for.
## What this skill does NOT do
- **Does not re-run the watcher.** State files are the input. If you want fresh fingerprints, run `npm run watch` first.
- **Does not PUSH.** It DOES commit — automatically, by explicit pathspec, at step 11 (that is this repo's one pre-authorised commit; everywhere else the rule is "commit only when asked"). Pushing stays the user's call. It never sweeps the index and never commits outside `data/`, `state/watch/`, `state/ingest/`, `state/perf/` and `state/upload/`.
- **Does not publish with its own commands.** Neither the bucket nor Cloud SQL — both are production — is touched by anything this procedure runs (step 6's `myarea:alerts:cloud` is the one exception: a derived feed that lives only in Postgres). It reloads LOCAL Postgres (inside each PG-backed skill), RECORDS the ordered `db:load:*:cloud` set and the touched bucket subtrees into `state/upload/pending.json`, and then — after the summary, and only when the run hit no major issue — invokes `/upload-watch-changes` (step 14), the ONLY path that publishes. On a run with a major issue it leaves the manifest pending and asks instead.
- **Does not judge its own timings.** It measures every step into `state/perf/process-watch-report.jsonl` and reports what is slower than usual; it never skips or reorders work to go faster. Optimisation is a separate decision made from the accumulated trace.
- **Does not auto-retry the watcher's Errors section.** Surfaced to the user only.
- **Does not silently skip failed skills.** A downstream failure halts the orchestrator until the user decides how to proceed (see Data-integrity contract above).
- **Does not skip a queued skill just because the latest report doesn't mention it.** The decision is state-driven (`lastChanged` vs `lastSuccessfulIngest`), not report-driven. Multi-day gaps still get fully ingested.
## Quick command reference
```bash
# Inspect today's report manually
cat data-reports/latest.md
# List recent reports
ls -t data-reports/ | head -5
# Trigger this orchestrator (you, by saying "process today's watch report")
# — the user invokes it via /process-watch-report in chat.
# --- the performance trace (Procedure steps 1, 4, 6-8b, 10) ---
npm run -s perf:step -- session # once per run; PASTE the id, don't use a var
npm run -s perf:step -- run --run process-watch-report --session "$S" --step X --phase derive -- <cmd>
npm run -s perf:step -- record --run process-watch-report --session "$S" --step X --phase ingest --started "$T0"
npm run -s perf:chain -- db:refresh --session "$S" --phase ingest # one row PER LINK
npm run -s perf:report -- --run process-watch-report --last 5
npm run -s perf:chain -- db:refresh --dry-run # list the links, run nothing
# --- the ingest → publish handoff (Procedure steps 9, 12) ---
npx tsx scripts/upload_manifest.ts show # is yesterday's ingest still unpublished?
npx tsx scripts/upload_manifest.ts write --session "$S" --commit "$COMMIT" \
--skills update-procurement --paths myarea,data_map.json --cloud "npm run db:load:pg:cloud"
# then: Skill(upload-watch-changes) ← step 14 invokes it after a run with no major issue;
# a run WITH one leaves the manifest pending and asks the user instead
# Cloud SQL commands the manifest carries — /upload-watch-changes runs them.
# Listed here because step 9 is where the set is RESOLVED (proxy on :5434 + .pgpass).
# Per-dataset (non-destructive; rebuilds cloud matviews too):
npm run db:load:pg:cloud # procurement contracts
npm run db:load:tenders:pg:cloud # tenders
npm run db:load:tr:pg:cloud # TR + NGO register
npm run db:load:funds:pg:cloud -- --full # EU funds — REFUSES without a scope flag;
# --full after an ИСУН re-ingest, else --payloads-only
npm run db:load:funds-fit:pg:cloud # sole applier of 143+144+145 (fit card, wire, hub cache)
npm run db:load:place-dim:pg:cloud # HARD prerequisite of 149 below
npm run db:load:municipal-fiscal:pg:cloud
npm run db:load:budget:pg:cloud # state budget: 152 КФП + 153 admin/programme
npm run db:load:budget-muni:pg:cloud # municipal side: 154 transfers/execution/capital
npm run db:load:budget-hub:pg:cloud # LAST — 156 reads all of the above
npm run db:load:ngo-funding:pg:cloud # NGO funding (EC FTS)
npm run db:load:awarder-seats:pg:cloud # awarder seats
npm run db:load:graph:pg:cloud # /connections graph — re-derive after tr / persons / contracts (run after persons-browse)
npm run db:load:tr-company-place:pg:cloud # place-companies tile — re-derive after tr / contracts / agri / funds
npm run db:refresh:risk:cloud # BOTH risk caches — after tr / contracts / debarred / КЗК / a re-applied migration
# Whole-DB reconcile (destructive --clean; local must be source of truth):
npm run db:sync:cloud -- --yes
````