CLAUDE.md · git:20261016.dd26116 · 2026-10-16 · sha256 bca6765170a70655
CLAUDE.md git:20261016.dd26116C
Immutable. This exact content never changes and is served at /api/v1/blob/bca6765170a70655.
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
**electionsbg.com** — A single-page application for visualizing and analyzing Bulgarian parliamentary elections since 2005. Live at [electionsbg.com](https://electionsbg.com).
The app is a JAMstack SPA: data is pre-processed offline into static JSON files in `/public`, then fetched client-side with React Query. There is no backend server.
## Commands
```bash
npm run dev # Start Vite dev server
npm run build # TypeScript check + production Vite build
npm run lint # ESLint (prettier runs as an eslint rule, not a separate step)
npm run preview # Preview the production build locally
# Tests
npm run test:unit # Vitest — unit + component tests (src/** jsdom, scripts/** node)
npm test # Playwright — E2E / SEO / perf / UI smoke (needs a built dist/)
npm run test:coverage # Vitest v8 coverage report
# Data pipeline (offline processing)
npm run data # Process election data (tsx ./scripts/main.ts)
npm run prod # Full pipeline with --all --prod flags
npm run sitemap # Generate sitemap
# Postgres (local Docker; db:refresh runs the whole load in order)
npm run db:pg:up # Start local Postgres (port 5433)
npm run db:pg:bootstrap # Create the app_readonly role (step 2 of db:refresh)
# LOCAL ONLY, and it refuses everything else: the serving
# database, any host:port that is not the docker-compose one
# (--yes-non-local to override), and a URL whose database name
# disagrees with the GRANT CONNECT in roles_readonly.sql.
# There is deliberately NO :cloud twin — creating a LOGIN role
# on Cloud SQL stays the hand-run, password-set-out-of-band
# step roles_readonly.sql's header describes. What covers the
# cloud side (and db:load:tr:pg, a REFRESH_EXCLUSIONS member
# this chain never reaches) is a WARNING on the DDL path:
# exec()/execEach() print once per process when the SQL grants
# to app_readonly and the role is absent. That matters because
# the guards INVERTED the failure — a bare GRANT used to 42704
# and roll its file back, loudly; a guard just skips, the load
# SUCCEEDS, and the objects carry no ACL until /api/db 42501s
# against a corpus that looks fully loaded.
# Why it exists: roles are CLUSTER-wide, so a virgin pgdata
# volume has no app_readonly, and 34 migrations GRANT to it
# bare. exec() sends a migration as ONE transaction, so the
# first of those 42704s and rolls its whole file back —
# measured, db:refresh died at db:load:pg applying 017.
# Invisible on any machine that ever ran the file by hand.
# bootstrap_roles.ts and grant_role_guard.test.ts pin this.
npm run db:refresh # Full reload: schema + every loader + resolve + generators
# + test:data
# "Every loader" is enforced: refresh_coverage.test.ts fails
# unless each local db:load:*/db:resolve:* script is in the
# chain or in REFRESH_EXCLUSIONS (scripts/db/refresh_coverage.ts
# — currently tr, cr-founding, company-founded, all keyed to
# gitignored caches). The nzok family + both ngo legs are IN
# the chain; the gitignored-input loaders skip-and-warn when
# their file is absent (fresh clone), and nzok-hospital runs
# with --tolerate-offline so an nhif.bg outage cannot abort
# the chain. A PRESENT-but-malformed input still throws.
# The same gate covers the two db:gen-* GENERATORS that write
# a committed artifact from PG (REFRESH_GENERATORS) — see
# "The two committed artifacts db:refresh regenerates" below.
# Deployment
npm run deploy # Deploy to Firebase (elections-bg project) — HOSTING ONLY
npm run deploy:db # Deploy the `db` Cloud Function (/api/db, the /officials 301,
# and the server-rendered /funds/contract + /company pages)
npm run staging # Deploy to Firebase staging (electionsbg-staging)
```
### The serving instance — and why every Cloud SQL timing below is an upper bound
**Prod Cloud SQL is `electionsbg-pg` (`elections-bg:europe-west3:electionsbg-pg`, proxy
`127.0.0.1:5434`, password in the repo-local `.pgpass`), and as of 2026-08-22 it is
`db-perf-optimized-N-2` on the ENTERPRISE_PLUS edition** — PG16, 37 GB PD_SSD, ZONAL.
Reported by the server: `shared_buffers` **5332 MB**, `effective_cache_size` ~6.25 GB,
`work_mem` 16 MB.
⚠️ **It was `db-g1-small` until then — a SHARED-CORE box with ~1.7 GB of RAM and
`shared_buffers` of 128–300 MB — and essentially every Cloud SQL wall-clock figure written
anywhere in this repo predates the upgrade.** That includes the ones in this file, in
`.claude/skills/**`, in `docs/plans/**` and in the assistant's memory. Those numbers were
honestly measured; they are now **upper bounds**, not predictions.
Two rules follow, and the second is the one that matters:
- **Never relabel an old measurement with the new tier.** A figure keeps the box it was
measured on ("measured 2026-08-06 on the then-current `db-g1-small`"). Restamping it
`db-perf-optimized-N-2` manufactures a claim nobody made — the exact defect this file
warns about everywhere else.
- **The upgrade does NOT retire a single *ordering* or *locking* rule.** `shared_buffers`
moved from ~0.3 GB to 5.3 GB, so the RAM-starvation arguments are weaker — but an
AccessExclusiveLock still blocks readers, a `DROP … CASCADE` still deletes dependents, a
matview created `WITH NO DATA` still raises `55000`, and the `10 s` pool
`statement_timeout` (set by the app, not the server — the server's own is `0`) is
unchanged. Read a fast timing as "this window is shorter", never as "this window is gone".
`docs/plans/**` deliberately keeps its original figures: those are dated design records
whose reasoning ("we precomputed this because it was 801 ms") was correct when written, and
rewriting them would falsify the record.
**Re-measured on `db-perf-optimized-N-2`, 2026-08-22**, one clean end-to-end publish (contracts
409,848 rows / tenders 237,941). A step absent from this table has NOT been re-measured:
| step (`…:cloud`) | db-g1-small | db-perf-optimized-N-2 |
| ---------------- | ----------- | --------------------- |
| `db:load:pg` (contracts) | 4077 s, then 722 s after the normalcy ship | **426 s** |
| `db:load:tenders` | 363 s | **318 s** |
| `db:load:procurement-scopes` (30 windows, 6 matviews) | unmeasured ("expect minutes") | **87 s** |
| `db:load:transport-project-map` | — | **210 s** (the TRUNCATE form; stage-merged since 2026-10-16, not re-measured) |
| `db:load:awarder-seats` | ~3 s (before it refreshed matviews) | **59 s** |
| `db:load:persons-browse` | — | **35 s** |
| `db:load:graph` | — | **28 s** |
| `db:load:tr-company-place` | — | **27 s** |
| `db:load:annexes` | — | **25 s** |
| `myarea:alerts` | — | **22 s** |
| `db:load:employer-links` | — | **19 s** |
| `db:load:mvr-directorate-map` | — | **12 s** |
| `db:load:grant-links` | — | **5 s** |
| `db:load:water-operator-map` / `transport-facility-map` | — | **2 s / 4 s** |
| **whole 15-step publish** | — | **21m20s** |
| `prices:payloads` | 256 s (4m16s) | **677 s (11m17s)** ⚠️ SLOWER |
**Re-measured again on `db-perf-optimized-N-2`, 2026-08-27**, a second clean publish (contracts
407,371 / tenders 238,082). Same box, five days later — so these are NOT a second opinion on the
table above, they are the steps that table left unmeasured, plus two figures elsewhere in this
file that were still quoting the retired `db-g1-small`:
| step (`…:cloud`) | previously documented | 2026-08-27 |
| ---------------- | --------------------- | ---------- |
| `db:load:tr` | **34.9 min** (db-g1-small) | **280 s** — 7.5× off |
| `db:load:rollcall-derived` → `mp_similarity` | **744.5 s** (db-g1-small, 2026-08-06) | **70.9 s** — 10.5× off, i.e. parity with local (67 s) |
| `db:load:pg` (contracts) | 426 s (2026-08-22) | 305 s |
| `db:load:tenders` | 318 s (2026-08-22) | 263 s |
| `db:load:awarder-seats` | 59 s (2026-08-22) | 56 s |
| `db:load:employer-links` | 19 s (2026-08-22) | 17 s |
⚠️ **The two top rows are the ones that mattered, because both were being used to AVOID work.**
"the cloud TR publish is not reader-safe: it is 34.9 min" and "budget a quarter of an hour and do
not chain `mp_similarity` behind anything urgent" were both being read as reasons not to run a
step; both were measured on the retired shared-core box and neither survives contact with the
current one. Both call sites below have been corrected in place to point here. The TRUNCATE
window on a TR publish is under five minutes, and the roll-call derived rebuild is a ~70 s step.
Re-measure before quoting either as a reason not to run something.
⚠️ **THE UPGRADE DID NOT MAKE EVERYTHING FASTER, AND THE ONE THAT GOT SLOWER IS THE LESSON.**
Server-side-dominated steps improved (contracts 1.7×, because its cost is matview refreshes,
the 30-scope risk precompute and index builds). Steps that stream large RESULT SETS back
through the proxy did not: tenders only 1.14×, and `prices:payloads` — whose
`loadGridsFromPg` pulls millions of rows client-side — came out **2.6× slower**. The plausible
reading is that the proxy round-trip path, not database CPU or RAM, bounds that class of work,
and `shared_buffers` cannot help it. One sample each and the proxy had been restarted
mid-session, so treat the direction as the finding and the ratio as provisional.
**Do not use "the box is bigger now" to justify deleting a precompute or running an aggregate
live.** Two independent facts about this run say the old shape still holds: the 6 scoped
matviews are still refreshed on every contracts load, and the whole publish is still 21 minutes.
The one thing that DID change is `tender_normalcy`, which used to be unbuildable on cloud
(the rank() sort spilled past `temp_file_limit` → **53400**, so shipping from local was the only
option) and now reports `built in-place on cloud (ship retired — v2-f)`.
⚠️ **The Cloud SQL proxy is a real failure mode, and it fails as `Connection terminated
unexpectedly`.** It died mid-run during this publish and took the chain with it (nothing was
corrupted — the load had not committed). If a `:cloud` step dies with that message, check
`nc -z 127.0.0.1 5434` before suspecting `temp_file_limit` or the database. Restart with
`npm run db:proxy:cloud`.
**Every URL this repo emits is the NO-slash form.** Hosting runs `"trailingSlash": false`
(`firebase.json`, hosting.main), so `dist/<path>/index.html` serves at `/<path>` and `/<path>/`
301s back to it. Canonicals, `og:url`, `hreflang`, sitemap `<loc>`, the ~350 `href="${SITE_URL}/…"`
links in `scripts/prerender/`, `functions/spa_page.js` and `scripts/llms/buildIndex.ts` must all
follow that. **The two roots invert the rule**: the bare `/` keeps its slash (hosting never strips
it) while the EN root is `/en`, NOT `/en/` — that asymmetry is the one thing to get right in any
new URL builder, and emitting `/en/` gives the EN homepage a canonical that 301s. Until 2026-08-03
hosting used the default (slash-adding) behaviour while the code emitted no-slash, so every
canonical on ~248k pages named a redirecting URL; see `docs/plans/parliament-hub-v1.md` §2.8.
`tests/seo.spec.ts` gates both the redirect direction and that no declared canonical / `og:url` /
`hreflang` redirects — the older test asserted only that the canonical _string_ was right, which is
why this survived unnoticed.
⚠️ **`npm run preview` INVERTS that rule, so never read a prerendered page through it.**
`vite preview` runs the SPA fallback, so the no-slash URL — the one production serves the
file at — falls through to `dist/index.html`, i.e. the HOMEPAGE prerender with the
homepage's `<title>`, canonical and `<h1>`; only `/<path>/` serves the real file. Measured
2026-08-22 on localhost:4173: `/procurement` returned „Парламентарни избори в България —
последен вот: 19 април 2026" and `/procurement/` returned „Обществени поръчки — договори и
народни представители", while the live site returns the latter at the no-slash URL. Read
`dist/<path>/index.html` directly, or `curl` the deployed site;
`scripts/prerender/distHeadings.data.test.ts` does the former for exactly this reason. It
has already produced one false finding — „every prerendered page serves two `<h1>`s, and
the first says it is about the April 2026 election" — against a `dist/` in which no page
carried two and every page carried its own.
**The prerender resolves `VITE_DATA_BASE_URL` a SECOND time, and it must agree with the
bundle's copy.** A route may declare `preloadData` (`scripts/prerender/routes.ts`) — the data
files it fetches on first render — and `scripts/prerender/index.ts` emits each as
`<link rel="preload" as="fetch" crossorigin fetchpriority="low">`. The href is built by
re-resolving the data origin through Vite's `loadEnv`, so prerender correctness now depends on
the **gitignored** `.env.production` being present and on the env MODE matching the one
`vite build` used. It is pinned to the literal `"production"` for that reason: `vite build`
takes no `--mode` anywhere in `package.json`, and Vite sets `NODE_ENV` _from_ the mode, so
reading `NODE_ENV` back would be an independent input that can silently disagree.
A mismatch is **not** a build failure. It yields an empty base, every hint becomes a
same-origin `/macro.json`, and `data/**` is not deployed to hosting — so the catch-all answers
with the SPA shell at **200**. Per page, in both languages, that is four wasted shell downloads
and four dead hints, with the real fetches still going to the bucket afterwards. Same failure
signature as the `/court/**` build-time dependency below: exit 0, quietly worse pages.
Two gates cover it. `scripts/prerender/index.ts` refuses to write when the resolved base is
absent from the built entry chunk, and `tests/perf.spec.ts` compares each emitted href's origin
against that chunk. `fetchpriority="low"` is load-bearing rather than cosmetic — `as="fetch"`
defaults to HIGH, which puts the data in bandwidth competition with the render-blocking JS the
page needs in order to paint at all; measured, HIGH is a net LCP loss at both 1.6 Mbps and
10 Mbps. Note the whole hint set is still a small net loss at 1.6 Mbps, so re-measure before
adding a fifth path to a route.
⚠️ **CI IS A THIRD RESOLVER OF THAT SAME VARIABLE, AND IT HAS NO `.env.production` — the
file is gitignored.** With an empty base `dataUrl()` is the identity, so every `data/**`
fetch goes SAME-ORIGIN; the Playwright suite runs against the Firebase hosting emulator,
which serves only `dist/` (and whose `ignore` list drops `procurement/**/*.json` outright),
so those requests reach the SPA catch-all. **They come back 200 with the SPA SHELL, stamped
`Content-Type: application/json` by `firebase.json`'s `**/*.json` header rule** — a fetch
that looks successful right up to `JSON.parse`, after which React Query never resolves and
every hub KPI band renders skeletons. `.github/workflows/test.yml` therefore sets
`VITE_DATA_BASE_URL` on its Build step, and `scripts/bucket_cors.json` carries
`http://127.0.0.1:5002` (the Playwright `baseURL`) so the browser is allowed to read the
bucket cross-origin. **Both halves, or the symptom is identical** — a missing CORS entry and
a missing base fail the same way.
That is what failed in CI run 33036199067: eight `hub head — the §3.0 height budget` tests at
**0 `data-kpi-cell`s**, and in every one of them the HEIGHT assertion PASSED, because a head
that lost its band is comfortably inside its budget — exactly the case the `cells` count
exists for. Only the nine hubs reading `/api/db/*` (forwarded to the DEPLOYED function) or
`/parliament/votes/**` were green; the second is reachable because `public/parliament/votes`
is a **tracked** symlink into committed data, while `public/procurement` and `public/myarea`
are untracked local symlinks that exist on a developer's machine and nowhere else. Read a
green data-backed browser test on a fresh clone with suspicion for that reason.
**`npm run deploy` ships hosting only.** When a change spans hosting and the `db` function
— a new `/api/db` route, a new hosting rewrite pointing at it — deploy in this order:
1. the Cloud SQL migration (`npm run db:*:cloud`),
2. `npm run deploy:db`,
3. `npm run deploy`.
Hosting first means the rewrite is live against a function that cannot serve it yet.
**`/funds/contract/**`and`/company/**` are page URLs served by that function**
(`functions/spa_page.js`), not static files. They exist because both families were serving
the SPA shell — i.e. the HOMEPAGE's `<title>`, description and canonical — so to a crawler
all 81,910 contract URLs were duplicates of the homepage. Prerendering them is not an
option: `dist/` already holds ~248k files and Firebase's ceiling is on file COUNT
(a 453k-file dist has failed to deploy), while these two families are ~256k more.
Deploying the hosting rewrite BEFORE the function is the one ordering that breaks a page
that works today: the rewrite would route every contract and company URL to a function
with no handler for it. `deploy:db` first, then `deploy`.
**`/procurement/contract/**` joined the family on 2026-10-13** (~417k contracts, the same
file-count wall), so it carries the same ordering rule — `deploy:db` before `deploy`, or the
rewrite points every contract URL at a function with no handler for it. Amendment rows
(`tag = 'contractAmendment'`, 6,764) share the key space and are worded as amendments, never as
their base contract.
**`/council/resolution/**`is a fifth member, and the only one that is function-served for a
CONTENT reason rather than a file-count one.** 4,813 resolutions as of 2026-08-25 (9,626 with the EN mirror)
would fit under the Firebase ceiling comfortably — but each body is one title and a vote
table, the shape that earns a thin-content penalty rather than traffic. So they get a real
head from the function and deliberately **no sitemap`<loc>`and no prerender**: discoverable
by a crawler already on the council page, never submitted en masse. The only inbound link is
the resolution title in`CouncilScreen`; without it the whole family is unreachable, which is
also what makes a routing bug here invisible to manual testing. Same ordering rule,
`deploy:db`before`deploy`.
Two things about it are easy to get backwards, and both shipped once:
- **The id charset must admit Sofia.** Fifteen of the sixteen council keys are `AAA99`;
Sofia's synthetic key is a bare `SOF`, so a `[A-Z]{3}\d{2}` id regex excluded all 413 Sofia
resolutions. `isSpaPagePath` still routed them to the function, `matchSpaPage` returned
null, and the fallthrough served the HOMEPAGE's head plus a `noindex` — on the largest
council in the corpus, cached an hour at the edge. `spa_page.council.test.js` derives its
cases from the committed shard tree for exactly this reason: every hand-picked example is
one of the fifteen that worked.
- **A link must use the FRONTEND code, never `obshtina_code`.** `/council/:code` resolves
through `council_muni_code` only, and eight of the sixteen council keys are not frontend
codes — three (BGS01, PDV01, VAR01) are OTHER municipalities' codes. Linking the internal
key put "we do not track this council" one click from that council's own decision, for
1,980 of 4,813 resolutions (measured 2026-08-25). `council_resolution_detail()` therefore returns
`councilFrontendCode` beside `councilCode`, and a NULL there means render plain text.
**`/person/*` is a sixth member of that family, and the one easiest to get wrong.** The
`/person/*` + `/en/person/*` rewrites (`functions/person_redirect.js`) serve the 301 from
`person_slug_retired` — 23,916 slugs a re-resolve retired, which before 2026-08-08 returned
200 with the homepage's title and canonical and then noindexed themselves client-side. Same
ordering rule, `deploy:db` first. Three ways it differs from the three above:
- **It is only PARTLY function-served.** 25,167 person pages are prerendered and Firebase
ranks exact-match static content above rewrites, so the function only ever sees the other
~101k. A missing rewrite therefore DEGRADES rather than breaks — which makes it easier to
miss, not safer.
- **The rewrite must stay single-segment (`*`, never `**`)**, and the `/person/\*\*`HEADER
entry must carry no browser`max-age`: that value is read by the 25,167 static pages, and a
browser-cached one pointing at a deleted `/assets/index-<hash>.js`is a white screen`main.tsx`'s stale-chunk recovery cannot reach (it only fires on dynamic-import failures).
`scripts/deploy/firebase_person_rewrite.test.ts` holds both.
- **The handler owns every `/person` URL the rewrite reaches**, so anything it does not
redirect it serves as the SPA shell — never a 404, or the ~101k non-prerendered people go
with it. Those still serve the homepage's head; giving them their own via a `loadPerson`
arm on `spa_page.js` is open work.
The function fetches the SPA shell from `https://elections-bg.web.app/` (Hosting's default
domain — never the Cloudflare-fronted naiasno.bg, whose bot rules could challenge a
server-to-server fetch; see `SPA_SHELL_ORIGIN` in `functions/index.js`) and swaps the
prerender's `<!-- SEO -->` / `<!-- BODY -->` marker blocks, caching it per instance for
`SPA_SHELL_TTL_MS` (10 min). Two consequences worth knowing: a cold instance makes one
extra outbound request, and if that fetch fails the page still serves correct head tags
without the SPA bundle (`FALLBACK_SHELL`) — complete for a crawler, degraded for a human,
which is the right way round for a failure nobody is watching.
⚠️ **This does NOT make the function deploy-order-free, and this paragraph claimed it did
until 2026-08-16 ("nothing needs re-deploying when the bundle hash changes"). Ship hosting
BEFORE the function whenever the bundle hash moves.** A WARM instance holds the PRE-deploy
shell for up to ten minutes, so a `deploy` that replaces `dist/` — deleting the previous
`/assets/index-<hash>.js` — leaves every function-served page advertising a script that is
gone. Firebase's catch-all answers that path with the SPA shell at **200 `text/html`**,
which fails strict MIME checking for a module script, so NO JS runs at all: a white screen
`main.tsx`'s stale-chunk recovery cannot reach, because it only listens for dynamic-import
failures and this is the ENTRY bundle. Measured on the 2026-08-16 declared-crypto deploy,
which is how this note came to exist.
**The edge is what turns a 10-minute window into an hour-long one.** `/person/**` carries
`s-maxage=3600` (and `spa_page.js` sets the same on its own responses), so the stale HTML
the warm function served AFTER hosting purged the CDN gets written back into it and pinned
— `x-cache: HIT`. The function's TTL then self-heals while the edge does not.
**THREE steps, always, and the third is not optional — an earlier draft of this note said
two and was wrong twice over.** No two-step order can avoid the window, because the
function can only fetch the new shell AFTER hosting is live, and only a HOSTING RELEASE
purges the edge. So whichever of the first two comes first, warm instances repopulate the
CDN with stale HTML in between, and it stays pinned for `s-maxage`:
```bash
npm run deploy # 1. hosting live with the new bundle
npm run deploy:db # 2. fresh instances fetch the CURRENT shell
SKIP_PREDEPLOY=1 npm run deploy # 3. purge the edge entries step 2 could not
```
Step 3 takes `SKIP_PREDEPLOY=1` deliberately: `dist/` is already the tree step 1 built and
validated, and the goal is a new RELEASE (which purges) rather than a rebuild — without it
the predeploy re-runs lint, both test suites and a ~10-minute build for nothing. When a NEW
`/api/db` route is involved the function must lead, so the order becomes
`deploy:db` → `deploy` → `deploy:db` → `SKIP_PREDEPLOY=1 deploy`.
Measured twice on 2026-08-16, once per deploy: after step 1 the homepage served
`index-pCBzDm2m.js` while `/person/mp-3643` still served the deleted `index-Zcb7Mede.js`,
and after step 2 the function was correct on a cache-busted request while the plain URL
stayed stale until step 3. `/company/**` and `/funds/contract/**` recovered at step 2 both
times and `/person/**` did not — do not read one family as evidence for the others. Verify
with
```bash
curl -sL https://electionsbg.com/person/mp-3643 | grep -oE '/assets/index-[^"]+\.js'
curl -sL https://electionsbg.com/ | grep -oE '/assets/index-[^"]+\.js'
```
— the two hashes must match. ⚠️ **`-L` is required**: electionsbg.com now 301s to
naiasno.bg, so a bare `curl -s` returns the redirect body, both greps print nothing, and
two empty outputs read as a match. The 25,167 PRERENDERED person pages are never affected (they
carry the build's own hash), which is exactly what makes this easy to miss: spot-checking a
person page picks a prerendered one and looks fine.
The same migration-before-writer rule applies to the hand-run ingests that have no
`db:load:*:cloud` wrapper. `company_founded` writes `http_status`/`attempts`, so
`033_procurement_risk_indexes.sql` must be applied to the target database first:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts 033_procurement_risk_indexes.sql
```
The founding-date writer (`upsertFoundingDates`) preflights and throws with that command in
the message if the columns are missing.
**CR Deeds full-capture** (`docs/plans/cr-deeds-capture-v1.md`) supersedes the old
`fetch_company_founded` crawl. The rate-limited crawl is an operator action — `npm run
tr:cr-deeds` (tiers 0/1/2a/2b/3; `--probe` first to gauge the block state); it writes the
durable raw store `raw_data/tr/cr_deeds.sqlite` (gitignored, never uploaded). Two projections
read that cache offline, no re-fetch:
- **owners** → `tr:daily-refresh` runs the persons projection automatically (inside
`daily_refresh.ts`, additive into `company_persons`), then `db:load:cr-founding:pg` folds
the founding dates into local `company_founded`. To publish owners + founding to prod, run
`npm run db:load:tr:pg:cloud` and `npm run db:load:company-founded:pg:cloud` (the latter
ships local `company_founded` → Cloud SQL, 033 applied there first). Nothing on the cloud
side is automatic.
- **declared activity (НКИД→CPV mismatch)** → `db:load:cr-nkid:pg` (in `tr:daily-refresh`,
a `REFRESH_EXCLUSIONS` member) parses each capture's `CR_F_6a_L` into `company_nkid`
(eik→КИД-2008 division) and reseeds the `nace_cpv_*` crosswalk from the committed
`src/lib/naceCpv.ts`. It backs the `nkidMismatch` risk flag (bit 12 of
`contract_risk_cache`) + the `/company` "% outside declared activity" chip. **The division
is classified from the LABEL, not the code** (`src/lib/naceLabel.ts`) — the НКИД field mixes
НКИД-2003 and КИД-2008 codes that reuse division numbers for different sectors, so a
code-based parse fired ~20k flags, thousands false (see `docs/plans/nkid-cpv-mismatch-v1.md`).
Cloud publish, in order: apply `033`+`112` (company_nkid shell + nkidByEik payload + bit-12
mask + crosswalk shells) via `apply_functions.ts`; `npm run db:load:cr-nkid:pg:cloud`
(fills company_nkid + crosswalk, refreshes `procurement_risk_indexes_cache`); then
`SELECT rebuild_contract_risk_cache();` on Cloud SQL so bit 12 reaches the served masks;
then `npm run deploy:db` for the `/api/db/company` `nace_div` field (additive — the page
renders without it). Nothing on the cloud side is automatic. Skipping the rebuild leaves the
contracts browser's risk column without the flag while local has it.
`tender_subcontracting` (migration 171, `db:load:subcontractors:pg`) is the declared
**подизпълнители** answer off each ЗОП award notice — „the money on a contract is not the
money that reaches the work". 53,854 declarations, 616 of them Да.
```bash
npm run db:load:subcontractors:pg:cloud
```
**It is a PROJECTION over `tender_notice`, not an ingest**, so its staleness trigger is
`db:load:tender-dossier:pg` and nothing else — re-run it after every dossier load, on both
sides. It reads no files and no network.
Three things about it are easy to get backwards:
- **NULL is a third answer and it is modelled as the ABSENCE OF A ROW.** 159,107 of 212,961
notices do not carry the question at all. Storing those as `has_subcontractors = NULL`
would let any consumer reading the column render „the winner performed it alone" about a
named contract nobody asked about — so the table holds only ANSWERED notices, and
`tender_subcontracting_for()` `GROUP BY`s specifically so an unknown УНП returns **no
rows** rather than one row of nulls (an ungrouped aggregate returns a row over an empty
set — that is how the leak got in).
- **The answer must end at a NON-LETTER.** `(Да|Не)` without a boundary reads „**Да**та на
сключване" as Да and „**Не**приложимо" as Не. A fabricated Не breaks the whole design; a
fabricated Да publishes a claim a named buyer never made and is shape-identical to a real
one, so no gate can catch it. Use `(?![\p{L}\p{N}])` with the `u` flag — **not** `\b`,
which is ASCII-only and never matches after a Cyrillic letter.
- **All four fields come from ONE occurrence of the block.** A notice can carry it twice (an
award plus a correction); matching each label independently across the document can take
„Да" from the first and its count from the second, producing a row that appears in neither.
`ted_notice` / `ted_coverage` (migration 172, `db:load:ted:pg`) are the TED cross-check —
217,525 Bulgarian notices, 99.8% carrying a buyer ЕИК that joins `contracts.awarder_eik`.
```bash
npm run db:load:ted:pg:cloud
```
⚠️ **The early years are the API's INDEX RAMP, not a trend.** TED's v3 index returns 0
notices for 2015 and 4,687 for 2016 against ~17,000 for 2019. Empty years are dropped rather
than recorded as zero — a stored zero would plot as „Bulgaria published nothing above the EU
threshold that year", the exact false finding this dataset exists to prevent — and
`ted_coverage` carries the per-year counts so the ramp stays visible. `ted_buyer_reconciliation()`
therefore takes an explicit date range and has no „all time" default.
`isun_clean_contract` / `isun_clean_beneficiary` (migration 175,
`db:load:clean-delivery:pg`) are the ИСУН „clean delivery" register — what the state
publishes about EU-funded contracts that ended with no financial correction. Source is
COMMITTED (`data/funds/clean_delivery.json`), so the loader is pure-load; rebuild it with
`npm run funds:clean-delivery`.
```bash
npm run db:load:clean-delivery:pg:cloud
```
⚠️⚠️ **IT IS AN ACHIEVEMENT REGISTER, NOT THE COMPLEMENT OF „WAS CORRECTED", and
inverting it is the one thing that must never happen.** Plan P9 asked which contracts were
clawed back. That is **not derivable** — a project can be absent because it finished LATE,
was terminated (3,656 `Прекратен` rows in our corpus), or is still in final verification.
Individual irregularity records go to OLAF's **IMS, which is confidential**; there is no
public complement anywhere. Subtracting these tables from `fund_projects` would manufacture
accusations against named beneficiaries out of ordinary lateness — the one direction that
cannot be walked back. `isun_clean_delivery_coverage.absence_meaning` is **NOT NULL** so a
row without that sentence cannot exist, and `isun_clean_delivery_for_eik()` returns it
beside every figure so a page cannot render the number without the caveat.
Four things about it are easy to get backwards:
- ⚠️ **THE JOIN KEY IS `contract_number`, NOT `reg_no`.** ИСУН's registration number carries
a `-C##` contract-VERSION suffix (`…-0001-C01`) that `fund_projects.contract_number` does
not. Measured 2026-09-02: the raw value matches **0 of 10,016**; the stripped base matches
**10,016 of 10,016**. The data test carries a mutation check on exactly this, because a
gate asserting „the join works" passes on an implementation that quietly stopped stripping.
- ⚠️ **THE TWO REPORTS DISAGREE AND MUST BE LEFT TO.** 10,016 clean contracts against
**40,289** on-time contracts declared by clean beneficiaries (2026-09-02; 9,940 against
41,530 on 2026-08-20 — the gap is not a fixed ratio and both sides move). That looks
exactly like a ~10,000-row export cap (BULSTAT's 999 ceiling, ЦПРС's cartesian product)
and **is not** — each listing prints its own pager and they agree with the exports to
within a part-page: contracts „Страница (1/401)" → 401×25 = 10,025 against 10,016,
beneficiaries „(1/1364)" → 34,100 against 32,559 + 1,534 excluded = 34,093. So the
4× gap is real: the reports count different populations (projects with no imposed
correction vs on-time contracts of beneficiaries with none). **Do not partition the export
per programme to „fix" it** — there is no cap, and `GetProgrammes` is WAF-blocked anyway.
- ⚠️ **„В СРОК" is a STRICTER test than „clean".** The beneficiary column is „Брой договори,
успешно приключени **в срок**" — on time. A contract can be on-time-but-corrected or
late-but-clean, which is why the two numbers cannot be reconciled.
- **ORGANISATIONS ONLY, and the omission is counted.** 1,534 beneficiary rows are natural
persons published with a first name and no id („Христо", org type „Друга"), and 2 carry a
10-digit **ЕГН**. Neither is stored — a first name identifies nobody and joins to nothing,
and an ЕГН is a personal identifier this project does not hold. `cleanEik()` drops them at
PARSE time rather than filtering downstream, so no consumer can reach one, and
`coverage.natural_persons_excluded` makes the exclusion visible.
- ⚠️ **`isun_clean_delivery_for_eik()` DRIVES FROM BOTH REGISTERS, and `on_time_contracts`
is NULL — never 0 — for a company in the contract register only.** It drove from
`isun_clean_beneficiary` alone until 2026-09-02, so it returned NO ROW for the **962 EIKs
/ 1,755 clean-contract rows (17.5% of the register)** that have no beneficiary entry: the
`/company/:eik` tile never mounted and ИСУН's own named, uncorrected projects were
discarded at the query. `beneficiary_listed` carries the distinction explicitly, because
„not listed as a correction-free beneficiary" and „listed with zero on-time contracts" are
different claims and only the second is a number — and the corpus cannot separate them
(**0 of 32,559** beneficiary rows carry a literal 0), so a consumer that coalesces the NULL
publishes the first as the second, on the one dataset here where a fabricated zero is an
accusation. The function also returns the named `contracts`, so a surface shows the
evidence instead of leaving a reader to subtract two counts that measure different things.
⚠️ It is `DROP FUNCTION` + `CREATE` under a **role-guarded** GRANT — the tables are
`IF NOT EXISTS` and keep their ACLs, the function does not — so re-applying 175 to a
database with no `app_readonly` REVOKES its EXECUTE, reports success, and leaves
`/api/db/company` raising 42501 on that arm until `db:pg:bootstrap` runs and 175 is
re-applied. Gate: `scripts/db/tests/isun_clean_delivery.data.test.ts`.
**The input fetches itself since 2026-09-02** — `npm run funds:clean-delivery -- --fetch`.
It was operator-downloaded until then, and the reason that changed is worth knowing, because
the same wall sits in front of the beneficiaries/projects exports (`isun_download.ts`) and
the old note there said a fancier client does not help.
The F5 in front of 2020.eufunds.bg has **two independent triggers**, isolated 2026-09-02 by
varying one at a time, repeatedly and hours apart:
```
node, no Referer → 245-byte „Request Rejected"
node, with Referer → 245-byte „Request Rejected"
curl, with Referer → 245-byte „Request Rejected"
curl, no Referer → the real XLSX, first time
```
So it refuses the node CLIENT whatever headers it sends — no shape gets through, including
curl's own UA, which makes it a TLS-handshake fingerprint rather than anything a header can
change — **and** it refuses ANY client that sends a `Referer` on these endpoints, including
the browser-shaped one `isun_download.ts` sets deliberately. Both ingests therefore fetch
through curl with no Referer. ⚠️ The RATE-BASED refusal that `isun_download.ts` documents
from 2026-08-05 is real and unretired; it is simply not the only mode, and „retry later" is
the wrong advice for the other one — `isun_clean_delivery` had errored on all 13 daily
watcher runs since it was wired, never once succeeding, while curl answered fine throughout.
`GetProgrammes` is a SEPARATE wall and stays blocked even from the page's own origin, which
is why the programme list is still derived from the rows.
`data/_cache/isun_clean_delivery/*.xlsx` is gitignored and `--fetch` only refreshes it before
the parse, so a re-run without the flag reproduces the same corpus from the same drops. A
hand-download still works: save the two files as `contracts__ALL.xlsx` and
`beneficiaries__ALL.xlsx`; the ingest folds several drops by key, so a future split works
unchanged.
The gate is `scripts/db/tests/isun_clean_delivery.data.test.ts` (12 tests) plus 14 pure
parser tests that need no Postgres. One of the twelve asserts the two reports still
DISAGREE, so that a future change „reconciling" them fails loudly rather than quietly
publishing a fabricated equality.
`aop_expert` / `aop_expert_area` (migration 174, `db:load:aop-experts:pg`) is the АОП
register of external experts under чл. 232а, ал. 2 ЗОП — the people a contracting authority
may co-opt onto an evaluation committee. Its source is COMMITTED
(`data/procurement/aop_experts.json`, 88 rows), so the loader is pure-load and a fresh clone
builds the table with no network; `npm run aop:experts` re-crawls.
```bash
npm run db:load:aop-experts:pg:cloud
```
⚠️⚠️ **THE REGISTER IS HISTORICAL — this is the first thing to know about it and every
surface must phrase it in the PAST tense.** Measured 2026-08-20 over the full crawl: 88
experts and **not one still valid** — the newest validity ended **2023-01-01** and no expert
has been added since **2020-01-01**. It answers „who WAS an approved external expert between
2017 and 2023"; it cannot answer „who is available now". `aop_expert_coverage` carries the
window so a surface states it rather than inferring it from the rows it happens to hold, and
`aop_expert_table` derives `is_current` at QUERY time — the `open_calls` rule (142), for the
same reason. Postgres enforces it here by accident and usefully: `CURRENT_DATE` is not
immutable, so a GENERATED column is refused outright.
Four things about it are easy to get backwards:
- ⚠️ **THE NAME IS TWO PARTS, and the link must REFUSE rather than grade.** The register
prints given + family only — its own form marks Презиме „за служебни цели и не се отразява
в публичната част" — while `person` holds three. So a link rides a weaker key than anything
else in this repo joins on. Measured: **58 of 88 match at least one person, but only 25
match exactly one.** `aop_expert_person_links()` returns those 25 and reports the other 33
as refused; it does not score candidates and must never be changed to. Naming the wrong
individual as a state-approved procurement expert is the harm the refusal exists to prevent,
and „25 of 88 are in our person layer" and „25 matched, 33 more share their name" are
different claims of which only the second is true.
- ⚠️ **VALIDITY BELONGS TO THE (EXPERT, AREA) PAIR, NOT TO THE EXPERT.** An expert admitted
to a second competence area later carries a different window there — 4 of 88 do (ЕТС-49 Анна
Савова is 2019→2022 in one area and 2020→2023 in another). A scalar pair on the expert holds
one of two true answers, picked by whichever area the crawl visited first. The facts live in
`aop_expert_area`; the pair on `aop_expert` is explicitly the UNION. The defect was found by
the WATCHER, whose fingerprint keys on (УНЕ, validity) and reported 92 against the ingest's
88 — a per-expert fingerprint would have agreed with the bug.
- **There is no „all areas" query.** The register has 77 competence areas and submitting the
blank form re-renders the form with no result table, so membership is only obtainable as one
query per area. An expert may hold several (28 of 88 do), so the per-area counts sum to more
than the register — **dedupe on УНЕ, never sum the areas**.
- **The page is windows-1251 and says so only in a `<meta>`.** `Response.text()` decodes UTF-8
ALWAYS, per the fetch spec, regardless of the Content-Type charset — so the default path does
not throw, it stores a corpus of mojibake names that passes every row count. `fetchText` takes
an `encoding` for this.
Finding it is worth recording, because the obvious search does not: aop.bg's own navigation
does not link the register, and `www.aop.bg/ee2014.php` — the URL the ЦАИС bundle carries for
the чл. 229 list — is a **404**. The live URL is the other config key in the same object
(`externalExperts229AUrl` in app.eop.bg's main bundle), and the page behind it is a plain 1990s
PHP GET form, not the Angular SPA that links to it.
The gate is `scripts/db/tests/aop_experts.data.test.ts` (9 tests) plus 13 pure parser tests
that need no Postgres. Two of the nine exist only to prove the refusal still DISCRIMINATES —
one asserts a non-trivial number of names are refused, the other re-runs the join without the
guard and requires strictly more experts, so a guard that had silently stopped filtering cannot
pass. The parser refuses a page whose row count disagrees with the register's own „Общ брой",
because a partial competence area is indistinguishable from a small one.
`procurement_annexes` (migration 114, `db:load:annexes:pg`) is the same shape: it resolves
against the `contracts` table and reads the raw ЦАИС ЕОП annex cache, so on the cloud side run
`npm run db:load:annexes:pg:cloud` **after** the contracts corpus is loaded and whenever
`ingest_anexi` refreshes the cache — otherwise the per-annex breakdown (and the чл.116 ал.2
vs ал.3 labeling on the contract page) goes stale on prod while local is current. `db:refresh`
runs the local equivalent automatically (after `db:load:pg`); nothing runs it on the cloud side.
**It is also mandatory after the cross-source reconcile, for a second reason** — evicting a
duplicate row orphans that row's `procurement_annexes` entries (16 across 9 contract keys on the
2026-08-04 run), and only this loader re-resolves them against the reloaded corpus.
### Cross-source reconciliation — the pass that must run between backfill and load
The contracts corpus is built from FOUR feeds distinguished by `release_id` prefix — `aop-legacy-`
(АОП annual CSV), `eop-` (ЦАИС ЕОП flat договори), `ocds-` (АОП OCDS export), `rop-` (РОП). Each
splits a contract's value across its OWN view of the supplier set, so when one contract arrives
from two feeds the rows **cannot be summed** and the corpus over-states.
`scripts/procurement/reconcile_cross_source.ts` removes those duplicates on the SHARDS (not in
SQL — `pg_roundtrip.data.test.ts` asserts Postgres is a lossless capture of them):
```
ingest → anexi_current_value --apply → backfill_unp --apply → reconcile_cross_source --apply
→ rebuild_from_cache → db:load:pg → db:load:annexes:pg
```
Both predecessors are mandatory and the pass enforces the second with a УНП-coverage preflight:
the identity it reconciles on needs the УНП, which **does not exist at parse time** (the OCDS
export carries none, and `backfill_unp` writes it onto the shards afterwards). Run out of order it
is a silent no-op.
**Cloud needs no separate command** — the pass rewrites shards, so `db:load:pg:cloud` carries it.
What the cloud DOES need is `db:load:annexes:pg:cloud` after it, per the note above.
The pass is idempotent and dry-run by default. It permanently refuses two shapes and prints both
in full — groups where a feed contributed more than one row (no 1:1 twin exists) and side-pairs
failing a supplier-set/completeness precondition. Those are expected output, not failures;
`single_source_per_contract.data.test.ts` allowlists exactly them, and
`scripts/procurement/measure_cross_source.ts` re-derives every figure read-only against either the
shards or a database. The identity and refusal rules live beside the shared implementation in
`scripts/procurement/cross_source.ts`.
**SAME-feed duplication is a separate class, and the pass above is blind to it by construction** —
every grouping it does requires `count(DISTINCT feed) > 1`. The harness measures it as **§6**, and
the one rule to remember is **always read that section split by `tag`**, because the two arms are
disjoint on it and only one is a defect (plan:
`docs/plans/procurement-same-feed-dedup-v1.md`):
- **`ocds` (238 groups / €591.1m) is 100% `contractAmendment` and is NOT duplication — never evict
it.** Those rows are distinct amendment events, verified 1:1 against
`procurement_annexes.notice_id` (an independent ЦАИС source); they render the `/contract/:key`
amendment timeline; and they already carry **zero € weight**, since `rollups.ts` excludes
amendments from every money rollup and every serving SUM filters `tag = 'contract'`. The trap is
specific: a tag-BLIND fold on identity E's fields pairs a `contract` row with its own
`contractAmendment` and deletes the base contract.
- **`aop` is 100% `contract` and IS real, but the printed €2.94m is a FLOOR** — identity E requires
a УНП that 42.2% of `aop-legacy-` rows lack. On full content identity it is €11.77m, of which
30 groups / €2,068,182.74 are provable stale-key orphans (rows minted before
`disambiguateContractKeys` shipped, which the key-merge in `writeMonthShards` never evicted).
That sweep is **open work, not done**; `dedup_contract_keys.ts` cannot find them because it
groups by the STORED key, so a stale-keyed row is a singleton group and is skipped.
### `count(*) FROM contracts` is NOT the shard count — and the delta is 2,680 synthetic rows
⚠️ **`data/procurement/index.json` and `count(*) FROM contracts` answer DIFFERENT questions,
and comparing them reads as a corpus defect.** Measured 2026-08-28: the shards hold **407,464**
rows (403,976 `contract` + 3,488 `contractAmendment`, matching `index.json`) while BOTH the
local and the Cloud SQL `contracts` tables hold **410,144**. The difference is exactly the
**2,680 synthetic `obed-` consortium carrier rows** minted by `rebuild_consortium()` (087),
which have no shard row by construction:
```
407,464 contracts WHERE contractor_eik NOT LIKE 'obed-%' == shard rows, exactly
2,680 contracts WHERE contractor_eik LIKE 'obed-%' == 1,633 distinct carrier EIKs
410,144 count(*) == local AND cloud, identical
```
Three things make this look like a local↔cloud asymmetry when it is not:
- ⚠️ **The loader's log line USED to print only the SHARD count** — `loaded 407464 contracts
→ Postgres`, i.e. `rows.length` from `readShards()` — so a `db:load:pg:cloud` that ended
with the table at 410,144 reported 407,464, and the two read as a 2,680-row leak when they
were the same successful load. Fixed 2026-08-28: it now reports both, as
`loaded 407464 shard rows → 410144 contracts (+2680 consortium carriers) in Postgres …`.
**An older run's output still carries the bare shard count**, so a log quoted from before
that date is the pre-fix form and is not evidence of a leak.
- ⚠️ **The parity guard also compares at 407,464, and that is correct.** `load_pg.ts` opens
ONE `withTx` (line 302) that runs the merge upsert, `CONTRACTS_MERGE_DELETE_SQL`, the
live-vs-stage parity check, and then `rebuild_consortium()` — all before COMMIT. So the
guard sees the pre-transform state by design, and **407,464 is never observable in a
committed table**. Any reading of it came from `index.json` or from the log, not from SQL.
- **The merge is NOT upsert-only.** `CONTRACTS_MERGE_DELETE_SQL` is a full unscoped anti-join
DELETE (`DELETE FROM contracts c WHERE NOT EXISTS (… contracts_stage s WHERE s.key = c.key)`),
so a row `reconcile_cross_source` evicts from the shards IS removed from the cloud table on
the next load. "Cloud needs no separate command — the pass rewrites shards" holds; evicted
rows do not accumulate on prod.
**No money is over-stated by the extra rows.** 087 MOVES a joint award's value onto the
carrier and zeroes the members, so the total is invariant — measured on both databases,
`sum(amount_eur) WHERE tag = 'contract'` = **93,915,805,242.82**, equal to
`index.json.totals.totalEur` to the cent, with members summing to **0.00** and carriers to
€6,226,327,402.79.
⚠️ **THREE money bases are in play here and none of them is wrong — check which one a figure
uses before calling two surfaces inconsistent.** `hub_stats.json`'s `totalEur`
(93,705,741,501) excludes the 628 empty-`contractor_eik` rows (€210,063,742.05, the documented
empty-string key); `index.json.totals.totalEur` (93,915,805,242.82) includes them at
`tag = 'contract'`; and the contracts browser's `aggregates.sumAmountEur`
(99,863,878,635) is the ALL-TAG sum, amendments included. The ROW COUNT, by contrast, does
agree everywhere: `hub_stats.json` publishes `contracts: 410144` and
`/api/db/table?resource=contracts` returns `total: 410144`, so the hub tile and the table
beneath it are counting the same corpus.
⚠️ **`index.json`'s two cross-reference summaries are a FOURTH and FIFTH basis, and they
differ from each other — the MP arm is PER-PAIR and the officials arm is PER-COMPANY.**
`crossReference` sums euro across every (MP, contractor) pair, so a contractor tied to two
MPs counts twice: measured 2026-09-02, **€2,629,744,300 over 114 pairs / 106 EIKs, against
€2,267,057,426 de-duplicated — a €362.7M (16%) overstatement if read as "public money won
by MP-linked companies"**. `officialsCrossReference` de-duplicates by contractor EIK:
**€6,557,542,819 over 872 pairs / 452 EIKs, against €33,309,147,605 per-pair — a 5x
difference.** Both are deliberate — the per-pair sum is the basis its own `pairCount`
counts, and `scripts/db/gen_procurement/index.ts` is a PARITY verifier that must keep
reproducing `ingest.ts` rather than "correcting" one side — so the fields NAME their bases:
`totalPerPairEur` / `totalOtherPerPair` and `totalPerCompanyEur`. There is no bare
`totalEur` on either, and reintroducing one puts the trap back.
Two things about that naming are easy to get backwards. **Keep the `Eur` SUFFIX on any
future rename**: `stabilizeNumber` (`scripts/procurement/validate.ts`) matches money keys
with `key.endsWith("Eur")` and quantizes them to cents, which is what keeps this file
byte-stable across rebuilds — the obvious `totalEurPerPair` drops out of that rounding and
jitters the last ULP into the committed diff, on a file whose diff is how a corpus change is
reviewed. And **SIX writers emit these fields and must stay in lockstep** — `ingest.ts`,
`rebuild_from_cache.ts`, `rebuild_derived.ts`, `dedup_legacy_twins.ts`,
`dedup_contract_keys.ts` and the parity generator — the last of which reports DIFF, not a
type error, when it drifts. Nothing in `src/`, `ai/` or `functions/` reads either field
today (`hub_stats.json`'s `connected` is a COUNT, and the AI `procurementTotals` tool reads
Postgres), so the double-count is LATENT; the naming is what keeps the first consumer from
making it live.
The gate is `pg_roundtrip.data.test.ts`, whose assertion is
`byKey.size - syntheticCount === onDisk` ("row count: Postgres (minus synthetic consortium
carriers) vs month shards"). It passes. A future audit comparing the two numbers should
subtract the carriers first, or just run that gate.
### Sector EIK rosters — `edu` is the widest, and the trap it documents
A sector's awarder allowlist is declared in up to four places (reference data →
`SECTOR_EIKS` in `gen_procurement/sector_stats.ts` → `SECTOR_DASHBOARDS[id].members` →
`SECTOR_BROWSE_PACKS[id].eiks`), and the reference data is **meant** to be the single source
of truth. ⚠ It is not universally so: `SECTOR_DASHBOARDS.energy.members` restates nine EIK
literals instead of mapping `ENERGY_MEMBER_EIKS`, so a roster edit there does not propagate.
Check before assuming a copy is derived.
`src/lib/educationReferenceData.ts` is the largest: **126 EIKs across five universes** (МОН,
its agencies and state companies, 33 state higher schools, БАН, Селскостопанска академия),
~€2.11bn. Before the 2026-08-18 audit the sector rolled up ONE EIK, so `/sector/edu` showed
€506M against that, and **€3.17M over 9 contracts on its default scope**.
Three things it records that generalise to any sector roster:
- **A sibling's `EXCLUDED_EIKS` is an ANTI-allowlist, and grepping an EIK finds it either
way.** Six institutions were excluded "to culture" while `kulturaReferenceData.ts` was
DISCLAIMING them (`principal: "ban_mon"`) — stranding НХА, НМА, НАТФИЗ and three БАН
museum-institutes in NO sector, the very defect the roster exists to fix. Verify ownership
against `SECTOR_DASHBOARDS` members / `SECTOR_BROWSE_PACKS` eiks, never against a string's
presence in a sibling file. Both halves are gated in `educationReferenceData.test.ts`.
- **Institutions ≠ EIKs.** Стопанска академия „Д. А. Ценов" changed EIK in 2016 and both
halves of its history are kept (`retiredEikOf`), so 126 EIKs are 125 institutions — and the
awarders footnote said „34 държавни висши училища" for 33 until that field existed.
Reader-facing prose quotes the INSTITUTION count.
- **`packIsThematic` — a registered sector pack normally IS the page and disables the group
model** (`useAwarderGroupModel(..., enabled: !Pack)`). MonPack is a cross-buyer analysis of
the textbook market and does not bind its `eik` prop, so widening the roster moved every
headline number and NONE of it reached the page while every other gate stayed green.
The hub headline stays `basis: 'budget'` on МОН's own node and deliberately does not sum the
group's procurement onto it — the roster spans three budget principals (МОН, БАН, ССА→МЗХ)
while the node spans one. The awarders footnote states that, derived from the roster so its
counts cannot drift.
### Supplier identity — the three synthetic key namespaces, and why a re-ingest cannot fix one
`contracts.contractor_eik` is **not always a Bulgarian EIK**. Four other kinds of token live
in that column. Measured 2026-08-19, **both columns at `tag = 'contract'`** — the € differ by
6.9% across that filter (the plain bucket is €92.24bn over all tags), and a table mixing the
two bases is the `tag`-blindness trap the same-feed-dedup note above is about:
| bucket | keys | € | linkable |
| ---------------- | ---------- | -------- | -------- |
| plain 9/13-digit | **27,531** | €86.30bn | ✅ yes |
| `obed-` carriers | **1,626** | €6.21bn | ✅ yes |
| `ph-` / `np-` | **177** | €87.3m | ❌ no |
| neither | **281** | €857.5m | ❌ no |
| the EMPTY string | **1** | €210.0m | ❌ no |
Unscoped, the first row is 27,553 keys and the fourth 282 — the synthetic rows are identical
either way, which is exactly why a scope slip here hides. The empty string is a key in the SQL
sense and in no other (623 rows); it is listed because omitting it under-reported the
non-linkable money by a fifth. `scripts/procurement/supplier_identity.ts` mints the synthetic
ones, each because the source token could not become a key: `obed-` a consortium carrier
keyed by its member set, `ph-` (91) a **filler** registration number keyed by the supplier's
NAME, `np-` (86) a natural person keyed by name so no ЕГН is stored.
⚠ **„NONE OF THEM HAS A PAGE" WAS TRUE ONCE AND HAS NOT BEEN SINCE 2026-07-06**, and this
file asserted it until 2026-08-19. `institution_identity()` really does return NULL for all
2,085 — but `8c8b9a9654` („render procurement body for corpus-only entities") gave
`/company/:eik` a procurement-only fallback body, and its dead-end branch now fires only on
`!company && !institution && !hasProcurement`. A key drawn out of `contracts.contractor_eik`
HAS contracts by construction, so it can never reach that branch. Re-check with that one grep
(`src/screens/dev/CompanyDbScreen.tsx` — yes, `dev/`; it serves `/company`) before tightening
anything here.
So `isLinkableCompanyKey` (`src/lib/companyKey.ts`) no longer asks „can the page serve this"
— it serves all of them — but **„is this key worth promising a page for"**, and the
namespaces answer differently. **`obed-` is IN**: its page is the richest of the four kinds,
carrying a „Обединение — участници" block naming each member firm that no other key kind has
— so de-linking it left a reader no route from a consortium's € to the firms behind it.
**`ph-`, `np-` and the odd ids are OUT**: their pages render too, but a made-up registration
number and a natural person are not identifiers anyone can check against a register. A link
promises somewhere to go.
`CompanyLink` is how a CONTRACTOR key should become a link.
⚠ It is NOT centrally enforced — `companyKey.test.ts` is a best-effort net over known
surfaces and field names, and it has already missed two (`p.eik` on
ProcurementSectorsScreen, `a.topEik` on the watchlist) that were rendering live dead links.
`contractor_rank` alone holds 11,813 synthetic rows, so treat any new surface reading it as a
candidate and check by hand.
⚠ It is contractor-only on purpose: awarder ids are validated by `isValidEik` (9–13 digits)
and two live awarders sit outside 9/13 — ЕСО `1752013040`, АДФИ `175076479999` — both of
which resolve, so routing an awarder through it de-links a working page.
**A separate question about the same rows: is this supplier SEVERAL FIRMS?** That is
`isConsortiumSupplier` (same file), and it is deliberately not the complement of linkability
— a carrier is now linkable, and `ph-`/`np-` are neither linkable nor consortia. It prefers
`AwarderSupplier.consortiumEur` (061) and falls back to the `obed-` prefix, because the €
is the only thing that sees a REGISTERED ДЗЗД: those carry an ordinary 9-digit EIK and are
**1,344 of 4,014 carrier rows, €5.63bn — 47.5% of all consortium money**. ⚠ `consortiumEur`
is a € and never a flag: 162 suppliers hold BOTH carrier and solo rows (€1.52bn vs €0.99bn),
so no boolean is true for them. `0` means „won nothing jointly"; `null` means „this producer
could not tell" and must not render as „none".
**Filler** is what a buyer types when ЦАИС demands a registration number the supplier has
none of. `isValidEik` used to reject only the all-zero form, so `000000001` passed as a
perfectly good EIK and became a company key **pooling nine unrelated suppliers** — Elsevier's
€32.8M and Clarivate's €11.2M rendered as ONE contractor on every leaderboard — while
`1234567899` pooled **22 distinct natural persons**. `isPlaceholderId`
(`scripts/procurement/eik.ts`) recognises it now.
⚠ **The rules are all-same-digit, an ascending run, and a three-entry DENYLIST — and the
obvious fifth rule is wrong.** "Small numbers are filler" would re-key real bodies:
`000000210` is ДГС Гърмен, a live awarder, and 29 Commerce-Registry cooperatives sit below
10000 starting at `000000491`. This corpus has already defeated four discriminators (listed
in `supplier_identity.ts`'s header); do not add a fifth without measuring it against
`tr_companies` AND every `awarder_eik`.
⚠⚠ **NEITHER RE-INGEST MODE CAN APPLY THIS FIX TO ROWS ALREADY ON DISK, and one of them
silently doubles the corpus's exposure.** Measured over the full 2020→2026 ЦАИС history:
- `ingest_eop --backfill` (gap-fill) keeps only buyers ENTIRELY ABSENT from the corpus. All
4,416 are present, so it kept **0 rows** and dropped 207,029.
- `ingest_eop --backfill --self-heal` dedups on `contentKeys()`, and **all four of its nets
embed `contractorEik`**. A re-parsed filler row therefore shares NO key with its own
on-disk twin — verified on the live Кларивейт row, 4 keys before, 4 after, 0 in common —
so it is kept as NEW while the old row stays. Applying it **double-counts** the rows
instead of correcting them.
The only path is the one-off `scripts/procurement/rekey_placeholder_suppliers.ts` (dry-run by
default, `--apply` to write, idempotent), then the usual chain — `proc:backfill-unp:apply` →
`proc:reconcile:apply` → `proc:rebuild-derived` → `db:load:pg` → `db:load:annexes:pg`. The
annexes step is **mandatory**: a re-keyed contract changes `key`, which orphans its
`procurement_annexes` rows until that loader re-resolves them.
⚠ **A re-key MOVES `/contract/:key` URLs, and that is inherent, not a choice** — the key is
`hash(releaseId::contractId::contractorEik::tag)` for the release-shaped feeds and
`hash(legacy::datasetUuid::documentId::contractorEik)` for the annual CSV, so the supplier IS
part of contract identity and a fresh parse moves them identically. On the legacy feed
`releaseId` embeds the supplier too, so that moves with it. All three builders live once, in
`contract_key.ts` (`releaseContractKey` / `legacyContractKey` / `legacyReleaseId`), so the
one-off and the ingest cannot drift; they were three private copies until 2026-08-19.
Cloud side, and nothing runs it automatically: `db:load:pg:cloud` then
`db:load:annexes:pg:cloud`, then **`db:load:graph:pg:cloud`** — `company_public_money` (127)
keys on `contractor_eik` and so holds rows for the filler ids themselves, and `load_pg` does
NOT refresh it; skipping it leaves `000000001` sitting on €45.7M on prod. Then
`db:load:tr-company-place:pg:cloud`, which denormalizes `money_eur` from it. `persons-browse`
/ `person-search` are NOT needed — `person_role` and `tr_companies` both hold zero filler
rows, so no person's money can include a re-keyed contract.
`scripts/db/tests/supplier_filler_ids.data.test.ts` is the gate.
### The prices schema — the migration TWO scripts apply from outside `scripts/db/`
`048_prices.sql` had **no applier at all** until 2026-08-20: the price tables reached every
existing database by hand, so a table added to the file existed only where somebody had
remembered to run `apply_functions.ts`. `scripts/prices/ingest.ts` applies it once per
run, which is what makes a schema change reach the serving database at all.
⚠️ **`scripts/prices/build_payloads.ts` is a SECOND applier, and it is not redundant.** It
READS a table 048 owns — `price_last_seen`, the retained-price fallback behind every chain
page — so with the ingest as sole applier, publishing payloads to a database the daily
ingest had not yet run against failed on a relation nobody knew was missing. Measured on
Cloud SQL 2026-08-20: `price_grid_days` was current and at the SAME day as local, and the
rebuild still could not run.
The ingest is not a substitute for it. It fetches and loads a DAY, and the moment these
pages matter most — a chain has gone silent, so the daily load is being held back — running
it merely to obtain a `CREATE TABLE` is not an option. Both appliers use `execEach` for the
reason below; the seed is `NOT EXISTS`-guarded, so the second is a no-op on a warm database.
**Publishing payloads to Cloud SQL is `npm run prices:payloads:cloud`**, and it needs no
ingest: it applies 048, seeds `price_last_seen` and rebuilds every blob from the corpus
already there. Measured 2026-08-20 against a corpus at 2026-08-14 — the seed wrote
**4,387,949 rows, exactly matching local**, which is the check worth repeating, since the
two databases deriving the identical open-run set is what proves they are the same vintage.
⚠️ **Ship the BUNDLE before the payloads whenever the chain pages widen.** A payload rebuild
brings back every chain retained by the `price_last_seen` fallback (measured: 98 → 215 blobs,
114 chains carrying prices 1–30 days old), and staleness is labelled in `ChainProfileScreen`.
An older bundle has no such notion and renders whatever it is sent. `build_payloads` therefore
withholds `marketMin` — today's cross-chain minimum — from any row not observed on the latest
day, because beside an older price it is not a comparison but a claim that a days-old price is
currently the cheapest on the market. That server-side NULL is what makes a payload-first
publish safe at all: verified against the deployed pre-T2c bundle, a null takes BOTH its
`cheapest` badge and its struck-through minimum to false. What it cannot supply is the notice,
so payload-first still leaves those pages undated until hosting ships.
Three things about it differ from the `scripts/db/load_*.ts` family:
- **It uses `execEach`, not `exec`, and that is load-bearing.** `exec` sends the file as ONE
transaction, so the AccessExclusiveLock taken by its no-op `ALTER TABLE … ADD COLUMN IF
NOT EXISTS` is held until the last statement — which includes a multi-million-row seed
INSERT — on tables `/api/db/price-history` and `/api/db/price-product` read. The same
TRUNCATE-shaped lock is what 500'd those readers on every ingest before the 2026-07 merge
rewrite; do not re-introduce it through the DDL path.
- **`price_last_seen` is seeded from OPEN RUNS, never from `price_current`.** The narrow
seed looks obvious and is precisely backwards: `price_current` holds only the chains that
reported on the latest day — measured 2026-08-20, **98 of 215** — so it omits the 170 that
had already gone silent, which is the entire population the table exists for, and nothing
would ever add them (only the daily `obs` write does, and they never appear in `obs`
again). The wide seed covers **215 of 215** at ~4.4M rows.
- **Its VACUUM is invisible to `reload_visibility_map.data.test.ts`'s scan.** That gate globs
`scripts/db/load_*.ts` and their escaping imports; the prices ingest has no such wrapper,
so `price_last_seen` / `price_current` carry RELOADED entries with no discoverable call
site. The entries still switch the per-table map assertion on, which is the half that
matters.
⚠️ **`price_last_seen` is upsert-only with ONE exception**: a re-published day that
withdraws a `(store, sku)` deletes that row, so the table matches the corrected feed rather
than asserting a price the source now says was never filed. Any future non-shrink gate must
exempt that case. Plan: `docs/plans/prices-chain-absence-v1.md`.
⚠️ **`price_products.title_fold` is a DEPLOY-ORDER dependency of `deploy:db`, and getting it
backwards is a 500 on every products search.** 048 declares it
`GENERATED ALWAYS AS (translit_bg_latin(title)) STORED`, and `db_table.js` gives the
`price_products.title` column `searchCol: "title_fold"` + `searchFold: true` — so the engine
emits the column name unconditionally. A missing COLUMN is **42703**, which no degrade helper
in `db_routes.js` handles (they cover 42883 / 42P01), and `badRequest()` rethrows anything that
is not a `DbRequestError`. Ship the DDL FIRST, then the function:
```bash
psql "$DATABASE_URL" -c "ALTER TABLE price_products ADD COLUMN IF NOT EXISTS title_fold text GENERATED ALWAYS AS (translit_bg_latin(title)) STORED;" -c "CREATE INDEX IF NOT EXISTS price_products_title_fold_trgm ON price_products USING gin (title_fold gin_trgm_ops);" -c "VACUUM (ANALYZE, PARALLEL 0) price_products;"
npm run deploy:db
```
⚠️⚠️ **THE `VACUUM` IS THE STEP THAT LOOKS OPTIONAL AND IS NOT — `ANALYZE` ALONE IS THE
DISGUISE.** A STORED generated column REWRITES the heap into a new relfilenode, whose
visibility map is EMPTY — the same mechanism the „reload visibility map" section describes for
`VACUUM FULL` and for a TRUNCATE-reload. Measured 2026-08-31, immediately after the ALTER **and
after an `ANALYZE`**, on BOTH databases: `relpages 4352 · relallvisible 0 · 0.0%`. So
`price_products_browse` — the index this file already has a paragraph about, because it is what
makes the `/consumption/products` arrival 27 buffers instead of 19,261 — stops being planable
as an index-only scan, silently, with `last_analyze` freshly stamped. One `VACUUM (ANALYZE,
PARALLEL 0)` restores it (measured: 0.4 s local, 1.5 s cloud, back to 98.8% / 98.7%).
`PARALLEL 0` stays for the reason the visibility-map section gives.
**Applied to Cloud SQL 2026-08-31** at 124,120 rows: ALTER **12.2 s**, index **2.7 s**, VACUUM
**1.5 s** — 16.4 s in total, off-peak. Verified afterwards that `app_readonly` reads the new
column (the grant is TABLE-level — `app_readonly=r/postgres` — so a column added later is
covered and needs no new GRANT; `has_column_privilege` confirms it), and that the two databases
return identical counts for every probe term, i.e. they are the same corpus vintage.
Four things about it are easy to get backwards:
- **Do NOT apply 048 through `apply_functions.ts`.** That uses `exec` (one transaction), which
is exactly what this file's own header forbids: the ALTER's AccessExclusiveLock would be held
until the last statement, and the last statements include the multi-million-row
`price_last_seen` seed, on tables `/api/db/price-history` and `/api/db/price-product` read.
The two statements above are what `execEach` would run; `npm run prices:payloads:cloud` and
the daily ingest also carry them, correctly, one statement at a time.
- **The ALTER REWRITES the table** — every STORED generated column does — so it takes an
AccessExclusiveLock for the duration. Measured locally at 124,120 rows: **~10 s including the
index build and ANALYZE.** Off-peak, and expect longer on Cloud SQL.
- **048 now depends on `000_search_fns.sql`**, because a GENERATED expression is resolved at
ALTER time. Both of its appliers (`scripts/prices/ingest.ts`, `scripts/prices/build_payloads.ts`)
apply 000 first for that reason; under `execEach` a missing `translit_bg_latin` would raise
42883 on that one statement and leave the catalogue without its fold while every other
statement in the file succeeded.
**What it buys, measured 2026-08-31 over the 46,682 browsable products:** the search box was a
RAW Cyrillic substring match, so „kafe" returned **0** rows against 1,389 titles containing
КАФЕ — and „mlyako" 0 against 2,366, „sirene" 0 against 1,499, „banani" 0 against 85. There is
no error and no empty-state distinction, so „no such product" and „this box cannot see
Cyrillic" render identically at a 200. The keyboard arm (`shlyo_query_fold`, 141) comes with
it: „6okolad" 0 → **968**, „4erven" 0 → **593**.
⚠️ **`price_products_trgm` (gin over the RAW title) STAYS — it is a different consumer.**
`/api/db/price-search` ORs Latin→**Cyrillic** candidates (`shlyoCandidates` in `db_routes.js`)
against `title`, because that route predates the fold; the registry engine compares Latin
against Latin. So the two search paths differ in coverage, and the dropdown is WIDER on one
class: the phonetic i-glide spellings („mliako" → мляко, „iaica" → яйца) that
`shlyo_query_fold` leaves alone. Unifying them is open work — the shape that would keep every
arm on ONE index is to fold each Cyrillic candidate back through `translit_bg_latin` rather
than OR-ing the raw column in, which the engine's own „OR across DIFFERENT indexes is slow"
measurement (292 → 722 buffers) rules out. Gate:
`scripts/db/tests/prices_search_fold.data.test.ts`.
### The two committed artifacts `db:refresh` regenerates
`data/procurement/derived/hub_stats.json` (the nine `/procurement` hub stat-tile numbers) and
`sector_stats.json` (the `/governance/sectors` headline per sector) are **committed and
bucket-synced**, but derived from Postgres — so unlike the rest of the PG-served procurement
tree they go stale in the repo whenever the corpus reloads. From 2026-06 until 2026-08-04 nothing
regenerated them at all; a contracts/tenders/agri/ngo reload moved the corpus underneath two files
that kept serving the old numbers at a 200. They are now in `db:refresh`:
```
… → db:load:ngo-funding:pg → db:gen-hub-stats → db:gen-sector-stats → db:load:judiciary-payloads:pg → …
```
**That slot is load-bearing, not cosmetic.** Five of `hub_stats`' nine fields come from tables
loaded across the whole chain (`tenders`/`kzk_appeals` at the tenders step, `awarder_seats` after
agri, `ngo_funding` last), and `sector_stats`' ДФЗ payout reads `agri_payloads`. Moving them
earlier — next to `db:load:annexes:pg`, where they visually belong — regenerates those fields from
the PREVIOUS vintage and commits it, which is the drift the wiring exists to end. After
`db:load:ngo-funding:pg` is the earliest safe position. `refresh_coverage.test.ts` holds chain
membership, and — since 2026-08-05 — a declared subset of the ORDER via its `ORDER_PAIRS` table
(each entry is "this loader must follow the step that rebuilds its input"). `hub_stats` /
`sector_stats` are NOT in that table yet, so for those two still check the dependency list in each
file's header before moving either; add a pair there when you do.
That table exists because the gap shipped a defect: `db:load:tr-company-place:pg` sat at step 25,
twenty steps ahead of the `db:load:graph:pg` that applies and rebuilds `company_public_money` (127)
— the money basis it denormalizes — so every contracts reload published the previous vintage to the
governance "фирми, регистрирани тук" tile with every row count reconciling. It now runs after
`graph`. Membership alone could never have caught that.
Three things about them differ from every `db:load:*` in the chain:
- **`hub_stats.ts` is the only applier of `062_procurement_hub_counts.sql`.** No `db:load:*` ships
it, and nothing did before 2026-08-04, so `procurement_hub_counts()` existed only on databases
where it had been applied by hand. Its `GRANT` is role-guarded (117/130 shape) because
`roles_readonly.sql` may not have run on the target — unguarded, it raises 42704 on a cold
bootstrap and rolls the whole file back, leaving no function at all. (`npm run db:pg:bootstrap`
now runs that file automatically for the LOCAL docker Postgres, as step 2 of `db:refresh`; it
refuses every other target, so on Cloud SQL it remains the one-time manual step it always was
and the guard stays load-bearing there.)
- **Both skip-and-warn rather than degrade.** A missing relation, function, empty `contracts`, or
absent `data/budget/ministries/` (gitignored — `sector_stats` reads eight ПРБ nodes from it)
logs and exits **0** without writing. Returning before the write is the point: a partial artifact
would overwrite a good served file with a worse one and reconcile against nothing.
- **`company_politicians` / `tr_companies` can be legitimately ABSENT**, not merely stale — their
only loader is `db:load:tr:pg`, a `REFRESH_EXCLUSIONS` member. `procurement_risk_feed` reads
both, which is why `hub_stats` probes relations instead of assuming the chain implies them.
**There is no `:cloud` half.** These are committed FILES, not tables — they ship via
`bucket:sync` (`scripts/bucket_sync_paths.ts`), so a cloud reload does not touch them and a local
`db:refresh` is what makes them current.
The other seven `gen_procurement/` entries (`db:gen-rollups`, `-lists`, `-shards`, `-derived`,
`-xref`, `-index`, `-byns`) are sql-migration-v1 **parity verifiers**: they re-derive the JSON
pipeline from Postgres and write nothing unless `--write` is passed. They are correctly outside
`db:refresh`. That `process.argv.includes("--write")` idiom is what `refresh_coverage.test.ts`
uses to tell the two kinds apart, so a NEW generator dropped into `gen_procurement/` must either
join `REFRESH_GENERATORS` (and the chain) or carry the gate — it cannot quietly land outside.
The data pipeline CLI (`scripts/main.ts`) accepts flags: `--all`, `--prod`, `--date`, `--election`, `--reports`, `--stats`, `--search`, `--financing`, `--parties`, `--machines`, `--candidates`.
### A LOCAL `db:resolve:persons` is never one command — the repair chain it owes
**`person_id` is a POSITIONAL ordinal handed out by a `DELETE FROM person` + re-COPY, so
every resolve reassigns a large slice of them.** Measured on the 2026-08-20 local resolve by
snapshotting all 133,723 `(person_id, slug)` pairs either side of it: **35,211 (26.3%) came
back pointing at a DIFFERENT person, and 0 dangled.** That zero is the whole hazard — a stale
stored `person_id` still joins cleanly, it just joins to the wrong human, so nothing errors
and every row count reconciles.
The drift has a **hard boundary at person_id 98,512** and it is not luck: ids 1..98,512 (the
65,065 `resolved` base persons plus verified officials) were 100% STABLE, and 98,513+ were
100% DRIFTED — that tail is the tier-V private owners, minted from the TR corpus, so it moves
whenever `tr_person_roles` does. Consequence: `candidate_person` was 0% misattributed while
`graph_person_node` was 42.2%, purely because candidates are low-ordinal base persons.
**Two dependents announce themselves and nine do not.** `declaration` and `council_vote` are
the only FKs to `person`, both `ON DELETE SET NULL`, so a resolve blanks them table-wide —
measured, `declaration.person_id` went 61,743 → **0** and `council_vote.person_id` 43,168 →
**0**. Loud, and recoverable. The silent set is every relation storing `person_id` with no FK:
`declaration_stake_company`, `person_cohort_wealth`, `person_wealth_year`, `candidate_person`,
`person_election_stats`, `graph_edge`, `graph_person_node`. Those keep a stale integer.
⚠️ **"Nothing else is needed locally" is the wrong instruction and it is the one people are
given.** Stopping at the resolver leaves 43,168 council votes and 61,743 filings unattributed
at a 200. The local repair chain is `db:refresh` steps 53–65; the necessary subset, in order:
```bash
npm run db:load:declarations:pg -- --resolve # refills declaration.person_id; rebuilds 090/096/097/159/169
npm run db:load:person-elections:pg # candidate_person + person_election_stats
npm run db:load:official-candidate-links:pg
npm run db:load:council:pg # re-attaches council_vote.person_id (90% floor)
npm run db:load:persons-browse:pg
npm run db:load:person-search:pg
npm run db:load:graph:pg
npm run db:load:agri-hub-stats:pg
npm run db:load:tr-company-place:pg
```
Steps 55–57 (`mp-roster`, `rollcall`, `rollcall-derived`) are deliberately NOT in that list —
none of them stores `person_id`, so a resolve does not invalidate them.
Three things about this are easy to get backwards:
- ⚠️ **`ORDER_PAIRS` STRUCTURALLY CANNOT express the declarations rule, so do not go looking
for the entry.** Its matcher is `steps.indexOf(name)` over `/npm run ([a-z0-9:_-]+)/`, and
BOTH declaration phases render as the same token `db:load:declarations:pg` — so `indexOf`
always resolves to phase 1 at step 49. The existing pair therefore pins phase 1 BEFORE the
resolver and there is no way to write "phase 2 after" without renaming a script. Phase 2 is
held by this section and by the loader's own header, nowhere else.
- **A redundant slug beside the ordinal is what makes drift DETECTABLE.** `graph_person_node`
and `candidate_person` store `person_id` AND a slug, so misattribution is a one-line join
(`p.slug IS DISTINCT FROM g.slug`) — that is how the 42.2% above was measured.
`person_election_stats` stores the bare ordinal, so drift there is invisible from inside the
row. Cheapest mitigation for a NEW consumer, in order of preference: derive the person at
query time from a fold (`aop_expert_person_links()` in 174 does this and is immune by
construction); failing that, carry the slug as a tripwire.
- **`db:resolve:persons` ANALYZEs but never VACUUMs**, so `person` / `person_role` rely on
autovacuum reaching them. It did on the measured run (100% `relallvisible` within a minute),
but that is timing, not a guarantee — see the visibility-map section for why a bare ANALYZE
is the disguise rather than half the fix.
### Person layer — the one step `db:refresh` cannot infer
Deploying the person layer to **Cloud SQL** needs one extra command after
`db:resolve:persons:cloud`:
```bash
npm run person:slug-redirects:cloud -- raw_data/person/officials_reslug_2026_07_24.json
npm run person:slug-redirects:cloud -- raw_data/person/officials_reslug_2026_07_29.json
```
They load the officials re-slug maps into `person_slug_retired`, so the ~20.8k `/person` URLs
minted under pre-2026-07-24 officials slugs 301 instead of 404. The loader upserts, so each
dated map composes; add a line here whenever a new one lands (2026-07-29 is the collision-fold
drop — see `raw_data/person/README.md`). `db:refresh` runs the local equivalent automatically;
nothing runs it on the cloud side. See `raw_data/person/README.md` and
`docs/plans/persons-pg-retirement-v1.md` (T1.0).
**"Composes" means rows accumulate — NOT that older targets stay valid.** When a later map
retires a slug an earlier map already pointed somebody at, that earlier row becomes a 301 into
a 404 (`…ivanov1-da0219 → …ivanov1-94805e → …ivanov-b85a89` is the one that got caught). Both
writers — the loader and `db:resolve:persons` — now call `collapseSlugRedirectChains()` after
their write, so flattening is automatic and `db:resolve:persons` carries "redirect chains are
flat" as a post-condition. That matters for the cloud flows above that re-run the resolver
**without** a following `person:slug-redirects:cloud` (the judicial-bodies rerun): they stay
correct only because the resolver does it itself. The collapse only ever re-points at an
`active` + public-figure person, so a chain with no servable end is left broken and reported
rather than quietly made to look healthy — `person_slug_retired.data.test.ts` fails on it.
**The resolver's SECOND gold key is recoverable only from a filename shape**, and a
re-resolve that moves the key count by tens is this rather than a data change. The Сметна
палата stamps most filings `<PERSON-GUID><filing-seq>.xml`, and that GUID is what stitches an
MP to their ministerial declarations and one official to their several postings — but in the
2019-2023 folders it also emitted a BARE guid with **no** sequence suffix, which is
per-DOCUMENT. Read as an identity it makes one declarant look like one stranger per extra
filing, and since `registerIdByRef()` guards with `HAVING count(DISTINCT guid) = 1` the cost
is not a wrong merge but **no key at all, with nothing logged**. Measured 2026-08-11: 70 refs
were being skipped as "two register persons" and 2 actually were. The rule lives once, in
`PERSON_GUID_SQL_PATTERN` (`scripts/officials/slug_identity.ts`), and
`person_register_guid.data.test.ts` runs the SQL and JS forms over the whole corpus. Do not
restate the pattern anywhere — the officials ingest had already learned this (66 document ids
once sat in `_slug_collisions.json`, splitting real people into orphan profiles) and the
resolver's own copy had not.
**A re-slug also invalidates `declaration.subject_ref`, and `--resolve` alone CANNOT repair
it.** For the `exec` and `muni` tiers `subject_ref` IS the officials slug, read out of the
per-person shard JSON by declarations **phase 1** (the `mp` tier keys on `mpId` and
`magistrate` on `declarantName`, so neither is affected). Phase 2 (`--resolve`) only joins
`person_role.ref = subject_ref` and fills `person_id` — it never rewrites the ref. So on the
cloud side a roster re-slug needs BOTH phases, phase 1 first:
```bash
npm run db:load:declarations:pg:cloud # phase 1 — rewrites subject_ref
npm run db:load:declarations:pg:cloud -- --resolve
```
**Phase 1 must also precede `db:resolve:persons` for a second reason since 2026-08**: the
resolver dates every officials posting from that tier's встъпителна / при напускане filings
(`date_basis = 'filing'`, 4,625 roles / 4,417 people), joining `declaration.subject_ref` to
the officials slug — and phase 1 is what writes `subject_ref`. Run out of order those roles
publish undated, and since the profile renders nothing without a basis the office periods
just vanish with nothing failing. `refresh_coverage.test.ts`'s `ORDER_PAIRS` holds the local
chain; on the cloud side the order is `db:load:declarations:pg:cloud` →
`db:resolve:persons:cloud` → `db:load:declarations:pg:cloud -- --resolve`.
⚠️ **`db:load:declarations:pg` PHASE 1 USED TO DESTROY BOTH COLUMNS, and the documented
cloud procedure ran it.** They are not in the shards — the shard writers never persisted
them — so they live only in Postgres, and phase 1 is a `TRUNCATE declaration … CASCADE` +
COPY of columns that do not include them. Every value came back NULL, the load reported
success, every row count reconciled, and the only symptom was `declared_label()` quietly
falling back to the register's LISTING label — a GROUP bucket that describes nobody. The
recovery costs another ~5-hour crawl or a `ship_filed_position.ts` run from a database that
still has them. Since 2026-08-18 the loader snapshots them by `source_url` inside its own
transaction and writes them back after the COPY, reporting `carried … for N/N filing(s)`;
a filing the register has withdrawn is named rather than silently dropped. Measured on the
reload that found this: 61,740 values lost locally, recovered from Cloud SQL — which still
had them only because nobody had yet followed the „roster re-slug needs BOTH phases, phase
1 first" instruction above against it.
**`declaration.filed_position` / `filed_institution` are SHIPPED, never re-crawled.** They
hold the declarant's own job and institution from each filing's `<Personal><Work>` /
`<Personal><Position>` — distinct from `institution` / `position_title`, which come from the
register's LISTING page and are GROUP labels: `position_title = 'Служебен министър-председател
и министър'` covers two people and describes neither (both were DEPUTY PM plus a minister).
Rendering a listing label as a person's job publishes a false claim about a named individual,
which reached a card on 2026-08-16. 089's column comments carry the rule.
The local corpus was filled by `scripts/declarations/backfill_filed_position.ts` — a ~5-hour
crawl of a rate-limited public register (54,071 fetches; 61,740 of 61,743 filled, the three
exceptions being filings whose `<Position>` the register itself leaves empty). **Do not point
that script at Cloud SQL.** A filing is immutable once published, so the derived values are
identical whichever database computes them; re-crawling spends five hours recomputing bytes
we already hold. Ship them instead:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts 089_declarations.sql
npx tsx scripts/db/ship_filed_position.ts --to postgres://postgres@127.0.0.1:5434/electionsbg --apply
```
The shipper keys on **`source_url`**, not `declaration_id`: the id is a `bigserial` handed out
in insertion order by the loader, so it is a property of how a database was loaded rather than
of the filing. The two happen to agree today (verified 2026-08-17, the md5 of the whole ordered
mapping matches), but a partial or re-ordered load would silently write every value onto the
wrong filing. It refuses below a 95% match rate — a low rate means the two corpora are not the
same vintage — and updates only rows whose values actually differ, so re-running it is free.
### `is_declared_holding()` — the ONE definition of what counts as the declarant's own
**Two of the form's tables are not holdings, and until 2026-08-18 every wealth figure on
the site counted them.** Tables **1.2** („Чуждо недвижимо имущество") and **3.4** („Чужди
моторни сухопътни, водни и въздухоплавателни превозни средства") record property and
vehicles owned by SOMEBODY ELSE that the declarant rents or is provided with. The
register's own column headers are what settle it — and they differ from tables 1/3 in
three places, not one:
| | table 1 / 3 (own) | table 1.2 / 3.4 (чуждо) |
| ------------------- | ------------------------------------- | ------------------------------------- |
| the money column | „Цена на **придобиване**" | „Цена **по договор**" |
| the basis column | „Правно основание за **придобиване**" | „Правно основание за **ползване**" |
| the year (3 vs 3.4) | „Година на придобиване" | „Година на **сключване на договора**" |
So the number is not a mis-attributed asset value — it is **what the use costs**. Пеевски's
2025 annual files tables 1 and 3 as `Declared="False"`, declares eight rented houses and
five provided cars, and was published at **€10,070,563** against a real **€9,760,147**.
`is_declared_holding(table_num)` (089, beside `declared_label`) is the rule, with a TS twin
`isDeclaredHolding()` in `src/lib/declarations.ts` because a route cannot import TS.
`declaration_asset.table_num` stores the **canonical 2018-form** number — same convention as
`declaration_stake.table_num`, since the printed number is version-dependent (the pre-2018
form's „4" is boats, the current one's is cash).
Four things about it are easy to get backwards:
- **NULL RETURNS TRUE, and the function is deliberately NOT STRICT.** The provenance exists
only in the source XML, so every row on a database that has not re-parsed is NULL —
reading that as "not a holding" deletes every real estate from every published figure at
once. STRICT would return NULL, which `WHERE is_declared_holding(…)` filters out, i.e.
exactly that. **The whole change is therefore INERT until the corpus is stamped**, which
is the correct order: `npx tsx scripts/declarations/backfill_asset_table_num.ts --apply`,
then `db:load:declarations:pg` (phase 1 — `--resolve` alone does NOT rewrite asset rows),
then the matview refresh. Applying 089/090 to Cloud SQL without shipping the re-parsed
shards changes nothing there while local is correct, with every row count reconciling.
- **`category` and `legal_basis` cannot stand in, and the second is the trap.** A rented
flat is still `real_estate`. And Пеевски's чужди cars carry `legal_basis = 'договор'` —
which is also what Румен Радев's OWN car carries. A probe on `legal_basis ILIKE '%наем%'`
finds 948 real-estate rows in the 2026 annuals and misses the vehicle side entirely.
- **The top of the leaderboard does not move, which is why this survived.** Of 19,188 people
with a published net worth the top 100 is unchanged — the largest fortunes are real. The
damage is per-profile: **882** people affected in their latest year, **217** ≥50% чуждо,
**106** ≥90%, one at **100%** (Стефан Добрев Стайков, €445,386 → €0), **552** falling 50+
ranks, and eight going from a published positive net worth to declared net liabilities.
No aggregate check can see this shape.
- **Excluding is not hiding.** The rows stay in `declaration_asset` and render on `/person`
under „Ползва, но не притежава" with the contract-price caveat — „declares no property of
his own and rents eight houses" is the finding, and dropping the rows would lose it.
`person_declaration_detail` carries `usedAssetRows` / `usedContractEur` beside the totals,
and every asset row carries `tableNum`, `legalBasis` and a server-derived `isHolding`.
⚠️ **THE LEASE ASYMMETRY — know this before quoting a negative net worth.** Table 3.4's
dominant use is a LEASED vehicle: **1,014 of the 1,826 filings carrying a 3.4 row also carry
a лизинг debt**. Under a lease the lessor owns the car, so 3.4 is the right table — but the
lease liability sits in table 7 and stays counted, so excluding the asset while keeping the
debt is asymmetric. **103 people move from a published positive net worth to a negative one,
70 of them on exactly this pairing.** **Decided 2026-08-18: strict, with the caveat** — they do not own the car
and they do owe the money, which is what the form says. The „ползва" block prints each row's
„правно основание" and, when a filing pairs a чуждо vehicle with a лизинг debt, says in words
why the net figure can go below zero. Do not "fix" this by dropping the matched debts without
measuring first: the table-7 description is free text and the same match would drop lease
debts against property the declarant DOES own. Pairing them and dropping both is NOT reliably implementable — the table-7
description is free text, and matching „лизинг" would also drop lease debts against property
the declarant DOES own. See `docs/plans/declaration-foreign-assets-v1.md` §4d.
**Measured locally 2026-08-18** (the pre-`table_num` estimates in that plan's §2 over-matched
and are superseded): 7,278 чуждо rows / €73.6m raw; **€55,565,683** left `person_wealth_year`
across 1,980 person-years and 1,120 people; corpus assets €4,213,997,747 → €4,158,432,064;
795 latest-year profiles affected, 77 of them reduced to €0; 562 fall 50+ ranks and the top
100 does not move. `/mp-cars` went 721 → 643 rows, €7,109,059 → €5,483,110, and **41 MPs left
the register entirely** because every car they declared was leased or provided.
Nine surfaces read the rule: 090 (the matview join + `person_declaration_detail`), 092, 100,
105 ×3, plus the TS builders `scripts/officials/rankings.ts`,
`scripts/declarations/build_assets_rankings.ts`, `build_car_makes.ts` (612 чужди cars were
counted as MPs' own, €11.5m) and `scripts/person/compare_declarations.ts`.
**The gate is `scripts/db/tests/declaration_foreign_assets.data.test.ts`.** Its last test is
an EXHAUSTIVENESS sweep in the `declared_label` style — every function/view/matview whose
definition reads `declaration_asset` and SUMs a value must route through the predicate or be
listed in `HOLDING_FILTER_EXCEPTIONS` with a reason, so a new wealth surface fails until
someone decides, and a stale exception fails too. It also carries a mutation check
(the filtered recompute is compared against the UNfiltered one, so an assertion satisfied by
two implementations that both forgot the filter cannot pass) and skips with a DISTINCT
reason when `table_num` is entirely NULL — "the corpus has no provenance yet" must never
read as "the rule is enforced".
⚠️ **Deferred, and it is a live parse-time ambiguity:** table 1.2's area column is headed
„Площ /**декара**/" while table 1's is „Площ кв.м.", and the parser feeds both into
`area_sqm`. Measured: all 1,787 declared 1.2 tables carry the декара header, but 5,375 of
5,830 rows are bare numbers, 451 say кв.м. and 4 say декара — the declarants largely ignore
it and the unit is unrecoverable per row. That matters because `perSqmAnchor` drives
`correctRealEstateSeparatorTypo`, which **mutates the stored value**.
`check_suspicious_values.ts` now marks a чуждо flag as such so a reviewer does not
"correct" a contract price as if it were a purchase price, but nothing else is fixed. See
`docs/plans/declaration-foreign-assets-v1.md` §6.
### `value_basis` — whether a declared euro figure is the declarant's or OURS
**A foreign-currency asset row used to vanish from every wealth total, silently.** Each money
table (4 налични, 5 банкови сметки, 8 вложения) carries a „Равностойност в лв./в евро." cell
that the DECLARANT fills in, and `pickEurValue` prefers it. Where it was left blank, a
USD/GBP/CHF row was stored with `amount` + `currency` and a NULL `value_eur` — and dropped out
of `person_wealth_year`, `/persons`, `/officials/assets`, `/mp-assets` and the officials
rankings with nothing flagging it. Measured 2026-08-18: **462 rows over 163 people**, 356 of
them on filings `person_wealth_year` publishes (155 people, 280 person-years).
It was never a rounding caveat. Лъчезар Богомилов Иванов's 2021 was published at €254,294
against a true €3,652,248 — **7% of the truth**; Пеевски's 2017 at €2,503,406 against
€5,064,422 (a single 4,481,442 USD balance); and Владимир Славев Табутов's 2023 at
**−€121,331**, i.e. declared net liabilities, against a true **+€504,142** — the _sign_ was
wrong. Twelve of the rows are `debt`, which OVERSTATES net worth, the one direction 090's
header says this must never fail in.
The rule is `scripts/declarations/fx.ts` and the column is `declaration_asset.value_basis`:
```
'equiv' the declarant's own Равностойност cell 'fx_ecb' OURS — see below
'peg' BGN/EUR at the locked 1.95583, or EUR 'legacy' valued by an older parser
NULL no euro figure — COUNTED in excluded_asset_rows
```
`fx_ecb` is the ECB reference rate at the last quoted day of the period the filing covers
(`COALESCE(fiscal_year, declaration_year)`), from the **committed** `data/declarations/
fx_year_end.json`. Plan: `docs/plans/declaration-fx-conversion-v1.md`.
Six things about it are easy to get backwards:
- **`pickEurValue` RUNS FIRST AND WINS.** We fill a blank, never override a filing — that is
what makes `fx_ecb` mean „the declarant stated no equivalent". A row they valued keeps their
number even when it implies an absurd rate (the corpus holds 10× and 0.1× errors).
- **YEAR-END, not an annual average**, because tables 4/5/8 declare a STOCK — a balance as of
31 December — and an average is the wrong statistic for a point-in-time quantity. And **not
a reverse-engineered „declarant convention" either**: the 4,347 declarant-valued rows imply
a median that matches the ECB year-end to four decimals for 2018/2019/2021/2025 and matches
nothing in particular for 2016/2020/2022. There is no convention to reproduce.
- **THREE currency lists, and merging any two is the defect.** `EUR_RATE`
(`src/lib/currency.ts`) = folds at a FIXED rate; `FX_CURRENCIES` (`fx.ts`) = converts at a
DATED one; `is_crypto_asset`'s fiat list (090) = „is this money at all". „ДОЛАРА" is fiat and
not fixed-rate; „ЕВРО" is fiat and IS fixed-rate. Putting USD in `EUR_RATE` is a fixed rate
for a floating currency — wrong in every year but one.
- **Keying the rates on SPELLINGS rather than a canonical code renders „евро" as „лв".**
`formatNative` asks „is this in EUR_RATE and not the string 'EUR'?" to choose between € and
лв, so a euro spelling sitting in the rate table is a lev to every formatter. Spellings fold
through `canonicalCurrency()` first, and `normCurrency` is deliberately identical to 090's
`asset_unit_norm` so the two sides cannot sort the same cell differently.
- **`excluded_asset_rows` HAS TWO ARMS, and the second must NOT carry the first's
`category NOT IN ('debt','credit_limit')` filter.** The ceiling arm excludes debts because
dropping a debt from the ASSET side is not a hole; an unvalued debt is one. Copying the
filter is the trap. `person_wealth_year` also exposes `imputed_asset_rows` / `imputed_eur`,
so any surface can say how much of a total we computed — converting silently would only
replace a silent omission with a silent invention.
- **An inverted rate is invisible to a tolerance band near parity.** Inverting USD 2016 moves
it from +10.4% off the declarant median to +22.7%; both fit any band loose enough to admit
the real corpus. `declaration_fx_conversion.data.test.ts` therefore holds hand-verified ECB
ANCHORS, and its calibration arm filters `value_basis = 'equiv'` — comparing against our own
converted rows would calibrate the table against itself and pass by construction.
**Re-parse, then reload — the same order as `is_declared_holding`, and for the same reason:**
no SQL can backfill this, because whether a figure came from the declarant's cell or the peg is
only recoverable from the source XML.
```bash
npx tsx scripts/declarations/backfill_asset_fx.ts --apply # offline, reads raw_data/
npm run db:load:declarations:pg # phase 1 — --resolve does NOT rewrite asset rows
npx tsx scripts/declarations/rebuild_post.ts # mp-assets/*, car-makes, rankings
```
Cloud side, and nothing runs it automatically — `db:load:declarations:pg:cloud` (phase 1) then
`-- --resolve`. Applying 089/090 to Cloud SQL **without shipping the re-parsed shards changes
nothing there while local is correct**, with every row count reconciling.
**The rate table is operator-run and moves once a year**, when a year closes:
`npx tsx scripts/declarations/fetch_fx_rates.ts --apply`. It REFUSES when a settled historical
rate has moved (the ECB revises nothing, so that means the parse changed) and exempts only the
still-open current year, whose entry is the last day quoted so far and legitimately moves until
the year closes.
⚠️ **The residue is the design, not a gap.** 8 rows stay unvalued — filings whose committed
shard row set disagrees with a fresh parse, so the backfill refuses to touch them (an older
parser had missed their `amount` column entirely). They are COUNTED rather than guessed, and
the gate asserts both that they are counted and that the residue has not grown.
### `held_scope` — whether declared money sits in Bulgaria or abroad
**Tables 5 („Банкови влогове") and 8 („Вложения в … фондове") carry a „В страната" /
„В чужбина" cell pair that we ingested none of until 2026-08-19.** The register publishes
it per account — Иво Христов Петков's 228,100 EUR account is marked „Белгия" while his
other five say „да" — and our rows for the two were byte-identical in every stored column.
`declaration_asset.held_scope` / `held_country` / `held_raw_in_country` / `held_raw_abroad`
close it. Rule: `classifyHeldPlace` in `scripts/declarations/held_abroad.ts`. Plan:
`docs/plans/declaration-held-abroad-v1.md`.
Measured over 76,953 money rows: **95.46% domestic, 4.15% abroad (€168.5m, 765 people),
0.38% unknown**.
Five things about it are easy to get backwards:
- ⚠️ **TABLE 4 („Налични парични средства") HAS NEITHER COLUMN, and its `Cell Num="7"` is
„Произход на средствата".** The obvious specification of this work said tables 4/5/8;
reading the pair off table 4 does not yield a blank, it yields the funds origin, so all
25,717 cash rows would publish as held in a country called „заплата". The **pre-2018**
form carries the pair on table 7 at cells **6/7**, not 7/8 — no special case was needed
only because the pair goes through `columnResolver` like every other cell.
- ⚠️ **NULL IS NOT `'unknown'`, AND NEITHER IS `'domestic'`.** NULL means the row's table
has no such question (every real-estate, vehicle and cash row, and everything parsed
before the column existed); `'unknown'` means the filing answered unintelligibly. The
answer is TRI-STATE because a boolean would have to invent one: the cells are free text
the register does not validate — 5,691 distinct spellings on the „В страната" side — and
346 rows leave both blank, ~130 tick both, and ~93 SPLIT one amount across the two
columns (151,744 + 967 against an amount cell of 152,711).
- ⚠️ **`held_country IS NULL` IS NOT EVIDENCE OF BEING DOMESTIC.** „да" in the „В чужбина"
column says abroad and names nowhere: a country is named on only **521 of 3,288** abroad
rows, **11.6% of the money**. „How much is abroad" is answerable over `held_scope`;
„where" is answerable only over the named subset, and a surface reporting it must say so.
- **Content overrides the column it sits in, and a named place beats a bare tick.** 47 rows
answer domestically inside „В чужбина"; and „РБългария" beside a bare „х" is a declarant
naming their country and STRIKING OUT the column that does not apply — „х" is a tick to
some filers and a strike-through to others. A lone denial asserts the OTHER column (the
pair is exhaustive), which is the only statement 81 rows make.
- **Re-parse, then reload — the same order as `is_declared_holding` and `value_basis`, and
`--resolve` alone does NOT rewrite asset rows.** No SQL can backfill this; the cells exist
only in the source XML, so the whole change is INERT until the shards are stamped.
```bash
npx tsx scripts/declarations/backfill_asset_held_abroad.ts --apply # offline, reads raw_data/
npm run db:load:declarations:pg # phase 1
npm run db:load:declarations:pg -- --resolve # phase 2 — refills person_id
```
Phase 2 is not optional even though nothing here reads `person_id`: phase 1 TRUNCATEs
`declaration`, so skipping it leaves every filing unresolved.
⚠️ **On Cloud SQL, SHIP these columns — do NOT reload for them.** The `:cloud` twin of those
two commands works and costs a measured **~8 minutes of 500s** on `/persons`,
`/officials/assets`, `/mp-assets` and `/declarations/crypto` (phase 1 NULLs every
`person_id`; phase 2 runs 090's `DROP MATERIALIZED VIEW … CASCADE`, and a DbDataTable
resource has no `missingMigration` degrade). **Re-measured 2026-08-29 on
`db-perf-optimized-N-2`, 61,743 filings: phase 1 = 39 s, phase 2 = 4m52s** — so the PAIR is
~5m30s of loader wall-clock. ⚠️ That is the loaders' runtime, NOT a re-measurement of the
outage. Note the window is bounded by the pair only when nothing runs BETWEEN them —
insert a resolve, as the person chain does, and the degraded span is the whole span.
⚠️⚠️ **THE OUTAGE WAS FINALLY POLLED ON 2026-09-04, AND IT DID NOT HAPPEN. Do not keep citing
„~8 minutes of 500s" as a reason not to run this.** One probe every 30 s across a full
`db:resolve:persons:cloud` → `db:load:declarations:pg:cloud -- --resolve` → 8-loader repair
chain (14:41:37→15:02:35Z, 20m58s end to end): **38 samples, every one HTTP 200, zero 500s.**
Probed `/api/db/table` for `persons`, `officials_rankings`, `crypto_holdings` and
`abroad_holdings` — i.e. three of the four surfaces named above plus one more.
Read that as ONE measurement, not a guarantee: `mp_assets_rankings` was not probed, a 30 s
interval cannot exclude a sub-30 s blip, and this was an idle-hours run on
`db-perf-optimized-N-2`. But the claim it replaces was never measured at all — the sentence
here used to say so — and an unmeasured figure had been load-bearing in arguments against
running the chain. If a future run does degrade these pages, re-measure and say so here rather
than restoring the old number. The values are derived from immutable filings,
so they are identical whichever database computes them — the `ship_filed_position.ts`
argument — and `scripts/db/ship_held_abroad.ts` writes them into the rows already there:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts 089_declarations.sql
npx tsx scripts/db/ship_held_abroad.ts --to postgres://postgres@127.0.0.1:5434/electionsbg # dry run
npx tsx scripts/db/ship_held_abroad.ts --to postgres://postgres@127.0.0.1:5434/electionsbg --apply
```
Measured 2026-08-19: **76,953 rows in 31 s at 13:15 on a Wednesday**, RowExclusiveLock only,
`person_id` / `filed_position` untouched, all five matviews still populated, all four pages
200 throughout. The key is `(source_url, seq)` — `declaration_id` is a `bigserial` handed out
in insertion order — and the payload carries `category` so that **any** row-identity
disagreement refuses the whole ship rather than publishing „Белгия" against somebody else's
account. Reload instead only when the SHARDS have moved for some other reason; the shipper
carries these four columns and nothing else. Applying 089 alone changes nothing while local
is correct, with every row count reconciling.
**The gate is `scripts/db/tests/declaration_held_abroad.data.test.ts`** (8 tests), plus 17
unit tests on the rule. It carries a mutation check — every stored value is re-derived from
the stored RAW cells, which is what those two columns are for — and skips with a DISTINCT
reason when `held_scope` is entirely NULL, so „the corpus has no provenance yet" cannot read
as „the rule is enforced".
### `is_spouse` — the ONE backfill in this family that needs no re-parse
`declaration_asset.is_spouse` answers „is this row somebody else's" — the form names a holder
per row („Собственик или титуляр на правото") and it is frequently not the declarant. The rule
is `isSpouseHolder` (`src/lib/declarations.ts`), read from TWO sides that cannot share a query:
the PARSER stores it on the asset row, while the `/person` stake renderer derives it live,
because `declaration_stake` has no such column.
⚠️ **It is a pure function of two STORED fields, and that changes the whole procedure.**
`table_num`, `value_basis` and `held_scope` above each exist only in the source XML, so each
needs a re-parse that indexes `raw_data/`, matches rows positionally and can refuse a shard.
This one needs none of that: `holderName` is on the row and `declarantName` is on the
declaration, both already in the committed shards. So the restamp reads only the shards, has no
mismatch class, and no operator should ever point a 5-hour crawl at it:
```bash
npx tsx scripts/declarations/backfill_asset_is_spouse.ts --apply # shards only, no network
npm run db:load:declarations:pg # phase 1
npm run db:load:declarations:pg -- --resolve # phase 2 — refills person_id
npx tsx scripts/declarations/rebuild_post.ts # car-makes.json + mp-cars.json carry isSpouse
npm run db:load:mp-roster:pg # mp-cars.json → mp_cars.is_spouse (104)
```
⚠️ **The last two look skippable and are not.** Neither load phase touches the COMMITTED
artifacts: `build_car_makes.ts` writes `isSpouse` into `data/parliament/car-makes.json` and
`mp-cars.json`, and `load_mp_roster_pg.ts` loads that file into `mp_cars.is_spouse`, which 105
serves to `/mp-cars`. `buildCarMakes` reads only each MP's LATEST filing, so a restamp is a
no-op unless a flip lands on one — twice now it has not, which is luck rather than design. The
first time it does and this step was skipped, `/mp-cars` ships „съпруг(а)" against an MP's own
car at a 200 with nothing failing. The `value_basis` sibling above documents the same step.
Cloud side is the `:cloud` twin of both and nothing runs it automatically. The stake side needs
neither — it is computed at render time, so a rule change reaches it with the next build, which
is exactly why the two desync in between.
**The fold is separator-only, and the distinction is load-bearing.** It removes case, hyphen
spacing and — since 2026-08-19 — every remaining separator, so a declarant who typed their own
name with a lost space („Николай МихайловКолибаров"), a hyphen for a space („Багдатова \_
Мизова") or a stray comma, slash or digit („Попдимитро3в") is no longer chipped as a third party
on their own page. It never folds TOKENS: a shared surname is the SPOUSE case this exists to
mark, so a token-set or reordering fold would delete real findings rather than typos. Measured
2026-08-19: 563 of 110,835 asset rows cleared, 8 of 4,830 stake rows.
**A cell with NO letters names nobody**, and the guard for that is separate and SYMMETRIC.
„-", „.", „0", „91697", „\*6" and a 255-char run of „#" all survive `normHolderName` (which
keeps punctuation and digits), so before 2026-08-19 they compared unequal and published „held
by someone else" against a named individual — 21 asset rows and 8 stake rows, one rendering a
chip whose entire text was „.". The declarant half is not hypothetical either: two filings name
the declarant „0" and two name them „4", carrying 9 marked rows between them, so guarding only
the holder would assert a third party on the strength of the DECLARANT's typo. Together those
took the asset side to **110,242** and the stake side to 4,814 / 5,184 / 8,571.
**The gate is `scripts/db/tests/declaration_is_spouse.data.test.ts`** (3 tests). Unlike its
neighbours it is a FULL-CORPUS recompute rather than a sample — the rows that move are hand-typing
accidents, rare and unevenly spread, so a sample misses all of them — and it carries a mutation
check: with the second pass deleted, two of the three fail and the first names the restamp
command. The measured split lives in `isSpouseHolder`'s header and nowhere else; do not quote a
raw `holder_name <> declarant_name` count, which INVERTS the majority.
### `declared_label()` — the ONE definition of which office label a reader sees
**Nine serving surfaces now read these columns, and all of them go through
`declared_label(p_filed, p_listed)` in `089_declarations.sql`** —
`COALESCE(nullif(btrim(p_filed), ''), p_listed)`, IMMUTABLE, PARALLEL SAFE and deliberately
**not** STRICT (STRICT would short-circuit the whole fallback). Shipped and deployed to Cloud
SQL 2026-08-17; the helper and its full-corpus gate are the current specification.
**Never restate that COALESCE at a call site.** Twelve hand-copied copies is the shape that
produced the six-way `magistrate_current` duplication where "someone missed one" fired twice
in one day. The precedent is `kzk_effective_suspension(suspension, status)` in 042.
**⚠️ Which surfaces take the filed value is decided by what the COLUMN is for, NOT by the
tier it spans.** Getting this backwards is the live defect this section exists to prevent:
| the column's job | surfaces | source |
| ------------------------------------------------------ | ------------------------------------- | ------------------------------ |
| a rendered label / substring search (`filter: "text"`) | 090 ×3, 093 ×2, 098, 100, 105, 159 | **filed**, listing as fallback |
| an exact-match FACET KEY (`filter: "in"`) | 120 `person_browse_table.institution` | **listing** |
| renamed into a different contract | 102 `municipality`, `role_raw` | **listing** |
The two exclusions are deliberate and their own files' headers say so — do NOT "finish the
job":
- **120** — `db_table.js` exposes `institution` as `filter: "in"` and the picker facets that
same column, so the distinct values ARE the picker. The listing is a 1,013-value controlled
vocabulary; the filed value is free text and takes the column **991 → 12,626**, which does
not make the picker noisier so much as stop it being a picker. The fragmentation is driven
by the EXEC tier, so narrowing to one tier does not rescue it. The accepted cost: exec group
buckets survive there as facet VALUES, which is coherent — „the heads of foreign missions"
is a usable bucket and only false as a claim about one named person.
- **102** — `ld.institution` is renamed `AS municipality` and holds the município NAME
(„Ямбол") where the filing holds the EMPLOYER („Община Ямбол"; 25 Видин rows say „Общински
съвет - Видин", a council), 6,576 of 6,613 rows differing. `ld.position_title` is renamed
`AS role_raw`: five clean roles against **563** free-text spellings, one of which names the
body instead of the role („Общински съветник" → „Общински съвет").
**A disagreement COUNT cannot tell correction from noise.** On exec the listing invents group
buckets describing nobody and the filed value is a fix; on muni the listing is a clean
controlled vocabulary and the filed value is unnormalised free text. Both read as "20-42% of
rows disagree". Only reading the VALUES separates them — this plan got 102 wrong twice by
trusting the rate.
**Apply order — `148` must be in the command and precede `120`.** 090 opens with
`DROP MATERIALIZED VIEW person_wealth_year CASCADE`, which takes SIX dependents, so every one
must be recreated in the same run; and 120's matview selects `person_company_bridge_a`, which
only 148 creates. Omit 148 and 120 raises 42P01 **after** the CASCADE has already deleted
`person_browse_table` — `/persons` down with nothing left to rebuild it. A local run can pass
without it purely because the view is already there.
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5433/electionsbg npx tsx scripts/db/apply_functions.ts \
089_declarations.sql 090_person_wealth.sql 093_declaration_events.sql 097_cohort_benchmark.sql \
098_new_filings.sql 100_officials_rankings.sql 102_municipal_officials.sql 104_mp_roster.sql \
105_mp_serving.sql 148_person_company_basis.sql 120_person_browse.sql 159_person_crypto.sql \
169_person_abroad.sql
```
Measured: 21.8 s local, **8m02s on Cloud SQL**, during which `/persons`, `/officials/assets`
and `/declarations/crypto` answer **500** — DbDataTable resources have no `missingMigration`
degrade. Off-peak only. Prefer `db:load:declarations:pg:cloud -- --resolve` (which applies the
same files in its own order — **measured 4m52s on `db-perf-optimized-N-2`, 2026-08-29**, i.e.
about half the hand-run apply above) unless the CASCADE dependents have been resolved against the
TARGET database, which is what makes the short command safe.
**⚠️ After SHIPPING VALUES, refresh — do NOT re-apply.** Only three matviews read
`declared_label` (`officials_rankings_table`, `person_crypto_table`, `person_abroad_table`) and all carry a UNIQUE
index, so `REFRESH MATERIALIZED VIEW CONCURRENTLY` on the pair is **14 s with no reader
blocking**, against the ~8-minute outage a re-apply costs. Re-apply only when a DEFINITION
changed:
```sql
REFRESH MATERIALIZED VIEW CONCURRENTLY officials_rankings_table;
REFRESH MATERIALIZED VIEW CONCURRENTLY person_crypto_table;
REFRESH MATERIALIZED VIEW CONCURRENTLY person_abroad_table;
```
**The gate is `scripts/db/tests/declaration_filed_position.data.test.ts`** (18 tests). Its last
one is an EXHAUSTIVENESS sweep: it enumerates every function, view and matview whose definition
reads `declaration` and mentions these columns, and fails unless each is either routed through
`declared_label` or listed in `LISTING_LABEL_EXCEPTIONS` with a reason — so a NEW surface fails
until someone decides, and a stale exception fails too. It also carries a shipped mutation
check, because an assertion comparing a payload to `declared_label` is otherwise satisfiable by
an inverted implementation both sides agree on. Note the sweep alone cannot catch **swapped
arguments** (`declared_label(d.institution, d.filed_institution)` passes it), which is why the
per-surface arms exist.
`db:refresh` never shows this because it runs phase 1 before the resolver every time. Skipping
phase 1 on the cloud leaves the stale ref joining to nothing: the filing keeps a NULL
`person_id`, so that person's declaration drops off `/person` and out of the "с декларация"
facet while every row count still reconciles. `--resolve` does print an
`N/total still NULL` line, so it is not literally silent — but a single-digit N against
47,983 reads as ordinary residue, which is exactly how this one was missed. Caught 2026-07-31 — after the
2026-07-29 collision fold, exactly one of 47,983 filings (`ivan-georgiev-ivanov1-94805e`)
stayed unresolved on Cloud SQL until phase 1 was re-run. Re-run
`db:load:persons-browse:pg:cloud` afterwards, since `has_declaration` reads `person_id`.
Likewise, after `db:load:declarations:pg:cloud -- --resolve` (which creates the municipal
roster matview), run the candidateLink loader so the Cloud SQL municipal roster carries the
party colours / councillor avatars the /governance + My-Area tiles render:
```bash
npm run db:load:official-candidate-links:pg:cloud
```
It populates `official_candidate_link` and REFRESHes `municipal_officials_table`. `db:refresh`
runs the local equivalent automatically; the cloud side does not. See
`docs/plans/persons-pg-retirement-v1.md` (T1.5).
Same shape, and it must run **BEFORE** `db:resolve:persons:cloud` rather than after: the
judicial dimension (migration 116) is what gives every magistrate their court on `/person`.
```bash
npm run db:load:judicial-bodies:pg:cloud
```
It rebuilds `judicial_body` + `judicial_body_alias` from `magistrate.court` ∪
`court_load.name`, so re-run it whenever **either** of those is reloaded on the cloud side
(`db:load:magistrates:pg:cloud`, `db:load:court-load:pg:cloud`) — then re-run
`db:resolve:persons:cloud`, which reads the alias table. Skipping it does not fail: the
resolve simply finds an empty alias table and ~2,700 magistrate roles publish with no
court, green locally and blank on prod. See
`docs/plans/person-role-place-consolidation-v1.md` (T2).
**`db:load:magistrates:pg:cloud` is itself a PREREQUISITE of the resolve, not merely a
trigger for the loader above**, and it is the newest member of this section:
```bash
npm run db:load:magistrates:pg:cloud # BEFORE db:load:judicial-bodies:pg:cloud → db:resolve:persons:cloud
```
`magistrate_holdings.json` used to be a latest-year snapshot, so the ИВСС register's yearly
turnover deleted every magistrate who stopped filing — and since `resolve_persons.ts` builds
its magistrate mentions from `SELECT name, court FROM magistrate`, that deleted their person
row and 404'd every `/person` URL they had been served under. The roster now ACCUMULATES
(each magistrate's most recent annual filing), which is only useful if the serving database
has it: run the resolve against a cloud `magistrate` table that predates the change and prod
drops the same 462 people at a 200, with every row count reconciling. `magistrate` is loaded
ONLY by this command on the cloud side. `magistrate_roster_retention.data.test.ts` catches it
locally; nothing checks the cloud.
**The current bench is `magistrate_current` (a view in 070), and 070 must reach a database
BEFORE 116 does.** Because the roster retains departed magistrates, every count captioned in
the present tense has to exclude them — and that predicate used to be copied six times across
070, 116 and `scripts/db/lib/seo_courts.ts`. "Someone missed one" then fired twice in one day
(5325a6ef37 scoped 116's two counts, fabf683666 the prerender's, hours apart), so the rule is
named once and the copies read it. The consequence for deploys: 116's two functions now
select a view that only 070 creates, and 116 sets `check_function_bodies = false`, so applying
it alone to a warm database SUCCEEDS and then raises 42P01 on the first `/court` call. Ship
them together, 070 first:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts \
070_magistrates.sql 116_judicial_body.sql
```
`db:load:magistrates:pg[:cloud]` applies 070, so a roster reload carries the view; a
function-body fix in 116 on its own does not wait for one. Two properties of 070 protect the
dependency and must not be undone: the view is `CREATE OR REPLACE`, never `DROP` (a DROP in
this loader-applied file is the 2BP01 that stalled `db:load:pg` for a day — 077/145 — and
`DROP … CASCADE` is the silent variant of 003), and 116's references stay in `LANGUAGE sql`
STRING bodies, which record no `pg_depend` edge; converting them to PG14+ `BEGIN ATOMIC` would
make a future DROP in 070 fail 116 outright.
Related, from the same consolidation: `magistrate.decl_year` is now `NOT NULL`, because
`decl_year = max(...)` is NULL-false and one year-less row would vanish from every
current-bench count at once. Warm databases get it from a GUARDED reconcile at the foot of
070 — guarded because the loader applies the schema BEFORE it truncates and reloads, so an
unconditional `SET NOT NULL` against a legacy NULL row would abort the ingest in the apply
phase and leave the corpus on its previous vintage.
**One-off, and Cloud SQL needs it by hand.** The retention's first cut keyed the roster on the
register's RAW name, so a magistrate the ИВСС re-spelled between years (hyphen spacing —
„… Средкова - Петрова" in 2025, „… Средкова-Петрова" in 2026) survived as two rows and minted
TWO person rows for one human. The writer now drops a retained record whose `normName` is
already on the current bench, which deletes the duplicate profile — and orphans the slug it
was served under. Unlike the 462, the redirect target here genuinely exists and is the same
person, so these two get one:
```sql
INSERT INTO person_slug_retired (slug, target_slug) VALUES
('maya-sredkova-petrova-9sqndm', 'maya-sredkova-petrova-j1j9ru'),
('milena-kirova-stoyanova-1lk9uj', 'milena-kirova-stoyanova-1aa2jp')
ON CONFLICT (slug) DO UPDATE SET target_slug = EXCLUDED.target_slug;
```
Only needed on a database that ran the roster's first cut (2026-08-11). It cannot recur: the
dedupe runs before emission, so a re-spelling is never published again.
**It is now ALSO what makes `/court/:bodyCode` truthful.** `judicial_body_source_name` is the
un-folded bridge `judicial_body_detail()` joins `court_load` through — the raw-name→body fold lives
in TypeScript, so SQL cannot do it. Applying 116 with `apply_functions.ts` (the normal way a
function change ships) CREATEs that table EMPTY, and an empty bridge returns `load: null` for every
body: shape-identical to a real prosecution office, so all 279 pages would assert at a 200 that the
ВСС publishes no workload for them — including Софийски районен съд. The payload therefore carries
`sourcesBuilt`, and the page says "not loaded yet" rather than "nothing published" when it is
false; the fix is to run this loader. `judicial_body_detail.data.test.ts` fails if the flag stops
discriminating.
**And it is now a BUILD-TIME dependency, which is the one requirement on this page that is
not a `:cloud` command at all.** `/court/**` is prerendered — 279 static pages plus 279 EN
mirrors, a `sitemap_judiciary.xml` shard and a table in `llms-full.txt` — and all four are
enumerated by `scripts/db/lib/seo_courts.ts`, a build-time reader over `judicial_body`. That
reader returns `[]` on any failure by design, so **the machine running `npm run build` needs
the dimension in its LOCAL Postgres**, not merely the deploy target:
```bash
npm run db:load:judicial-bodies:pg # local — before `npm run build`
npm run db:load:judicial-bodies:pg:cloud # the serving database, as above
```
Building without it does not fail — it emits zero court pages, no `<loc>`s for them and a
corpus missing that section, all at exit 0, which is the good failure but an invisible one.
Two gates make it visible instead: `scripts/sitemap/families.data.test.ts` asserts every
`/court` and `/pension-fund` `<loc>` has a `dist/<path>/index.html` (the sitemap is COMMITTED
while `dist/` is not, so a sitemap minted with Postgres survives a build without it — but the
gate only sees a `dist/` that exists, so run it AFTER `npm run build`), and `buildFull.ts`
refuses to rewrite `llms-full.txt` when the judiciary section would disappear from it. The
`tests/seo.spec.ts` court samples skip rather than fail on a database-less checkout.
`/pension-fund/**` is the same page-family shape with none of this exposure: its source
`data/budget/kfn/funds.json` is committed, so it builds anywhere.
**Sofia's courts had FIVE duplicate bodies until 2026-08-05, and the shape is worth
knowing because it can come back.** Sofia's institutions have adjectival names
(`Софийски районен съд`), so they get curated entries in `judicialBodies.ts`'s `NATIONAL`
list, which is checked BEFORE the generic seated rules — precisely so `Софийски районен
съд` cannot fall through and mint `rs-sofiya`. That defence only ever covered the
spelled-out spelling, while `court_load` — the ВСС's own workload series — writes the
abbreviated one (`РС-София`, `ОС - София`, `АдмС - София-град`). So the abbreviated form
sailed past the national rule into the seated one and minted exactly the five codes the
national rule exists to prevent, each splitting one court's magistrates onto one page and
its workload onto another: `/court/as-sofia-grad` stated the ВСС publishes no workload for
it while publishing eight years of it under `as-sofiya-grad`.
`foldJudicialName` now spells out a LEADING institution abbreviation (`РС` → `РАЙОНЕН СЪД`
…), so both spellings reach the national rule as one key. `АС` and `ВС` are deliberately
excluded — they collide across families and only `resolveJudicialBody`'s `tier` hint can
settle them. Two gates in `judicial_body_detail.data.test.ts` hold it: no two bodies share
a (kind, tier, seat) outside a named allowlist, and no court has its magistrates on one
row and its workload on another.
Same shape again, and also **BEFORE** `db:resolve:persons:cloud`: the canonical place
dimension (migration 117) is the code→name dictionary `082_person_api.sql` JOINs for the
`mir` / `obshtina` label on every `/person` role, and `/procurement/by-settlement` joins
for settlement names.
```bash
npm run db:load:place-dim:pg:cloud
```
It is built from `data/settlements.json` + `data/municipalities.json` (labels via
`scripts/person/places.ts`), so re-run it whenever either file changes. `db:refresh` runs
the local equivalent automatically; the cloud side does not.
Related, and the only loader whose trigger is a **calendar rollover** rather than a source
change — the pscope windows (migration 118) **and the per-scope precomputes they drive**
(migration 119, `procurement_settlement_rank` + `procurement_geo_payloads` behind
`/procurement/by-settlement`; migration 122, `contractor_rank` + `contractor_scope_kpis`
behind `/procurement/contractors`; migration 123, `procurement_settlement_payloads` behind
every `/procurement/settlement/:ekatte` page and My-Area procurement tile; migration 124,
`procurement_payloads` behind the six `/api/db/procurement-*` dashboard routes):
```bash
npm run db:load:procurement-scopes:pg:cloud
```
It writes the window rows, applies 119 + 122 + 123 + 124 and REFRESHes all six matviews
(**46 s** local, measured end to end; `contractor_rank` fans ~29.5k contractors × ~30 windows ×
CPV-division rollup ≈ 9 s, 123 fans 869 settlements × 30 windows ≈ 10 s, and 124 fans 6 kinds ×
30 windows ≈ 10 s) — so "the scopes changed" and "the precomputes match the scopes" can never
be two separate states.
`contractor_scope_kpis` reads `contractor_rank`, so it is refreshed after it. **Cloud SQL is
unmeasured and will be materially slower** — the whole reason 123 exists is that the same
per-settlement call is 401 ms locally and had not finished at the 10 s `statement_timeout`
that aborted it on a cold `db-g1-small`. **Measured 2026-08-22 on `db-perf-optimized-N-2`: 87 s**
(46 s locally), so the old "expect minutes, not seconds" warning was pessimistic — but the six
matviews are still rebuilt and a plain `REFRESH` still takes an AccessExclusiveLock, so it is
still a reader-visible window. Re-run it:
- whenever a new election lands in `src/data/json/elections.json` (a new `ns:` window);
- **every January** — the year windows are enumerated `SCOPE_FIRST_YEAR..currentYear`, so on
1 January the `?pscope=y:<new year>` option appears in the UI while the table still stops
at the old year, and that scope serves an empty page;
- after a standalone `db:load:place-dim:pg:cloud` **only when it reports that it skipped its
own refresh** — it changes the English settlement names the ranking joins, and it now
refreshes them itself whenever the dimension actually moved (see the side loaders
below). Re-running this loader after it already did that is a needless multi-minute cloud
rebuild.
The refresh list, its ORDER, **what each matview is built FROM** and the not-populated
fallback live in ONE place — `scripts/db/lib/scopedMatviews.ts`. A migration that adds a
per-scope matview must join `SCOPED_MATVIEWS` there or no loader will ever refresh it;
`procurement_settlement_payloads.data.test.ts` fails on any matview that reads
`procurement_scopes` and is missing from the list.
**The `inputs` array on each entry is as load-bearing as the name, and it fails differently.**
A missing NAME means nothing ever refreshes that matview; a wrong `inputs` means the loader for
the undeclared table skips it, so one page keeps the previous attribution while the rest of the
site has moved on. That is invisible to a row count and to the exhaustiveness gate above, which
only checks presence. `procurement_payloads.data.test.ts` closes it for 124: it reads that matview's six
function bodies out of `pg_get_functiondef` and fails if it reads a table it does not declare.
(For 124 only — no gate does this for the other entries yet.) It is not a hypothetical: 124's
first committed draft declared `["contracts", "awarder_seats"]` and missed `company_politicians`
and `tr_companies`, which four of its six aggregates read.
**Three loaders outside this one also refresh part of the list**, because they change what
those matviews are built FROM, and skipping them is invisible:
```bash
npm run db:load:awarder-seats:pg:cloud # WHICH buyers are seated in a settlement (119+123+124)
npm run db:load:place-dim:pg:cloud # the place hero 123 STORES in its payload (119+123)
npm run db:load:tr:pg:cloud # company_politicians / tr_companies (122+124)
```
All three now do it themselves, and each gets only the matviews its input can actually move.
`place-dim` fingerprints the table either side of its rewrite and skips the refresh when the
rows are byte-identical, which is the usual case, so it stays cheap to run blind — it says
which it did. Without these, a standalone reload moves a buyer between settlements, blanks a
place hero, or leaves every MP-tied figure on the previous link set, everywhere on the site
**except the precomputed pages**, which keep serving the old attribution at a 200.
Two of those pairings are worth stating because they look wrong:
- **`awarder-seats` refreshes 124**, which has no settlement dimension — because one of its six
aggregates, `procurement_concentration`, resolves each row's `oblast` from `awarder_seats`
(86.6% of the `all` scope's 2,755 rows carry one).
- **`tr:pg` refreshes 122 and 124** but neither settlement matview. It TRUNCATEs and reloads
`company_politicians`, the politician↔company link set every MP-tied figure derives from —
so before this it was already leaving `/procurement/contractors`' MP-tied KPIs on the
previous vintage, with nothing red anywhere.
`nzok_activity_proc_periods` (migration 053, `db:load:nzok-activities:pg`) is the MONTHLY
activity panel — (period × entity × procedure), i.e. the annual matrix before it is folded
across months. It exists so the activity corpus can be joined to `nzok_hospital_payments`
period-exactly; the annual matrix cannot, and a facility with four months of payment rows
against a full year of cases reads as absurdly cheap.
Four things about this family are easy to get backwards:
- **`nzok_activities` is ANNUAL and must stay that way.** Ten call sites across five
migrations (053 ×5, 059 ×2, 054, 065, 075) read `max(period) FROM nzok_activities`
meaning "the latest ANNUAL matrix". Re-graining that table to months silently redefines
every one of them as a single month — the case-mix ratio would compare one month of cases
against a full year of payments, wrong by ~12x, at a 200, with every row count
reconciling. That is why the monthly panel is a separate relation.
- **The loader is a PER-YEAR MERGE, not a TRUNCATE.** `activities.json` is single-year by
construction, so a TRUNCATE made every run REPLACE the corpus: loading 2024 swapped 2025
out. The write is now `DELETE … WHERE EXTRACT(YEAR FROM period) = $1` + insert, and the
writer emits a per-year `activities-<year>.json` (gitignored) beside the latest-year
`activities.json`. Backfill with `npx tsx scripts/db/load_nzok_activities_pg.ts --year 2024`.
- **A backfill run must not touch the LATEST-year artifacts.** `activities_overview.json` is
COMMITTED, and the writer used to rewrite it on every run — so building 2024 replaced the
2025 overview with 2024 and would have shipped it, with nothing red. The writer now only
writes `activities.json` + the overview when the year being built is the newest on disk.
- **The panel records NO changelog of its own, on purpose.** It is the same corpus change
the `nzok_activities` batch already reports — same loader, same transaction, same source
file — so a second batch reports one event twice. Measured: recording it put 291,414 rows
into `ingest_first_seen` on a two-year load (94% of everything the one-day window holds)
and failed `recent_updates_plan.data.test.ts` at 308,980 rows scanned for a one-day
window. Summary mode does not help — it stops the day being itemised, but the branch
still scans.
- **Backfilling on the CLOUD needs an env var, not a flag.** `db:load:nzok-activities:pg:cloud`
is a nested npm script and the inner `npm run` swallows `--year 2024` (argv arrives as
`["2024"]`), so the loader would silently reload the LATEST year instead. Use
`NZOK_ACTIVITY_YEAR=2024 npm run db:load:nzok-activities:pg:cloud`.
The panel must sum to the annual matrix EXACTLY, per year — they are the same rows either
side of the fold, so drift means one aggregation lost or double-counted a period. The
loader asserts it in-transaction and `nzok_activity_proc_periods.data.test.ts` holds it
plus the annual-grain and entity-key invariants. Corpus floor: activity data begins
**Jan 2024**; nhif.bg serves nothing earlier.
`procurement_award_criteria()` (migration 164) is the ЗОП чл. 70 award-criterion lens behind
`/api/db/procurement-award-criteria` and the `AwardCriteriaTile` on `/procurement`. It rides the
TENDERS loader — `db:load:tenders:pg[:cloud]` applies 164 — because it reads only `tenders` and,
being a serving FUNCTION, carries no data that any other path would ship. Without an applier the
route degrades to `null` for ever on the serving side (the tile simply never appears) while
`db:refresh` FAILS at its final `test:data` step rather than skipping, since the gate's skip
predicate covers an empty corpus, not a missing function.
Four things about it are easy to get backwards:
- **Its no-call predicate is NOT 037's, and the two must not be "consolidated".**
`procurement_benchmarks` (037) reads `contracts.procurement_method`; this reads
`tenders.procedure_type`. The vocabularies intersect on 17 values but diverge (contracts also
carries the OCDS codes `open`/`limited`/`selective` and 45% NULL), and 037's list **includes**
„Покана до определени лица", which is 0.0% blank across 2,229 tenders — excluding it would drop
criterion-bearing rows from the denominator — while **omitting** three types that genuinely carry
none. `award_criteria.data.test.ts` asserts the divergence in both directions so a future
refactor cannot quietly adopt the contracts-side list.
- **`award_method` starts in 2020**, so `byYear` is floored at that year IN THE FUNCTION rather
than by each consumer: a 2018-2026 series renders a data-availability cliff as a policy change.
The rows dropped by the floor come back as `preCriterionTenders` so the omission stays visible.
- **`unknown` (not stated) and `other` (an unrecognised non-null value) are separate buckets.**
Seven `award_method` values exist today; an eighth arriving in a future ingest must surface as
`other` rather than merging into "not stated". The bucket set has ONE home —
`AWARD_CRITERION_BUCKETS` in `src/data/procurement/useAwardCriteria.ts` — which the tile draws
from and the data test asserts the SQL emits, so a seventh bucket cannot be counted in `total`
and silently never drawn.
- **It is deliberately NOT in the 124 precompute.** Measured: the first cut ran 188,591 buffers
with a temp spill (658 ms) because four scalar subqueries re-evaluated the CTE chain; referencing
each CTE once and adding `idx_tenders_award_criteria` (a covering index on `publication_date`
INCLUDE the three read columns) took it to **3,542 buffers / 233 ms** full-corpus and **845
buffers / 23 ms** on the default parliament window. That is inside the per-view budget, so the
precompute would be machinery for nothing — but re-measure before adding a column to the
function, since the index is what keeps the pass Index Only.
`cpv_catalog` (migration 121) is the same shape but rides the TENDERS loader: `db:load:tenders:pg`
applies it and calls `rebuild_cpv_catalog()` right after the corpus commits, so on the cloud side
`npm run db:load:tenders:pg:cloud` keeps it in step with no extra command. The only ordering that
matters is the FIRST deploy — the loader must run **before** the `deploy:db` that ships the route
reading it, because that route no longer degrades a missing table to an empty array (an empty CPV
picker served with a 200 is exactly the failure it was created to end). `cpv_catalog.data.test.ts`
fails on an empty or stale table.
`vote_item` / `vote_cast` / `mp_seat` / `party_dim` (migration 134,
`db:load:rollcall:pg`) are the National Assembly roll-call corpus — 16,741 items and 4M
casts loaded from `data/parliament/votes/sessions/*.json`, the same tree the derived JSON
metrics read. In `db:refresh`; on the cloud side:
```bash
npm run db:load:rollcall:pg:cloud # facts
npm run db:load:rollcall-derived:pg:cloud # the four precomputes — ALWAYS after facts
```
Re-run it after **every** `update-rollcall` ingest that added a session; the skill's Step 5b
says so, and nothing runs it automatically. Stage-merge, so it is safe against a live
database.
**Two things about this table are easy to get backwards and both fail silently.**
`vote_item` holds ALL 16,741 raw items — the 1,645 re-votes `dedupeRevotes` collapses carry
`superseded_by`, so **every aggregate needs `WHERE superseded_by IS NULL`** or it
over-counts by 9.8% at a 200. And party affiliation lives on `vote_cast.party_id`, the
affiliation AT CAST TIME: 179 of 2,366 seats change party mid-term, so grouping on
`mp_seat.party_id` instead compares those members against a group they had already left.
`rollcall.data.test.ts` holds both, plus the 26 recycled `mp_id`s that make `(ns, mp_id)`
the only safe key.
**`/api/db/session?date=` + `/api/db/session-casts?date=` + `/api/db/session-item?item=`**
replaced the day file `/votes/<date>` used to download whole — 482 KB on an average day,
4.97 MB on 2025-06-19, because that file carried every MP's vote on every item. The day
route is the agenda and the tallies (14 buffers); `session-casts` is that day's per-MP
matrix; `session-item` is one item's per-MP votes (~64 — the PK scan is 7, and the joins
onto `mp_seat` and `party_dim` for names and party labels are the rest). Neither of the
last two can be planned into the seq scan that costs 21,904 buffers, because both are
driven from an explicit key rather than a join on date.
⚠️ **The SCREEN uses the first TWO — `session-item` is the deep-link route, not the
screen's data path.** `useRollcallSession.tsx` fetches `session` + `session-casts` in one
pass, because `SessionScreen` renders the whole day's matrix rather than expanding one
item at a time. An earlier draft of this section named the pair as `session` +
`session-item`; that was a prediction, and it is not what shipped.
Unlike every matview, the day route does **not** filter `superseded_by`: it is the day's
RECORD rather than a statistic over it, and a motion put to the floor twice is a fact about
the day.
**[json-retirement-v2 Tier 1] THE RETIREMENT IS DONE — this section described it as pending
until 2026-08-29, and the sync guard is what settles it.** `SessionScreen` is on the routes
above; `parliament/votes/sessions/` and `parliament/votes/derived/per-mp/` are excluded
from every sync path (the `isExcluded` refusal, its `CHILD_EXCLUDES` twin, and the `-x`
regex in both `bucket:sync` and `bucket:sync:dry` — the three places
`bucket_sync_paths.test.ts` holds in lockstep); and the objects are **GONE, not frozen**:
`gsutil ls` returns "matched no objects" for both trees. `derived/dissents.json` and
`derived/party_pair_breaks.json` went with them.
**The FILES stay on disk, and that is the design** — `load_rollcall_pg.ts` reads them as
its load source and `scripts/prerender/votesFacts.ts` reads them for each `/votes/<date>`
body, both from `PROJECT_ROOT`, never the bucket. What was retired is the BUCKET COPY,
which nothing fetched.
⚠️ **Consequence for a publish: a scoped `bucket:sync:paths -- parliament/votes` pushes
NEITHER the new session files NOR the per-MP shards, and that is correct.** Both reach
production through `db:load:rollcall:pg:cloud` + `db:load:rollcall-derived:pg:cloud`. A
sync that reports 0 session objects after an ingest that wrote two is the guard working,
not a missed upload — measured 2026-08-29, when it read as a defect and was not one.
⚠️ **An exclusion FREEZES; only an `rm` retires. Keep this rule even though this family is
closed**, because it governs every tree in the frozen list below. `gsutil rsync -x`
excludes a match from DELETION as well as from upload ("not copied or deleted", per
gsutil's own help), and `syncPaths` passes `-x` together with `-d` — so once the exclusions
are in place no sync will ever remove those objects, and scoping a sync to the subtree is
refused by `isExcluded` by design. Removing them is an explicit
`gsutil -m rm -r gs://<bucket>/<tree>`, which is what closed this one.
**[2026-08-16] Under `parliament/`, EIGHT families are in that frozen state** — the eight
site-hygiene-v1 T6b excluded: `mp-connections/`, `official-connections/`, `by-id/` and five
`connections-*.json` singletons, **12,533 objects and 52.5 MB**. None has a reader. The three
retired MP↔company shard trees are NOT among them: `gsutil ls` returns "matched no objects" for
each, so their objects are gone (WHAT removed them is not recoverable — no lifecycle rule,
versioning Suspended).
⚠️ **`company-connections/` WAS the ninth and is now retired — this entry used to say the
decision was open.** It had a LIVE READER (the AI chat's `companyConnections` tool fetched it per
EIK) while being excluded from sync, so it was not merely frozen but answering from a
2026-07-29 snapshot at a 200 for weeks, and an `rm` would have 404'd a shipped feature. Closed
2026-08-16 by moving the reader to Postgres — migration 158 `company_political_links` behind
`/api/db/company-connections` — and then removing the 16,609 objects.
Three things about that move are worth carrying, because the next retirement will look like it:
- **The reader was invisible to the usual grep.** `src/ scripts/ functions/` reported zero
readers of the tree and was wrong: `ai/` is none of those. Any "is this readerless?" sweep
must include it.
- **It was NOT a like-for-like port, and the shard builder is not its specification.** The
shards matched a TR officer to a power roster BY NAME, kept the match only if the name
appeared in exactly one company, and graded it `medium`/`low` on whether the name had three
parts. 158 reads the gated `person_role` tr/ngo set — the Commerce Registry's own people
count per name fold (`tr_name_fold_people`, 148) decides, an unmeasured fold is REFUSED, and
there is no confidence grade. Per EIK that is sometimes fewer links; corpus-wide it is wider
on both arms (9,982 companies with a direct link vs 3,843; 26,047 answerable vs 19,232).
- **`scripts/declarations/tr/build_company_connections.ts` is DELETED (2026-08-16), and its
output is not.** The builder was retained for a day on the theory that it was the only path
able to reconstruct the removed objects. That was wrong twice over: it was a **git-tracked
source file** (so `git show <sha>:<path>` restores it — the versioning-Suspended argument
applies to bucket objects, which have no history, never to code in git), and its 19,232-file
output was still sitting on disk. Both call sites are gone (`scripts/declarations/index.ts`
phase 7, `scripts/declarations/tr/daily_refresh.ts`) along with the
`tr:build-company-connections` npm script, so a TR refresh no longer writes 83 MB nothing
reads.
⚠️ **Deleting the producer did NOT delete the tree, and the sync guards therefore STAY.**
`data/parliament/company-connections/` is gitignored, so its 19,232 files persist on every
machine that ever ran a `tr:daily-refresh` or `--declarations`, with nothing left to refresh
or remove them. Both halves in `scripts/bucket_sync_paths.ts` — the `isExcluded` branch and
the `CHILD_EXCLUDES` twin — plus the `-x` arms in `bucket:sync` / `bucket:sync:dry` are what
stop the next `bucket:sync:paths -- parliament` from re-creating the retired tree in the
bucket. Retiring a builder makes those guards MORE load-bearing, not less: while it ran, a
stray re-upload was merely pointless; now it would republish a permanently frozen snapshot.
"Nothing writes this any more, so the exclusion is dead config" is the one inference to
refuse here.
(Scoped to `parliament/` deliberately: `funds/` and `procurement/` are excluded too and far
larger — `funds/` alone is 182,075 objects and 560 MB — but those are PG-served by design, not
retired artifacts.)
The files stay on disk either way: they are the loader's input AND the prerender's fact
source (`scripts/prerender/votesFacts.ts`).
**`bill` (migration 136) rides the SAME loader** — `db:load:rollcall:pg` applies 136 and fills
it, so `db:load:rollcall:pg:cloud` carries it with no extra command. 504 rows, one per
(ns, title stem) that **reached a second reading** — NOT every bill the chamber saw, because
that is the set the /parliament tile counts and a table whose `count(*)` disagreed with the
number on the page is exactly the drift the hub plan exists to remove.
Two things about it are easy to get backwards. The stem split is **TypeScript**
(`secondReadingStem` in `scripts/parliament/derived/hub_stats.ts`), never a SQL regex — the
rule has a documented trap in it, since a title carrying „второ гласуване" in a PROCEDURAL
position is a FIRST reading, and matching the phrase instead of requiring the split to fire
counted 8 extra bills on the 52nd. And **`final_item` is always NULL**: it is where a
whole-bill adoption marker would live, and this corpus has none, so NULL means "not
derivable" rather than "not adopted". `bill_and_topics.data.test.ts` fails if either changes.
**`/api/db/vote-day-summary?ns=` + `/api/db/contested-votes?ns=`** retire the 8 MB
`topic_index.json` from the two consumers that fetched it whole — the `/votes` table (a topic
chip set and a four-segment outcome bar per day) and the contested-votes tile. 39 rows and
34 ms for the 52nd against an 8 MB download.
Both **filter `superseded_by`**, unlike the `session` route above, because both are
statistics over the day rather than the day's record — and the artifact they replace was
computed after `dedupeRevotes`. Both also keep `topic_index.json` as a **fallback**, so a
checkout without Postgres and a first cloud deploy before the loader runs still render;
the file therefore stays in `rebuildDerived`'s `--upload` list.
The outcome bucketing exists **twice** — in SQL in the route, and in `outcomeBucket()` /
`outcomeFor()` in TypeScript for the fallback — because a route cannot import TS.
`bill_and_topics.data.test.ts` re-derives every day's buckets from the session files and
fails on any disagreement; without it the two drift silently and the symptom is a bar of the
wrong colour. Measured coverage of that gate: the `abstain = cast` branch is the one clause
no item in the corpus reaches, so it rides on `outcomeFor()`'s definition alone.
**The derived half (migration 135, `db:load:rollcall-derived:pg`)** builds `mp_attendance`,
`party_cohesion`, `mp_dissent`, `mp_vote_norm` and `mp_similarity`, declared once in
`scripts/db/lib/rollcallMatviews.ts`. ~70 s locally, dominated by the quadratic
`mp_similarity` — measured on Cloud SQL 2026-08-06 at **801 s end to end, of which
`mp_similarity` alone was 744.5 s (12.4 min, 11x local)** on the then-current db-g1-small at 4,017,519
casts. ⚠️ **That is no longer the cost — re-measured 2026-08-27 on `db-perf-optimized-N-2`,
`mp_similarity` is 70.9 s**, i.e. parity with local (67 s) and 10.5× off the figure above. The
"budget a quarter of an hour and do not chain it behind anything urgent" advice was written for
the retired shared-core box; it now over-states the step by an order of magnitude. The facts
half (`db:load:rollcall:pg:cloud`) is dominated by ~2,900 single-row round trips
through the proxy before the COPY starts. `/api/db/mp-dissents` and
`/api/db/mp-similarity` read them and DEGRADE to an empty array on `42P01 · 42883 · 55000 ·
55P03 · 42501` — `55000` is in that set because a matview created `WITH NO DATA` RAISES
rather than returning zero rows, which is every first deploy; `57014` is deliberately out,
because it is the pool's own timeout and degrading there turns a 10 s failure into a 20 s
one. `mp_similarity` stores `dot` + `overlap`, NOT an agreement rate: the score consumers
are calibrated for is a cosine (`score = dot / (norm_a * norm_b)` via `mp_vote_norm`), and
substituting a rate would relabel "voting twins" sitewide.
`tender_dossier` + six sibling tables (migration 146, `db:load:tender-dossier:pg`) are
the ЦАИС ЕОП per-procedure capture — the attachment manifest, the parsed обявления, the
award-stage trail, contract items, buyer profiles and extracted document TEXT (never
document bytes; the 3.65 TB blob tier was dropped, so `/api/db/tender-document` mints a
signed redirect out to app.eop.bg instead). **`db:load:tender-dossier:pg` is 146's ONLY
applier** — the tenders loader below carries 147 but NOT 146 — so a database that never
ran this loader has the search index and none of the corpus.
Its input is `raw_data/procurement/eop_dossier.sqlite`, **gitignored**, written by
`npx tsx scripts/procurement/ingest_eop_dossier.ts` — a rate-limited ~26 h crawl of a
shared public register, so it is an operator action rather than a pipeline step
(`--probe` first to gauge the block state; `--to-year` for a stratified sample).
`scripts/procurement/ingest_eop_spec_text.ts` is the tier-B pass over the same store:
sign → fetch → extract text → **discard the bytes**. There is no `npm run` alias for
either, on purpose — neither belongs in a chain.
`tender_search_text` (migration 147, applied by `db:load:tender-dossier:pg` **and** by
`db:load:tenders:pg`) is that corpus's SEARCH index — the long „Кратко описание", every rendered
обявление and every extracted specification, folded per procedure so the words a
reader actually searches for are findable. `tenders.subject` is a 138-char headline,
so before this a procurement for nine kinds of coffee was searchable only as
„Доставка на хранителни продукти".
```bash
npm run db:load:tender-dossier:pg:cloud
```
**⚠️ The publish is SLICED, and the reason is a failure class no local run can
reproduce.** One transaction for the whole capture exceeds Cloud SQL's
`temp_file_limit` — **2,569,247 kB there, `-1` (unlimited) on the local docker
Postgres** — and the observed failure was not a clean `53400` either: the first
attempt died on the limit and the second dropped the connection outright
(`Connection terminated unexpectedly`). Local was green both times, at the full
50,283 procedures. Any future bulk publish on this instance wants the same check.
The loader now commits one slice of 5,000 tenderIds at a time. **The trade is
all-or-nothing PUBLICATION, not consistency**: every merge is УНП-scoped and no УНП
spans two slices (measured — 0 of 50,283 span more than one tenderId), so an
interrupted run leaves some procedures at the new vintage and the rest at the old,
never a half-written procedure and never an orphan row. **Re-running is the repair,
and it is idempotent** — so on a failed cloud run, re-run it rather than reasoning
about which slice died.
⚠️ **Slice on the tenderId RANGE, never on a УНП set filtered out of a full walk.**
The generators read the 1.7 GB SQLite capture, so a set filter still decompresses
all of it once per slice to discard ~90% — measured at **>13 min per slice against
~1.5 min** for the ranged read, i.e. the store walk, not the database, becomes the
cost. `EopDossierStore.iterate()` takes an optional `(after, upTo]` range that rides
its existing keyset index for exactly this.
**Three things about it invert the usual rules of this file, and the first is a
live-breaking deploy hazard rather than the customary staleness:**
- **The tenders search reads it UNCONDITIONALLY — there is no degrade.** `db_table.js`
emits `… FROM tender_search_text …` on every global search of the `tenders`
resource, and `db_routes.js`'s `badRequest()` rethrows a non-`DbRequestError`, so a
`42P01` is a **500** on every `/procurement/tenders` search box keystroke and every
`?q=` deep link. That is why **`load_tenders_pg.ts` applies 147**, not just the
dossier loader: `db:load:tender-dossier:pg` is a `REFRESH_EXCLUSIONS` member whose
input is a gitignored ~26 h crawl, so on most machines it has nothing to do. The
table now exists wherever `tenders` does, EMPTY if never filled — which is exactly
the pre-B3 behaviour, since the arm then adds no hits.
- **Its loader applies DDL BEFORE checking for the capture.** The order matters: with
the capture guard first, `db:load:tender-dossier:pg:cloud` on a machine without the
crawl printed „Nothing to load", applied no DDL and exited **0** — a deploy that
looks successful and creates nothing. 147 also carries the `app_readonly` GRANTs for
146's seven tables, which shipped with none — so applying it is also what would repair
a 146 that had already reached a serving database. (That line said "as of 2026-08-12
none has: the whole dossier family is local-only" — no longer true. Checked 2026-08-20:
Cloud SQL holds 50,283 dossiers and 418,841 documents, and the `app_readonly` GRANTs
ARE present there, so the hazard did not materialise. It stays worth knowing because it
is invisible locally — every loader and data test connects as the owner — and would
surface only as `/api/db` 42501 against a corpus whose row counts all reconcile.)
- **COVERAGE IS THE POINT, and it is PARTIAL.** 50,283 of 237,806 procedures (21.1%)
as of 2026-08-20 — it was 1,861 (0.78%) when this note was written, and the crawl has
been running since, so read the live figure from `/api/db/tender-search-coverage`
rather than from this line. The arm may therefore only ever **ADD** hits — it is one OR arm
beside buyer and subject, so a missing row can fail to add a hit and never suppress
one. It must never become a filter, a facet or a count: absence there would read as
„no such procedure" for the ~79% not yet crawled. `/api/db/tender-search-coverage`
returns the two live numbers any UI must cite before claiming it searched documents.
Two performance rules are load-bearing and both were measured, not reasoned:
- **The arm is `unp = ANY(ARRAY(SELECT …))`, never a correlated `EXISTS`.** An EXISTS
cannot participate in a BitmapOr, so it drags the WHOLE tender search onto a Seq
Scan — 37 ms (the pre-existing arms) → **6,617 ms**, a 178x regression on every
search, not only ones the dossier can answer. As an uncorrelated InitPlan array the
side lookup runs once and the key equality joins the BitmapOr: 21.5 ms. The inner
key is named `t_unp` so the correlated form cannot be written by accident — with two
columns called `unp`, `WHERE unp = unp` binds both sides to the inner scope, a
tautology matching every tender that raises nothing.
- **FTS only — deliberately NO gin_trgm index**, unlike every other fold here. The
`%>` word-similarity arm that 009 pairs with FTS on `subject_fold` recomputes
trigram sets over the whole body per row, and these bodies are documents: **0.073 ms
vs 13,490 ms** on the 1,861 rows the index then held. At corpus scale that arm is minutes, i.e. past the
10 s `statement_timeout`. The cost is real — no mid-word or near-spelling matching on
document text.
**Three routes read this family, and one of them has a cache rule that is easy to
undo.** `/api/db/tender-dossier` (the per-procedure page), `/api/db/tender-document`
(the signed-URL redirect) and `/api/db/tender-search-coverage` all ship with
`npm run deploy:db`.
**A loader that applies 147 MUST reach the serving database BEFORE the `deploy:db` that
ships these routes** — either `db:load:tenders:pg:cloud` or
`db:load:tender-dossier:pg:cloud`, both apply it. Deploying first is a **500** on every
`/procurement/tenders` search keystroke and every `?q=` deep link, not a narrower
answer, because `db_table.js` reads the table unconditionally and `badRequest()`
rethrows anything that is not a `DbRequestError`. The other two routes impose no
ordering of their own: `tender-dossier` catches 42P01 → `null` and
`tender-search-coverage` degrades to nulls.
`/api/db/tender-document` carries **`"Cache-Control": "no-store"` in `firebase.json`**,
and its entry must sit **AFTER** the `/api/db/**` rule — Firebase applies the **LAST**
matching header, so placed before it the `no-store` is dead config and the route
silently inherits `max-age=300, s-maxage=3600`. (That is not hypothetical: it shipped
that way and was caught by review. The global `**` rule sitting at index 0 with
`no-cache` is the independent proof of the direction — first-match-wins would make
every asset on the site uncacheable.) The register's blob URLs are S3v4-presigned with
a **30-minute expiry**, so a cached redirect is a link that works when it is written
and 403s later — the failure arrives long after the deploy that caused it. The
function cannot fix this itself: a hosting `headers` rule overrides a function-set
`Cache-Control`, which is why the header lives in `firebase.json` rather than beside
the route. `firebase_person_rewrite.test.ts` holds the order.
Related, from the same step and applying to **every** DbDataTable: the free-text arms
now escape `%` and `_`. They are LIKE wildcards, so before this any user typing one
turned the search into a scan of everything — measured, `50%_x` on tenders was
**11,672 ms** end to end, past the `statement_timeout`, of which 8,256 ms was
`buyer_fold ILIKE '%50%_x%'` — a pattern the trigram index can extract nothing usable
from, so it returns all 237,321 rows as candidates and the heap recheck then discards
236,496. Now 188 ms. The `searchFold`
arms escape in SQL rather than in JS, because their text is produced server-side by
`translit_bg_latin` and a JS-side escape would be undone by the transliteration.
**Also every DbDataTable: a free-text term shorter than `SEARCH_MIN_CHARS` (3) is
REFUSED with a 400, not answered.** pg_trgm extracts no trigram from a 1–2 character
pattern, so `col ILIKE '%q%'` stops being an index probe and becomes a full scan of the
gin index — measured on `contractor_rank` (432,228 rows) under a generic plan, `ст`
returned **all 432,959** index entries at 3,447 buffers / 359–490 ms, and again at 3,441
on the count aggregate, against 257 buffers for the 3-character `апи`. The 250 ms
debounce does not help: a user typing a five-letter term passes through both states.
Three things about it are easy to get backwards:
- **It refuses rather than returning zero rows, and that is the point.** `ст` has 6,462
matching rows; answering „0 изпълнители" at a 200 reads as "no such contractor exists"
with nothing saying the search never ran — the wrong-answer-at-200 shape this file
warns about everywhere else. A 400 cannot be misread.
- **It is TWO-SIDED, like `FIT_MIN_QUERY` / `useFundsFit`.** `DbDataTable` mirrors the
constant (`searchMinChars`, default 3), suppresses the term below it and renders a
„въведете поне 3 знака" hint, so no ordinary reader ever produces the 400 — what
reaches it is a hand-built request, an AI tool, or a bundle older than the deploy.
Shipping only the server half turns every 1–2 character keystroke, and every `?q=`
deep link shorter than the floor, into the destructive „Данните не можаха да се
заредят." panel on **23 of the 24** registry resources. Only `tenders` survives, via
its `unp` `searchPrefix` arm — anchored `LIKE 'q%'` over a btree is floored at 1,
because a short term there _widens_ the match instead of un-indexing the scan.
- **The length is counted in CHARACTERS, not UTF-16 code units**, on both sides
(`[...s.normalize("NFC")].length`). `show_trgm('👍👍')` is the **empty set** — four
code units, two characters, and no trigram at all, i.e. strictly worse than the `ст`
the floor exists for — so a `.length` check lets exactly the worst case through.
⚠️ **Measure a short term with `PREPARE`, never a psql literal.** A literal
constant-folds, the planner estimates through pg_trgm, picks the good plan and hides all
of this — the same trap the `OFFSET 0` search fence was written for.
`transport_facility_geo` (migration 132, `db:load:transport-facility-map:pg`) is the static
crosswalk behind the `/sector/transport/bodies#map` facility map — the same 073/074 family as the water
and МВР maps, in `db:refresh`, curated from `TRANSPORT_ENTITIES` with a Варна physical-facility
override for the two maritime bodies. First cloud deploy needs
`npm run db:load:transport-facility-map:pg:cloud` (after `db:load:awarder-seats:pg:cloud`)
**before** the `deploy:db` that ships the `/api/db/transport-facility-map` route — the route
degrades a missing migration to an empty map, so ordering is cosmetic, not breaking. Re-run it
whenever `TRANSPORT_ENTITIES` changes; `transport_facility_map.data.test.ts` fails on drift.
**The /sector/transport hub (docs/plans/transport-hub-v1.md) adds an ordering on top of
that.** 076 (the project map) gained a `scope` column, a consortium-member filter and a
`coverage` key, and 132 a member filter; both are FUNCTIONS, so only their loaders or
`apply_functions.ts` carry them. `db_routes.js` also gained `awarder-funds-rollup?basis=grant`
and two extra `awarder-group-top-contracts` fields. Ship in this order:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts 076_transport_project_map.sql 132_transport_facility_map.sql
npm run db:load:transport-project-map:pg:cloud && npm run db:load:transport-facility-map:pg:cloud
npx tsx scripts/funds/install_query.ts --cloud && npm run deploy:db
npm run bucket:sync:paths -- transport/derived/hub_stats.json procurement/derived/sector_stats.json transport/rail_subsidy.json transport/rail_ridership.json
npm run deploy && npm run deploy:db && SKIP_PREDEPLOY=1 npm run deploy
```
Every step degrades rather than breaks if skipped — the hub without its blob renders no KPI
band (a 404 maps to null), `/money#funds` says the grant basis is „not published yet", the
map loses its coverage line, the PSO captions vanish — which is exactly why a skipped step is
easy to miss. Check the live hub's KPI band and `/money#funds` after the last deploy.
`tr_company_place` (migration 133, `db:load:tr-company-place:pg`) is the company↔settlement
crosswalk behind the "фирми, регистрирани тук" tile on every governance dashboard and
`/api/db/place-companies`. It resolves the free-text `tr_companies.seat` to an EKATTE code
offline through the shared `EkatteResolver` (99.6% of seated companies; an ambiguous name
stays unresolved rather than guessing a village). In `db:refresh`; on the cloud side:
```bash
npm run db:load:tr-company-place:pg:cloud
```
**Its two ranking columns are DENORMALIZED, which makes its re-run trigger wider than its
input.** `money_eur` / `political_n` are copied from `company_public_money` (127) and
`company_politicians` (008) so the tile's top-N is an index scan — measured on Sofia's
110,474 companies, the live-join form of `place_companies()` ran **979 ms**, the stored form
**57 ms**, and prod is a shared database serving live traffic. So re-run it after `db:load:tr:pg` (which rebuilds BOTH
tr_companies and company_politicians) **and after any contracts / agri / funds reload**.
Skipping it is the usual silent shape: the tile keeps ranking and counting the previous
vintage at a 200. `tr_company_place.data.test.ts` fails on an empty/stale table, on either
denormalized column drifting from its source, and on the Sofia call exceeding 400 ms.
The route degrades a missing migration to an empty place, so first-deploy ordering is
cosmetic. The tile self-suppresses on `count === 0`, so a cloud database that never ran the
loader simply shows no tile rather than an empty one.
**It gained a THIRD denormalized column, `person_link_n`, and a THIRD trigger with it.**
That column counts DISTINCT public figures holding a gated registry role at each company —
`person_role(tr,ngo)` ⨝ `person(active, is_public_figure)` — and it is what
`place_mp_companies()` (151) filters on. So this loader must now also run **after
`db:resolve:persons`**, which is what rebuilds `person_role`; `refresh_coverage.test.ts`
carries the pair, and the local chain already ordered it that way.
⚠️ **`person_link_n` is NOT `political_n` widened, despite reading like it.** `political_n`
comes from `company_politicians` (008), whose two arms are derived in SQL from the gated person
layer and INNER-JOIN procurement money, so it is MONEY-restricted: **135 companies at 48
places** (re-measured 2026-08-21; it was 113 at 43 when this note was written, and the ratio,
not the figure, is the point). (**[2026-08-21]** it used to be built FROM
`mp_connected`/`pep_connected`; those two now read it —
`docs/plans/company-page-consolidation-v1.md` Tiers 4-6.) `person_link_n` is the whole gated
identity layer: **10,385 companies at 1,341 settlements / 260 municipalities**. Gating anything
on the first while the page filters on the second is a live defect that already happened once —
the tile's link to `/settlement/:id/companies` was hidden on 218 of 260 municipalities and
1,290 of 1,332 settlements that HAVE a page, and one place (ekatte 80217) had a political link
with an empty page. `place_companies()` therefore returns BOTH counts and the tile reads
`personLinkCount`.
`company_browse_table` (migration 188, `db:load:declarations:pg -- --resolve`) is the general
company registry browse behind `/companies` — one row per `tr_companies.uic`, the FULL Commerce
Registry corpus (~1.02M rows), superseding `official_companies` (178, now a tombstone). The old
page's whole population — "linked to a person in public life" — is one boolean column here
(`is_official_linked`, the `?political=1` filter `/governance/companies` 301s into), not a
separate matview and a separate page.
**`has_signal` is a DEFAULT-VIEW FLOOR, never a population cut.** Every company is in the
table; `db_table.js`'s `companies` resource applies `has_signal=true` as a client-side
`extraFilters` entry (the `ngos` resource's own pattern — no server-side `defaultFilters`,
since overriding one is more fragile than a client simply omitting it), so an exact name/EIK
search always finds a row and the "show all" toggle is just not sending the filter.
`has_signal := public_money_eur > 0 OR is_official_linked OR contract_count > 0 OR
entity_class IN ('ngo_assoc','ngo_found','chitalishte')` — roughly 10% of the corpus.
**FOUR loaders touch it, each for a different staleness reason, and none of the three refresh
call sites is redundant:**
```bash
npm run db:load:declarations:pg:cloud -- --resolve # the ONLY create path (person layer)
npm run db:load:graph:pg:cloud # public_money_eur (127)
npm run db:load:tr-company-place:pg:cloud # settlement/obshtina/oblast (133)
npm run db:load:pg:cloud # contractor_total_eur/contract_count (122)
```
Two of its LEFT JOIN sources are read through plpgsql wrapper functions
(`company_public_money_rows()`, `contractor_rank_all_rows()`) for the usual 077/145/178 reason:
`company_public_money` and `contractor_rank` are each DROPped and rebuilt by a DIFFERENT
loader, and a direct read (or a `LANGUAGE sql`/`BEGIN ATOMIC` wrapper, both parsed at CREATE)
would record a `pg_depend` edge that CASCADEs `company_browse_table` away on every one of
those OTHER loaders' runs, exiting 0 with nothing to show for it — caught live by
`migration_drop_dependents.data.test.ts` during this table's own construction, on
`contractor_rank`. `declaration_stake_company` (096) is read directly and is SANCTIONED in
that gate instead, the same shape 178 had: `load_declarations_pg.ts` applies 096 then 188 a
few statements later on the same path, so the CASCADE and the rebuild always travel together.
⚠️ **First-bootstrap ordering is BACKWARDS, and `load_tr_company_place_pg.ts` self-heals it.**
`db:refresh`'s chain runs `db:load:declarations:pg -- --resolve` — 188's only CREATE path —
**before** `db:load:graph:pg` and `db:load:tr-company-place:pg`, the loaders that build two of
its own dependencies. So on a genuinely fresh database the preflight there finds
`company_public_money`/`tr_company_place` missing and skips (skip-and-warn, not abort — same
as 178's own preflight always did), and nothing later in the chain would ever create the
matview, since the other three loaders only REFRESH an existing one
(`refreshMatviewConcurrently` returns `false` for an absent relation and does nothing).
`load_tr_company_place_pg.ts` runs LAST of the three dependency-providers in `db:refresh`'s
order, so it carries a FALLBACK CREATE: if the plain refresh reports the matview absent, it
re-checks all of 188's dependencies (now guaranteed present) and applies the migration itself.
A standalone `db:load:tr-company-place:pg:cloud` on a database that has never run the other two
still skip-and-warns, correctly.
**Place coverage is ~32%, same shape as `tr_company_place` above and for the same reason** —
`oblast_name` is a NAME (133 has no oblast CODE), NULL for the ~68% of the corpus with no
resolved seat, and the `?oblast=` picker facets that same column so counts stay exact. Never
read `oblast_name IS NOT NULL` as "this company has no place" — it means "the seat did not
resolve," which is the majority case for the wide corpus this table now covers (178's
officials-linked population had ~60% place coverage; the full registry does not).
The route (`companies` DbDataTable resource) has no `missingMigration` degrade, so a
`deploy:db` shipping it before this loader first reaches the target 500s `/companies`
outright — same ordering rule as `cpv_catalog` / `contractor_rank`.
⚠️⚠️ **THAT ORDERING HAZARD IS NOT HYPOTHETICAL — IT HAPPENED. ✅ RESOLVED 2026-08-27; the
rest of this section is kept as the worked example, not as a live incident.** Verified that day
against the serving database: `to_regclass('company_browse_table')` resolves, the matview holds
**1,022,592 rows**, and `/api/db/table?q={"resource":"companies"}` returns real rows at **200**.
`db:check-cloud` also reports Cloud SQL matching local on all 1,181 schema-owned objects, which
it could not do while 188 was absent.
What it looked like while it was live, measured 2026-08-26: `to_regclass('company_browse_table')`
was **NULL** — 188 had never been applied to Cloud SQL at all, which still carried 178's
`official_companies` — while the deployed function DID know the `companies` resource. So
`/api/db/table?q={"resource":"companies"}` returned **500 `{"error":"db error"}`** on every
request. `/companies` itself answers 200 because that is only the SPA shell; the table inside it
renders the „Данните не можаха да се заредят." panel. **A 200 on the page is not evidence the
resource works** — query the API directly, which is the check that found this.
Applying 188 there is the fix and needs nothing else first — it defines both plpgsql wrappers
(`company_public_money_rows`, `contractor_rank_all_rows`) itself, and the second does NOT yet
exist on cloud, so no other file can supply it. Verified 2026-08-26 before recommending it:
all six inputs are present and populated, `app_readonly` exists (188's GRANT is `pg_roles`-guarded
regardless), and the `official_companies` it DROPs has no reader anywhere — none in `functions/`,
`ai/` or `src/` outside comments, no `pg_depend` dependent on cloud, and the deployed API answers
`unknown resource` for it. Cloud inputs are the same vintage as local (`tr_companies` 1,022,592,
`tr_company_place` 327,161, `contractor_rank` 432,605, `company_public_money` 81,464 — all
identical; `person_role` at tr/ngo is 199,531 against 201,082, the known resolve-history drift).
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts 188_company_browse.sql
```
⚠️ **OFF-PEAK.** This is a `CREATE MATERIALIZED VIEW` over 1,022,592 `tr_companies` rows joined
to five more relations, plus ~15 index builds, on a box serving live traffic; the duration is
**unmeasured on Cloud SQL**. The usual „re-applying 188 blanks /companies" warning does NOT apply
here — the page is already down, so this can only improve it. Afterwards, 193 is a no-op (188
carries the same four indexes) but is worth running to confirm, and the check that matters is the
API, not the page:
```bash
curl -s "https://electionsbg.com/api/db/table?q=%7B%22resource%22%3A%22companies%22%2C%22limit%22%3A1%7D"
``` `company_browse.data.test.ts`
fails on an empty/stale table, on `is_official_linked` disagreeing with an independent recount
of 178's old two-arm union, and on either plpgsql wrapper drifting from its source.
**Every sortable numeric column needs a `has_signal`-LEADING index, and three of the four were
missing one.** `has_signal` is a client-side `extraFilters` entry, so it is in the WHERE of every
default-view query — and an index that sorts correctly without leading on it still produces a
clean „Index Scan" with no Sort node while reading the whole 1.02M-row table and discarding 90.3%
as a row Filter. Measured 2026-08-26 at `OFFSET 20000`, which is a page a crawler reaches:
| column | before | after |
| ------ | ------ | ----- |
| `public_money_eur` (had one) | 206 | 206 |
| `contractor_total_eur` | **1,031,019** | 20,112 |
| `contract_count` | **1,030,451** | 20,068 |
| `person_count` | 23,331 (full Sort) | 20,122 |
⚠️ **The tiebreak must be `uic`, and it is not a free choice** — `buildOrder` in `db_table.js`
appends exactly one, `r.columns.key ? "key" : r.select[0]`, and Postgres compares pathkeys
structurally, so `(…, name, uic)` serves nothing. Nor does a plain `DESC` btree, which is NULLS
FIRST while the engine emits `DESC NULLS LAST`.
⚠️ **Ship them with 193, NEVER by re-applying 188.** 188 opens with `DROP MATERIALIZED VIEW IF
EXISTS company_browse_table`, so re-applying it rebuilds 1,022,592 rows while `/companies` — a
resource with no `missingMigration` degrade — 500s throughout. That is the 122 hazard exactly.
193 is the same statements with `IF NOT EXISTS`, and both files carry them so a fresh clone and a
warm database converge:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts 193_company_browse_sort_indexes.sql
```
Cloud SQL needs it by hand and nothing runs it; the four `CREATE INDEX`es take a ShareLock, so
`/companies` reads through them and only a concurrent write would block. The gate's arm is a
**buffer ceiling**, not a sort-node ban: neither „is it an Index Scan" nor „is there a Sort" can
see this defect, since the broken plan satisfies both.
### The two MP↔company serving functions, and the shard families they retired
`mp_tr_roles(mp_id)` (migration 150, `/api/db/mp-management`) and
`place_mp_companies(ekatte, obshtina, page, pageSize)` (migration 151,
`/api/db/place-mp-companies`) replaced three bucket-served shard families —
`parliament/mp-management/` (896 files), `companies-by-ekatte/` (376) and
`companies-by-obshtina/` (270). Plan: `docs/plans/mp-tr-edges-pg-v1.md`, revised by
`data-hub-lateral-edges-v1.md` §11.10.
**Neither is a table and neither has a loader** — they are "applied, never loaded", and their
appliers are chosen so a corpus reload carries them: **150 rides `db:resolve:persons`**
(SCHEMA_FILES, after 148) and **151 rides `db:load:tr-company-place:pg`** (after 133, which
owns the column it reads). Both were written without an applier first, which is the migration-144
defect: `db:refresh` then fails at its final `test:data` step on any database not hand-patched,
and no cloud path ships them at all. Ship a body change on its own with the usual hatch:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts \
148_person_company_basis.sql 150_mp_tr_roles.sql 151_place_mp_companies.sql
```
**148 must be in that command and first**: 150's body joins `person_company_bridge_a`, and a
`LANGUAGE sql` body is validated at CREATE time, so applying 150 to a database without the view
fails the whole file with 42P01 — the 081→082 trap again.
Cloud side, in order — nothing here is automatic:
```bash
npm run db:resolve:persons:cloud # applies 150
npm run db:load:graph:pg:cloud # company_public_money → money_eur, 151's sort key
npm run db:load:tr-company-place:pg:cloud # applies 133 + 151, fills person_link_n
npm run deploy:db # the two routes
npm run deploy # ⚠️ NOT optional — see below
```
⚠️ **`npm run deploy` is the step that actually retires the shards, and leaving it out inverts
the point of the whole change.** The hooks were repointed in the bundle; until hosting ships
that bundle, production keeps fetching the bucket copies and keeps publishing the 410
attributions the person layer refuses. `deploy:db` alone changes nothing a reader sees.
**[2026-08-16] This has since been deployed AND the three trees removed from the bucket**
(`gsutil ls` → "matched no objects"), so the hazard is historical — kept because the ordering
rule it teaches is not.
**Ordering `deploy:db` before the loaders is cosmetic but not free.** Both routes degrade a
missing migration — and they use the NON-logging `missingMigration` variant, unlike the
`psp:`/`pp:` routes, so a premature deploy reads as „this MP holds no registry roles" and „no
companies here" indefinitely, with nothing in the logs and nothing red.
**Both refuse a shared name rather than grading it, and that is the whole point.** Neither
function re-implements a guard: both read `person_role` at source tr/ngo, the set
`resolve_persons` mints through Bridge A/B and gates on `tr_name_fold_people` (148). So a name
the Commerce Registry records for more than one person is refused — the shards published 410
such (MP, company) attributions, of 2,014, while `/persons` and the `/person` profile had
already stopped. `MpManagementRoles` and `PersonCompanies` are the two
surfaces that list one person's companies — mutually exclusive branches of `CandidateScreen`
rather than literally one page, but a reader reaching the same person by either route must not
be told two different things. That is why 150 derives from `person_role` rather than minting its
own set, and why both render the basis from the same `person_company_bridge_a` view.
⚠️ **`COMMON_NAME_TR_ROWS = 11` was DELETED, not ported.** It counted officer ROWS as a proxy
for "is this name one person", and it is wrong in both directions — it dropped a rare-name MP's
whole medium set behind one busy registered agent, and passed a name held by two people with six
companies each. Do not reintroduce a row-count heuristic beside a people count.
⚠️ **[2026-08-16] Those bucket objects are GONE — this paragraph used to say they were not.**
`gsutil ls` returns "matched no objects" for all three trees, so the `rm` the exclusion site
described as pending has been run. The mechanism it documents is still true and still the reason
an exclusion is never a retirement: `gsutil rsync -x` excludes a match from DELETION as well as
upload, and `syncPaths` passes `-x` with `-d`, so an exclusion FREEZES a tree and removing it is
always a separate operator action. What is frozen NOW is listed at the exclusion site in
`scripts/bucket_sync_paths.ts`.
The state-budget corpus (migrations 152 + 153, `db:load:budget:pg`) is the КФП execution feed,
its per-fiscal-year roll-up, the admin/programme reconciliation, personnel, COFOG and the
budget-document index. Plan: `docs/plans/budget-hub-v1.md` T1.
```bash
npm run db:load:budget:pg:cloud
```
**It is a `REFRESH_EXCLUSIONS` member, and the axis is the INPUT, not the cost.** The admin and
programme grain lives in `data/budget/reconciliation/` and `data/budget/ministries/`, both
gitignored (bulky regenerable shards, bucket-shipped only) — measured, `git ls-files` returns 0
for each against 24 and 55 files on a machine that has run the pipeline. The load itself is
~2 MB and seconds. Run it from the `update-budget` path, by hand.
Four things about it are easy to get backwards:
- **It is NOT the only applier of 152/153.** `db:load:budget-muni:pg` is in the chain and
applies `152 → 153 → 154 → 157 → 155` in that order (155's `LANGUAGE sql` bodies are
validated at CREATE, so anything else 42P01s and rolls the file back). A fresh clone
therefore HAS the budget tables, EMPTY — this excluded loader is what fills them, both the
gitignored admin/programme grain and the committed КФП half. `budget_pg_roundtrip.data.test.ts`
skips on that empty state; its `pg_class` probe covers the narrower case of a hand-built or
partial database with no tables at all. `refresh_coverage.test.ts` holds the in-chain applier
and its order. (This bullet claimed the opposite until 2026-08-15, having been written before
T2/T3 shipped the second applier.)
- **The merges REFUSE a >5% shrink** (`--allow-shrink` overrides). `mergeFromStage`'s delete is
an unscoped anti-join and its parity guard compares counts AFTER that delete, so an empty
stage wipes the corpus and passes 0 == 0. Measured: an empty admin stage removes 55 nodes,
873 facts (via `ON DELETE CASCADE`, so they are not even in the DELETE's count) and 727
programme rows, with the loader exiting 0.
- **`budget_program_fact.node_id` is the OWNING SPENDING UNIT, and `program_code` is the
programme.** `by-program.json` keys its rows on a `prog-…` slug and names no owner — 0 of 86
join `budget_admin_node` — so the owner is recovered from `data/budget/ministries/`
(727/727 rows, 124 programmes, 0 ambiguous). NULL means that gitignored tree was absent, not
that the programme has no owner.
- **`budget_cofog` is a DIFFERENT CORPUS from the rest** — Eurostat `gov_10a_exp`, sector S13
general government, which includes municipalities and the social funds. It is NOT a
decomposition of the КФП state-budget expenditure it will be rendered beside.
`council_muni` / `council_muni_code` / `council_resolution` / `council_vote` (migration 160,
`db:load:council:pg`) are the municipal-council corpus behind the My-Area council tile, the
My-Area alerts feed and the AI chat's `councilResolutions` tool. In `db:refresh` after
`db:load:official-candidate-links:pg`; on the cloud side:
```bash
npm run db:load:council:pg:cloud
```
**Its input is the DURABLE per-resolution shard tree** (`data/council/<code>/<YYYY>/<id>.json`,
4,813 files as of 2026-08-25, COMMITTED) — deliberately not `index.json` (capped at 200 rows per
município, and eleven of sixteen exceed it) nor `votes/*.json` (rebuilt from that capped,
`perCouncillor`-stripped index until 2026-08-16, which left 530 resolutions and 10,754 named-vote
rows on disk and unserved). The tree being committed is why this loader is in the chain proper
rather than `REFRESH_EXCLUSIONS`.
Five things about it are easy to get backwards:
- **It is UPSERT-ONLY and must stay that way.** A council resolution is a permanent public record,
so a scrape that misses a protocol — or a parser regression on one município — must not erase
history; `last_seen_at` records the absence instead. `mergeFromStage` is therefore NOT used (it
couples an unscoped anti-join DELETE to the upsert). The one deletion the loader performs is a
targeted purge of `norm_key ~ VOTE_LABEL_SOURCE`, a provable invariant violation — 840 PER32 rows
where the parser absorbed the vote label into the councillor's name.
- **`roster_code` is NOT `obshtina_code`, and conflating them attaches votes to the WRONG council.**
This corpus's `BGS01` is Бургас, but `official_roster` holds a DIFFERENT município under `BGS01`
(28 councillors, disjoint names) — Burgas city is roster `BGS04`. Sofia is council `SOF` → roster
`SFO_CITY`. The loader derives it via `rosterShardForObshtina()` and **refuses** rather than
guessing when a council resolves to more than one roster.
- **The name fold has ONE definition, `councilNameKey()` in `scripts/council/lib/tally.ts`**, and
both sides must use it. It was briefly written twice — TS on the vote side, `lower(split_part(…))`
in SQL on the roster side — and the two diverged on `й`→`и` (NFD) and on hyphens, costing 4,899 of
28,214 votes their attribution AND evaluating the "refuse a shared name" guard over a different
equivalence class than the join used, which can attach a vote to the wrong person (that count is
the 2026-08 corpus; it is 46,121 now). Attribution is **93.8%** — 43,261 of 46,121, measured
2026-08-25; it was 94.1% of 28,214 when the single fold landed, and the gate's floor is 90%
rather than a ratchet on either. A run reporting ~77% means the folds have drifted apart again.
- ⚠️ **`meta.resolutionCount` in `index.json` counts the SHARD TREE, and it had two definitions
until 2026-08-25.** `mergeMuniResult` counted `|index-window ∪ durable ∪ this scrape|` while
`rebuildShardsFromDurable` counted `|durable|`. They were equal on every município until
`cbbcd220e4` purged 84 phantom resolutions — deleting their shards and leaving their rows in
`resolutionsByObshtina` — after which the value ALTERNATED run to run (RSE01 published 507
against 426 shards) and a scrape that wrote zero shards still moved it. Nothing reader-facing
was wrong: this loader takes `resolution_count` from the tree (`rows.length`) and reads
`index.json` only for `meta.name` / `meta.lastIngest`, and the whole `council/` tree is excluded
from bucket sync. What DID read the bad number was the `update-council-minutes` verification
step and the ingest ledger. Repair with `npm run council:rebuild-shards` — which now prunes the
ROWS as well as resyncing the count, the half whose absence made the 2026-08-22 purge look
complete when it was not. `scripts/council/lib/index_corpus.test.ts` gates the invariant in both
directions.
- **`db:resolve:persons` nulls `council_vote.person_id` table-wide** (ON DELETE SET NULL, because
`person_id` is a positional ordinal and the resolver does DELETE + re-COPY), so this loader must
run AFTER it and is what re-attaches attribution — the declarations `--resolve` trap, one table
over. The loader carries a 90% attribution floor that refuses rather than republishing a corpus
with its attributions wiped (`--allow-attribution-drop` overrides); its roster input,
`official_roster`, has exactly one writer, `db:load:ngo-board-links`, which degrades to NULL
`obshtina` on a clone with no municipal shards.
`municipal_fiscal` (migration 149, `db:load:municipal-fiscal:pg`) is the per-município
quarterly financial-indicators corpus (ЗПФ чл. 130г ал. 2) — 265 общини × quarter, carrying
the three liability stocks Bulgarian public finance distinguishes and the site previously
collapsed into one. In `db:refresh` right after `db:load:place-dim:pg`; on the cloud side:
```bash
npm run db:load:municipal-fiscal:pg:cloud
```
**`db:load:place-dim:pg:cloud` is a hard PREREQUISITE, not merely a trigger.** Two of 149's
three serving functions JOIN `place_dim`, and a `LANGUAGE sql` body is validated at CREATE
time —
so applying the migration to a database without it raises `42P01`, and because `exec()`
sends the file as ONE transaction the target gets **no `municipal_fiscal` table at all**,
not merely unlabelled rows. `refresh_coverage.test.ts` carries the pair for the local chain;
nothing covers the cloud side.
Five things about it are easy to get backwards:
- **It also owns `obshtina_population`**, the NSI Census 2021 per-município
denominator behind the per-resident default sort on
`/governance/municipal-finance` and the rank on every governance dashboard.
Built from the committed `data/census_2021.json`, so it needs no fetch — but
it resolves Sofia's census code `SOF46` through **`place_dim.price_code`**,
which makes `db:load:place-dim:pg:cloud` a prerequisite for its CONTENT and
not merely for the JOIN to compile. The loader REFUSES rather than degrades on
an incomplete match: an unresolved município would sort last on the page that
exists to surface it. Related, and the reason 149 declares that table at the
TOP of the file: a `LANGUAGE sql` body is validated at CREATE time, so a
function created above the table it reads raises 42703/42P01 and — exec()
sending the file as one transaction — rolls the whole migration back on every
database that does not already have it, i.e. everywhere except the machine
that wrote it.
- **The corpus has TWO consumers, and the loader is only one of them.** It also feeds three
NATIONAL series in `data/macro.json` — `municipalCommitments` /
`municipalExpenseObligations` / `municipalArrears`, behind the `/indicators/fiscal`
commitments tile — built by `scripts/macro/municipal_stocks.ts`. `fetch_eurostat.ts` is
the durable writer (it pushes them onto `CURATED_INDICATORS`, so a macro refresh carries
them), and `npm run macro:municipal-stocks` folds them into an existing `macro.json` when
the corpus moved and nothing else did. A new quarter that reaches only the PG loader
leaves `/indicators/fiscal` a quarter behind at a 200 — `macro.json` is bucket-served from
the committed file, so nothing about the database refreshes it. Note the assembler's
10%-shrink regression gate exempts these three by RATIO (`MAY_SHRINK`) but not at zero:
losing a quarter is designed behaviour here — МФ freezes a column and the ingest withholds
it rather than carrying it forward — while dropping to no points at all still aborts.
- **The corpus is the LOADER'S input, and the fetch half is manual.**
`data/budget/municipal_fiscal/*.json` is committed and the loader is pure-load (works on a
fresh clone, no network). What produces it is
`scripts/budget/municipal_fiscal/ingest.ts`, which reads the **gitignored**
`data/_cache/minfin_municipal_fiscal/` workbooks — minfin.bg serves an interactive
Cloudflare Turnstile, so downloading them is an operator action. See that directory's
README for the exact filenames.
- **It is NOT bucket-synced, and that took FOUR exclusions, not one.** There are three
independent upload paths into `data/budget/`, and an exclusion on any one of them leaves
the other two shipping the corpus:
1. `bucket:sync` / `bucket:sync:dry` — a `gsutil rsync -x` regex; needs its own
`^budget/municipal_fiscal/.*` arm beside the `funds/` and `opencalls/` ones.
2. `bucket:sync:paths -- budget` — `isExcluded` guards only the top-level ARGUMENT, so the
scoped push walks into the subtree; that needs the `CHILD_EXCLUDES` twin.
3. `npm run budget:ingest -- --upload` — the `update-budget` step, which `gsutil cp -r`s
the whole tree. `gsutil cp` has no `-x`, so `uploadTextTree` now expands the top level
itself and filters through the SAME `isExcluded`.
All four live in `scripts/bucket_sync_paths.ts` + `scripts/lib/upload.ts` and read one
definition. Removing any of them puts a second copy of a PG load source on a bucket
nothing reads — the shape that once pushed ~16.8k company-connection shards.
- **`meets_threshold` is NULL on almost every row WITHOUT an official verdict, and that is
correct.** The statute has **SEVEN** criteria, not six — МФ's year-end-anchored releases
enumerate them 1..7 and `scripts/budget/municipal_fiscal/criteria.ts` reads them, so on the
year-ends those releases cover (2018, 2020, 2021, 2023, 2024 today) the verdict is the
ministry's and is decisive in BOTH directions. Elsewhere only three of the seven are
computable from this source — т. 1 needs debt SERVICE
(the workbook publishes only the debt STOCK), т. 5 three consecutive years, т. 6 the
national collection mean. `criteria_evaluable` records which were checkable, so „2 met"
cannot be read as „2 of 6". A verdict is TRUE only when three are actually met (decisive
by monotonicity, whatever the unchecked ones say) and FALSE only when all six were
evaluable. Measured 2026-08-12: 5 municipalities decisive TRUE, and all 5 are
independently on the official чл. 130д recovery list.
`interreg_programmes` / `interreg_operations` / `interreg_partners` (migration 137,
`db:load:interreg:pg`) are the Interreg cross-border corpus — 1,958 operations, 12,015
partnerships, 1,494 Bulgarian partner rows, €401.77m — from keep.eu (INTERACT), which is
where Interreg lives because it runs on **Jems** and not on ИСУН. That is why
`fund_projects` holds zero Interreg rows: the gap is a system boundary, not a filter.
In `db:refresh`; on the cloud side:
```bash
npm run db:load:interreg:pg:cloud
```
**Its re-run triggers are wider than its own source**, and two of the three are the
non-obvious ones. Place resolution happens IN THE LOADER (Tier L1 reads `awarder_seats`,
L2 reads `tr_company_place`), so 199 of the 1,469 placed rows depend on the _content_ of
those two tables and not merely on their existence. Re-run it after:
- a keep.eu re-import (`npm run funds:crawl-interreg -- --full` then `funds:ingest-interreg`);
- **`db:load:awarder-seats:pg:cloud`** — 158 placements;
- **`db:load:tr-company-place:pg:cloud`** — 41 placements.
**`place_dim` is a hard PREREQUISITE, not a trigger, and it is the one that bites on a
first cloud deploy.** The loader reads it for the obshtina/oblast label on every placed row,
including the `nuts3` column — so a database whose `place_dim` predates 117's `nuts3` fails
with `42703 column "nuts3" does not exist` AFTER applying 137/138/139 and before writing a
single row. Measured on the 2026-08-08 production deploy, which is exactly how this note
came to exist: prod's `place_dim` had the right ROW COUNT (5,720, matching local) and the
wrong columns, so a count-based preflight passed it.
```bash
npm run db:load:place-dim:pg:cloud # BEFORE db:load:interreg:pg:cloud
```
Budget for it: the rows change, so its fingerprint check fires the refresh it guards and
rebuilds `procurement_settlement_rank`, `procurement_geo_payloads` and
`procurement_settlement_payloads` — 46 s locally and 87 s on Cloud SQL (measured 2026-08-22 on
`db-perf-optimized-N-2`; the old "minutes on a db-g1-small" figure is superseded), and a plain
`REFRESH` takes an AccessExclusiveLock, so `/procurement/by-settlement` and every settlement
page block for the duration.
Skipping it after either crosswalk moves is the usual silent shape: the corpus keeps the
previous placements at a 200. `refresh_coverage.test.ts` carries both as `ORDER_PAIRS`
entries, which covers the local chain; nothing covers the cloud side.
The loader **refuses rather than degrades** on an absent or empty crosswalk, and on a
placement share below 90%. That is not defensive padding: all three tables are
stage-merged, so the upsert SETs `ekatte`/`obshtina`/`oblast`/`place_basis` from the stage
like any other column — a run with a broken cascade writes NULL over good placements and
`mergeFromStage`'s parity guard **passes**, because it counts rows, not places. Measured:
empty crosswalks take placed rows 1,469 → 1,270 with the guard green.
**Never build the stage from one programme.** `ingest.ts`'s `--programme` is a debugging
filter that refuses to write, and the loader has no such flag at all, because
`stageDeleteSql` is an unscoped anti-join: a partial stage deletes every other programme's
operations, `ON DELETE CASCADE` takes their partners, and the parity guard passes again.
**The corpus is only half of a publish.** Loading `interreg_partners` moves nothing on the
money surfaces by itself, because `company_public_money` (127) — the ONE reusable per-EIK
broad-money basis — grew an Interreg arm, and the GRAPH loader is 127's only applier and
refresher. So the cloud order is:
```bash
npm run db:load:interreg:pg:cloud # the corpus
npm run db:load:graph:pg:cloud # 127's 4th arm reads it; nothing else applies 127
npm run db:load:tr-company-place:pg:cloud # money_eur is denormalized from 127
```
Skipping the second leaves every Interreg euro out of `/connections`, out of the governance
"фирми, регистрирани тук" ranking and out of `/company/:eik`'s money, with the corpus itself
fully loaded and every row count reconciling.
**That arm closes a dependency CYCLE, and `db:refresh` therefore runs one vintage behind.**
The local chain is graph (47) → tr-company-place (49) → interreg (50), and the order is
forced from both ends: `tr_company_place` denormalizes 127, while the Interreg place cascade
reads `tr_company_place`'s EKATTE. Adding the arm made 127 read `interreg_partners`, so no
`ORDER_PAIRS` entry can express it — the three form a loop. Consequence, stated rather than
hidden: the Interreg arm is built from the PREVIOUS run's corpus, and a first-ever Interreg
load contributes no money until graph runs again. Re-run `db:load:graph:pg` after an Interreg
reload to close it in one pass. (`load_graph_pg.ts` applies 137's DDL before 127 so a
database that never loaded Interreg still builds the matview — see 127's header for why a
`to_regclass` branch was the wrong fix.)
**The serving layer is "applied, never loaded" and needs its own command.** 138
(`interreg_by_place` / `interreg_by_eik` / `interreg_overview` / `interreg_operation` /
`search_interreg_operations`) and 139 (`funds_muni_combined_v` + the per-capita ranking) are
FUNCTIONS and a VIEW — they carry no data, and `deploy:db` ships `functions/` code, which is
a different thing from a Postgres function. `db:load:interreg:pg[:cloud]` applies both, so a
corpus reload carries them; a function-body fix on its own does not wait for one:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts \
138_interreg_serving.sql 139_funds_muni_combined.sql
```
**`/funds/interreg/**`is a page URL served by the`db` function** (`functions/spa_page.js`),
like `/funds/contract/**`and`/company/**` — so the same ordering rule applies and it is the
one that breaks a working page: **`npm run deploy:db`BEFORE`npm run deploy`**. Hosting
first points all ~1,954 operation URLs at a function with no handler for them. They are not
prerendered and carry no sitemap `<loc>`; without the function they serve the homepage's
`<title>` and canonical, which is the duplicate-content shape this handler exists to end.
`funds_fit_procedure(code)` (also 143) is the base-rate card on `/funds/procedure/:code` —
beneficiaries, the grant MEDIAN with its quartiles, org-kind mix, and the disbursed share. A PK
seek on `ux_fund_fit_code` (0.05 ms), which is the whole reason `fund_fit` is materialised. It
ships with the same loader.
**The reference price („5% от медианния грант тук = €X") is arithmetic, and the split is
deliberate.** The route returns the MEDIAN and nothing derived from it; the division happens
client-side, in the open, at three percentages so no single one reads as an endorsement. There is
no consultancy-fee corpus in Bulgaria, so „a fair fee is Y" is a verdict we cannot support and is
explicitly out (plan §8.4-4) — what we publish is the denominator, and the card says so. The
measured demand is Appendix A category D: „поискаха ми 4000 € и 5% от сумата — това реални цифри
ли са?".
**Nothing on that card is an approval rate.** `paid_project_count` is DISBURSEMENT — ИСУН
publishes no rejected applications, so the denominator for „одобрен ли е бил" does not exist.
`ProcedureBaseRates.test.tsx` fails if the word „одобрен" appears anywhere except inside the
disclaimer that rules it out.
`funds_hub_stats_cache` + `funds_hub_stats()` (migration 145) is the `/funds` HUB's one stat
call — the figures on its tile grid, behind `/api/db/funds-hub-stats`. **`db:load:funds-fit:pg`
applies and refreshes it**, and that placement is not where it looks like it belongs: its
primary input is the funds corpus, but `CREATE MATERIALIZED VIEW` resolves its query at
creation and 145 needs `canon_oblast` — which 143 defines, applied by that same loader one
`db:refresh` step after `db:load:funds:pg`. Applied from the funds loader it fails with
`function canon_oblast(text) does not exist` and rolls back a 57-step chain at step 10.
```bash
npm run db:load:funds-fit:pg:cloud
```
**It is materialised because the live aggregate cannot be served.** Measured: 18,855 buffers
with a spill to temp, against the ~2,000 the dashboard-hub skill allows for a call every view
makes. The seek is 40.
Three things about it are easy to get backwards:
- **Its Interreg arm closes a CYCLE, so it is refreshed from BOTH ends.** 145 reads
`interreg_operations`/`interreg_partners` (step 52) while its primary input is at step 10/11,
so `db:load:funds-fit:pg` refreshes it with the PREVIOUS Interreg vintage and
`db:load:interreg:pg` refreshes it again. Consequence, stated rather than hidden: a
first-ever run is populated by step 52, and after any complete `db:refresh` the numbers are
current. Skipping the interreg-side refresh leaves the hub's Interreg tile a vintage behind
at a 200 with every row count reconciling.
- **Every key names its BASIS, and that is load-bearing rather than verbose.**
`absorptionPctOfGrant` (53.8%) and `absorptionPctOfContracted` (41.1%) are both true and 12.7
points apart; `beneficiaryCount` (47,599, EIK-or-name) and `beneficiaryCountEikOnly` (46,174)
differ by 1,425 organisations; `bgPartnerOrgCount` (983) and `bgPartnerRowCount` (1,493) by
52%. A key called `absorptionPct` invites a consumer to pick a denominator by accident.
- **`placedMoneyPct` is a MONEY share, never a row share, and the two are 45 points apart.**
Only 4.6% of rows carry no oblast — but they hold **50.05% of the money**, because the
national-scope programmes have no single oblast to sit in. „4.6% от договорите нямат място"
is true and misleading; a place surface declares the money coverage.
`funds_wire()` / `funds_news()` (migration 144) are the `/funds` band-0 wire and band-2 news
rail. **`db:load:funds-fit:pg` applies 144 as its last step**, so a corpus reload carries it —
`funds_news`'s third card reads `fund_fit`, which is why that loader is its home. Nothing else
applies it, and it needed an applier: `db:refresh` ends with `test:data`, so an unapplied 144 makes
a full reload fail at its final step. A body fix ships on its own with the usual escape hatch:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts 144_funds_wire.sql
```
**144 must DROP `funds_wire` before recreating it**, and the line is load-bearing: `checked_on` is
`text` rather than `date`, so the signature changed and `CREATE OR REPLACE` cannot alter a
function's OUT-parameter row type. The type is `text` because node-postgres converts a PG `date`
using the SERVER PROCESS's timezone — under `TZ=Europe/Sofia`, `2026-08-09` leaves as
`2026-08-08T21:00:00Z` and the page renders the day BEFORE. Prod is correct only because Cloud Run
leaves `TZ` unset, which is luck rather than design.
**144 also creates `idx_ifs_source_seen (source, first_seen_at)`, and that index is the whole
reason the wire is servable.** `idx_ifs_seen` carries only `first_seen_at`, so a time-range
predicate over `ingest_first_seen` pulls every dataset's rows out of a 15M-row table: measured at
30,105 buffers and 1,046 ms for one figure, on a function that runs on **every** /funds view. With
it, 2.3 ms. Any future „what did source X first see in window W" query wants the same index.
**Three things the corpus cannot do, which shape every label on those surfaces:**
- **`fund_projects` has NO date columns** — no signing, start or end date; ИСУН's beneficiary
export publishes none. So every figure is an INGEST window and the copy says „нови в ИСУН",
never „нови договори". The plan's „event date, not ingest date" rule was written for the
procurement corpus, which has `contracts.date`.
- **„Процедури, приключили наскоро" is therefore not buildable** — there is no completion date to
order by, and using the ingest date would present the crawl order as the finishing order. The
rail ships three cards, not the four the plan lists, and `funds_wire.data.test.ts` asserts
exactly those three so the omission stays a decision.
- **The disbursement card is restricted to the CLOSED 2014-2020 period** (`program_code LIKE
'2014%'`). Without that restriction a procedure at 0% is indistinguishable from one signed last
month — measured, the top three were all 0% and all 2021-2027, i.e. recency rendered as
underperformance.
**A backfill is not news, and the threshold is NOT re-derived here.** The `summarised` rule from
`007_query_builders.sql` (`rows_new > 500`, or any batch in summary mode) is applied in both
functions, so the wire and `/data/updates` cannot disagree about what counted. Backfill days are
reported SEPARATELY rather than dropped — on a freshly loaded corpus that is the difference
between „quiet" and „broken pipeline".
`fund_fit` (migration 143, `db:load:funds-fit:pg`) is the „финансирано ли е нещо като моето"
resolver behind the `/funds` tile and `/api/db/funds-fit` — a per-PROCEDURE rollup over the ИСУН
corpus (project count, grant quartiles, org-form mix, oblast breakdown) that answers the question
~68% of the measured audience actually asks, and which every other funds surface ignores. In
`db:refresh` right after `db:load:funds:pg`; on the cloud side:
```bash
npm run db:load:funds-fit:pg:cloud
```
**Its only staleness trigger is a funds reload** — `fund_projects` plus the procedure NAMES in
`fund_payloads(kind='procedure')`, both rebuilt by `db:load:funds:pg`. That pairing is in
`refresh_coverage.test.ts`'s `ORDER_PAIRS`. The Interreg arm needs nothing here (see below).
**Four things about it are easy to get backwards:**
- **BOTH corpora, served as SEPARATE arms.** ИСУН holds zero Interreg projects, and Interreg money
is cross-border so it lands almost entirely on BORDER municipalities — an ИСУН-only resolver
answers „нищо подобно не е финансирано наблизо" to exactly the readers whose neighbours hold
grants. They are never summed: an ИСУН figure is a contract's own value, an Interreg figure one
partner's published budget. `funds_fit_basis()` returns the declaration IN THE PAYLOAD so a
consumer cannot render one arm as the whole corpus.
- **Only the ИСУН arm is a matview.** `funds_fit_interreg()` is a plain function — 1,958 operations
against 82,011 contracts, which any index scan answers live. So an Interreg reload does NOT need
a `funds-fit` refresh.
- **The Interreg arm is reached through a BG→EN query bridge** (`functions/interreg_topics.js`).
keep.eu publishes 86% of these titles in English only (272 of 1,954 carry a Bulgarian one), so
without it a Bulgarian query matches almost nothing there and the arm that exists to fix the
border bias is invisible to the audience it is for. It bridges the QUERY only, never the ИСУН
arm, and the route returns `interregQuery` so the page can say which English term it used.
- **`paid_project_count` is DISBURSEMENT, not approval.** ИСУН publishes no rejected applications,
so an approval rate has no denominator and is not computable from this corpus. Any UI that
relabels it („N% одобрени") is making a claim the data cannot support.
The route degrades a missing 143 to empty arms and logs `ff:not-built` once per process, so first-deploy
ordering is cosmetic — but a STALE matview still serves the previous vintage's answer at a 200.
`funds_fit.data.test.ts` fails on an unbuilt matview, on a rollup that disagrees with
`fund_projects`, on a place filter that has stopped ranking, on a truncated oblast breakdown, and
on an empty Interreg arm.
`open_calls` / `open_calls_crawl` (migration 142, `db:load:open-calls:pg`) is the open-calls
register behind the `/funds` band-1 tile and `/funds/calls` — what a reader can APPLY to, from
ИСУН's `/Active` + `/PublicDiscussion` and ДФЗ's indicative Strategic-Plan schedule. In
`db:refresh` (after `db:load:funds:pg`); on the cloud side:
```bash
npm run db:load:open-calls:pg:cloud
```
Re-run after every `npm run opencalls:isun` / `opencalls:sp2023` crawl. Nothing runs it on the
cloud side, and the failure is the usual one: prod keeps the previous vintage at a 200.
**Four things about this family invert the rules the rest of this file states, all for the same
reason — here a stale number is a MISSED DEADLINE rather than a wrong figure:**
- **Nothing stores a status.** `open_calls_table` derives it by comparing `closes_at` to `now()`,
and every consumer reads that view. Stored-at-crawl-time would show expired calls as open all
weekend after a Friday failure; query-time derivation makes the worst case UNDER-reporting.
- **The loader must NEVER anti-join delete.** The crawler reads `/Active`, so a call that closes
is absent BY DESIGN — deleting absent rows would erase exactly the closed calls that make base
rates possible. It is upsert-only, and records absence via `last_seen_at`.
- **Money needs a provenance** (`enrichment`: `none`/`source`/`auto`/`reviewed`). A CHECK bars the
numeric columns unless it is `source` or `reviewed`, so an unreviewed LLM extraction cannot
drive the page's sort or range filters; it rides in `enrichment_meta` with its verbatim quote.
The loader's upsert deliberately does NOT downgrade a `reviewed` row back to `none`.
- **The money columns are `double precision`, not `numeric`**, and 142 carries a reconcile ALTER
because `CREATE TABLE IF NOT EXISTS` cannot retype a warm table's column. node-postgres
serializes `numeric` as a STRING, which blanks every money cell on the page while the number is
present in the payload — invisible to every row count and to any assertion made through SQL.
The ALTER has to DROP `open_calls_table` first (Postgres refuses to retype a column a view
reads); the file recreates it further down.
`open_calls_list(status, kind, audience, q, limit)` is the one serving function, and **a NULL
limit means unbounded** — that exists for the `/api/db/open-calls` count path, since a `count(*)`
through the function would otherwise saturate at its own `LEAST(p_limit, 2000)` ceiling. Counting
through the function rather than with a second WHERE is what keeps the tile's heading count and
the rows beneath it on one predicate.
**`/funds/calls` covers ИСУН + ДФЗ + 2 of 6 Interreg programmes, and the page says so.**
Interreg runs on Jems rather than ИСУН, so an Interreg call can never appear in `/Active` — a
system boundary that falls on the border municipalities. `npm run opencalls:interreg`
(`interreg_parse.ts`, one parser per programme shape) covers Greece-Bulgaria and Black Sea Basin;
Romania-Bulgaria publishes no calls index, and BG-RS / BG-MK / BG-TR reset the connection on both
ports from two independent clients (measured 2026-08-09). Do not add a programme to `PROGRAMMES`
without reading it first: a listed-but-unreadable programme makes a crawler report „0 calls" for
a site it never fetched, which is a hole that looks like a finding.
Two rules in that parser are load-bearing:
- **The deadline comes from its LABEL, never from the latest date on the page.** Greece-Bulgaria's
6th call prints `31.12.2029` — the programme period, in a state-aid paragraph — next to its real
`22/06/2026` deadline. A max-date heuristic publishes closed calls as open for years. A page with
no labelled deadline becomes an `indicative` row with a period label; it never gets a guessed date.
- **A down programme is not a change.** The crawler keeps the programmes it could read instead of
aborting, and the `interreg_calls` watcher excludes an unreachable programme from its fingerprint
— folding it in as „zero calls" would report every one of its calls as closed, then as new when
the site returned. Measured: Black Sea Basin went down mid-crawl on the first real run.
Two more rules, both learned the hard way:
- **The deadline keeps its TIME OF DAY.** Both programmes print one; a bare date resolves to
midnight, which marks a call closed for the whole of its final day and NULLs `days_left` a day
early. `sofiaWallClockToUtc` is reused from `isun_parse` — both zones are EET/EEST, and a fixed
+02:00 is an hour wrong for every summer deadline.
- **A page with no labelled deadline is REJECTED**, the way `isun_fetch` rejects a procedure with
no Краен срок. The only bucket an undated row could reach is `indicative`, which the UI labels
„Очаквани приеми" — expected intakes — and the pages that reach it are dead calls.
Zero open Interreg calls is a normal result (5 rows, 0 open on 2026-08-09) — the closed rows still
load, because `open_calls` accumulates and „the last one closed on 22 June" is the answer a border
municipality currently gets nowhere else. `open_calls.data.test.ts` asserts shape, never „at least
one is open".
**The coverage line says „част от", never a count.** It named „2 от 6" and „Черноморски басейн"
for one draft while the committed snapshot held zero Black Sea rows — that site answered once and
then refused every later attempt. A hard-coded fraction is a claim about data the component cannot
see and goes stale in both directions. `interreg_fetch.ts` carries a **completeness guard** that
refuses to write when a programme which HAD rows returns none; `writeSnapshot`'s shrink guard
cannot do this job, being a per-source ratio for which 9 → 7 is 22% and under its threshold.
**One-off, and Cloud SQL needs it by hand.** `open_calls` never deletes, so the two undated rows an
earlier parser wrote survived the rejection fix:
`DELETE FROM open_calls WHERE source = 'interreg' AND closes_at IS NULL;`
**Enrichment (money + eligibility) is a SEPARATE, human-gated skill** — `enrich-open-calls`,
never part of the daily refresh. `update-open-calls` gives a call its title, deadline and link;
ИСУН publishes the budget, aid rate, grant range and eligibility only inside each procedure's own
PDF/DOCX. That skill reads one document, extracts each field **paired with a verbatim quote**, and
puts both through `scripts/opencalls/enrich_gate.ts` before anything is stored — **two checks,
not one**: the quote occurs in the extracted text (a plain normalised substring), _and_ the quote
states the value. A field failing either is dropped and reported. Nothing about that guarantee
comes from trusting the model.
The second check is the one that looks redundant and is not. Checking the citation is not
checking the claim: with only the substring test, a fabricated `budget_eur: 999 000 000` attached
to a real unrelated sentence from the document passed with no rejection, and so did a 100×
magnitude error (`aid_rate_pct: 0.6` cited from „…60 %…"). Both are the shape a model produces
when it answers from memory and then hunts for a sentence to cite. Neither check can judge
whether the quoted sentence is the _right_ one — a sub-component's „максимален размер" cited
against the whole procedure's budget is a real number, correctly attributed and still wrong —
which is why `auto` may not reach a money column at all.
Two things about it are easy to get backwards:
- **`enrichment='auto'` publishes no number.** It may write the verbatim eligibility text and the
provenance blob; 142's `open_calls_money_needs_provenance` CHECK bars it from all four money
columns. Only `npm run opencalls:enrich-review -- --promote <key>`, one row at a time after a
human reads the quotes, promotes to `'reviewed'` and lets a figure into sorting, range filters
and the tile's total. There is deliberately no `--promote-all`.
- **The crawl must not own an enriched column.** `load_open_calls_pg.ts` splits its upsert into
`SOURCE_OWNED` (bare assignment) and `FILL_NEVER_BLANK` (COALESCE), and `enrichment` never
downgrades from any stored value to an incoming `'none'`. Measured 2026-08-09, before the
split: one ordinary `db:load:open-calls:pg` took a promoted row from its eligibility text to
NULL while leaving `enrichment='reviewed'` and the quotes in the meta — a row asserting a human
signed off on text that was gone, at 66 → 66 rows with nothing red.
`load_open_calls_pg.test.ts` derives the protected set from the writer's own `MONEY_FIELDS`, so
a new money column cannot be added on one side only.
**Enrichment lives only in Postgres — it is NOT in `data/opencalls/*.json`.** So it does not
travel with `db:load:open-calls:pg:cloud`; enrich against the database you intend to serve.
Measured on today's corpus, the money yield is near zero and that is the gate working: 0 of 4
readable ИСУН documents stated a euro amount, 3 stated levs, and the currency rule correctly
refuses a lev figure offered as euro. The eligibility text is the real yield until the documents
are re-tabled in euro.
**When it happened on the wrong database anyway, `opencalls:sync-enrichment` carries it over.**
Dry-run by default; `--apply` writes:
```bash
npm run opencalls:sync-enrichment:cloud -- --apply
```
It copies the overlay — `enrichment`, `enrichment_meta`, `beneficiaries_raw` and the four money
columns, the payload derived from `enrich_apply.MONEY_FIELDS` so a new money column cannot be
added on one side only — for rows at `enrichment IN ('auto','reviewed')`. `source` rows are
excluded because the loader already reproduces them from the committed snapshot. Source defaults
to local, target to `DATABASE_URL` (`--from`/`--to` override); a same-database sync is refused
rather than reported as a no-op.
Three things about it are easy to get backwards:
- **It NEVER downgrades**, on the same total order the loader's upsert uses (`ENRICHMENT_RANK`,
exported from `load_open_calls_pg.ts` and imported here rather than restated — it is now the one
definition, rendered as SQL by `enrichmentRank()` and compared in TypeScript by `outranks()`).
A row a human promoted **on the target** outranks a local `auto` and is kept, and the guard is
in the UPDATE's WHERE as well as in the plan, so a promotion landing between the read and the
write cannot be overwritten by a stale decision. A TIE is not a downgrade: the source wins it,
which is what lets a re-gated `auto` refresh its own meta.
- **It never INSERTs.** The crawl owns row existence; a source row missing on the target is
reported, and the fix is `db:load:open-calls:pg:cloud` first, then re-run. It also reports rows
enriched on the TARGET only, so a divergence between the two databases is visible rather than
inferred from a silence. **A missing or refused row exits non-zero** — including on a dry run,
whose job is to report whether a full sync is possible — so a wrapper can tell a complete sync
from a partial one without parsing stdout.
- **The overlay moves as a UNIT, NULLs included** — never a per-column COALESCE. `enrichment` is a
claim about the whole set of figures, so merging two provenances under one flag is exactly the
thing it exists to prevent.
`sync_enrichment.test.ts` mutation-checks the never-downgrade rule: it asserts a `reviewed` target
survives an `auto` source, then re-runs the same input with the rank predicate stubbed out and
asserts it flips to an update — otherwise the first assertion is satisfied by any plan that
happens not to write.
`/funds/calls` is PRERENDERED (`scripts/prerender/routes.ts`) and has a sitemap `<loc>`, but
**there are no per-call `<loc>`s and the prerendered body lists no calls** — a static snapshot of
live deadlines would serve expired calls as open. An individual procedure has no page here; every
row links out to ИСУН or ДФЗ, which is where you apply.
`nzok_pathway_tariffs` (migration 059, `db:load:nzok-tariffs:pg`) is the НРД price factor
behind the pathway-spend tree and the case-mix signal on `/awarder/121858220`. Its source is
the НРД **contract body** (чл. 368/369/370, re-tabled by each amendment), parsed by
`scripts/nzok/write_pathway_tariffs.ts` into the gitignored `pathway_tariffs.json`. Publish:
```bash
npm run db:load:nzok-tariffs:pg:cloud
```
Re-run writer + loader when the `nzok_nrd_tariffs` watcher flags a new НРД/amendment PDF
(the parse needs a human pass). The loader is absent-safe (applies 059 and exits when the
JSON is missing), so it sits in `db:refresh` — but only this manual flow ever _fills_ the
table.
`nzok_casemix_expected_vs_actual()` (migration 059) is the per-hospital case-mix signal behind
`/api/db/nzok-casemix-by-eik` and the case-mix line on `NzokReportCardTile`. It divides what
НЗОК actually paid a hospital by what the НРД list price says its OWN case mix should have
cost.
**It is a FUNCTION, so no `db:load:*` ships a body change** — the tariff loader applies 059,
but a guard fix must not wait for a tariff reload. Ship it with the usual hatch, then
`deploy:db` (the route) and `deploy` (the tile copy):
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts 059_nzok_pathway_tariffs.sql
```
Two guards suppress the RATIO (never the whole payload) and name the reason, so a surface can
say WHY instead of dropping the row. Both conditions LOOK like a cheap hospital:
- **`partial-payment-year`** — the numerator is summed over the payment months the corpus
holds. The floor is the YEAR'S OWN full complement, **never a constant**: the payment
corpus holds 9 months for 2023, 12 for 2024, 11 for 2025 and 6 so far for 2026, and since
the activity corpus went multi-year the year this reads is not fixed. A hard 11 would
suppress every hospital in 2023 or 2026. `fullYearMonths` rides in the payload so copy says
„4 of 11" rather than implying twelve.
- **`low-tariff-coverage`** — `expected` only counts cases whose procedure has a tariff, so
below 80% it compares against too little of the hospital's work.
Measured on 2025: 236 published, 6 `no-payments`, 3 partial-year (including the
€1.1-per-case facility — 4 months of payments against 1,646 cases), 3 low-coverage.
`nzok_casemix_guards.data.test.ts` holds the invariant (`ratio` NULL iff `suppressed`), the
derived floor, non-vacuity, and a ceiling on how much the guards may swallow. **No ranking is
published** — the ratio appears on one hospital's own card, never as a league table.
`agri_subsidies` + `agri_payloads` (migration 046, `db:load:agri:pg`) are the ДФ „Земеделие"
farm-subsidy corpus behind `/subsidies` and `/farm/:eik`. The loader is the pure-LOAD half of
the fetch/load split: it reads only the **gitignored** `raw_data/agri/` cache (egov year sheets
- СЕУ CSVs) — on a fresh clone it skips-and-warns; on a PARTIAL cache it throws rather than
publish a corpus missing a financial year. Publishing to prod:
```bash
npm run db:load:agri:pg:cloud
npm run db:load:agri-hub-stats:pg:cloud # the /subsidies hub cache — see below
```
**`db:load:agri-hub-stats:pg` is a SECOND, much cheaper loader for the same corpus, and it
has its own trigger list.** It applies migration 162 and rebuilds `agri_hub_stats_cache`, the
scope-keyed matview behind `/api/db/agri-hub-stats` and every figure on the `/subsidies` hub —
**5.9 s**, against the 5m44s the full agri ingest takes. `db:load:agri:pg` also applies and
refreshes 162, so a corpus reload carries it; this exists because the cache has FIVE inputs and
the agri ingest owns one:
| input | filled by |
| ---------------------------------- | ------------------------ |
| `agri_subsidies` / `agri_payloads` | `db:load:agri:pg` |
| `person_role`, `person` | `db:resolve:persons` |
| `fund_projects` | `db:load:funds:pg` |
| `contracts` | `db:load:pg` |
| `budget_muni_transfer` | `db:load:budget-muni:pg` |
Re-run it after **any** of those. `db:refresh` runs it at step 56, after the person chain, for a
reason worth knowing: the agri ingest is step 14 and `db:resolve:persons` is step 45, so the
cache the ingest builds is always one vintage behind on the political arm — and on a FIRST run,
against a person layer `081` has just created empty, that arm is not stale but **zero**, which
is a claim („0 фирми") rather than an absence. The loader warns when it sees that state.
⚠️ **The agri ingest is now a second applier of `081_person_identity.sql` and
`154_budget_municipal.sql`.** `CREATE MATERIALIZED VIEW` resolves its query at creation, so 162
cannot compile against a database where `person_role` or `budget_muni_transfer` is absent — and
on a cold `db:refresh` both are created after step 14. Applying their DDL first (the pattern
`load_graph_pg.ts` uses for 137) is what keeps a fresh clone's chain alive; both files are
`CREATE TABLE IF NOT EXISTS` throughout, so it is idempotent and free on a warm database.
`contracts` and `fund_projects` are guaranteed by chain ORDER instead, and preflighted — a
missing one skips the migration with a warning rather than aborting the load.
Run the base loader after any `raw_data/agri/` refresh (a new egov financial year, or a fresh
`npm run agri:seu` pull); the fetch+load path stays `npm run agri:ingest` (the update-agri
skill). Nothing on the cloud side is automatic. The subsidies table publishes via an UNLOGGED
stage + one-transaction DELETE+INSERT (RowExclusiveLock only — readers stay on the MVCC
snapshot, never blocked; <5%-shrink guard), `agri_payloads` via the shared stage merge — so the
~2.5M-row reload cannot 55P03 the served browse. **Not a rename swap**: `person_browse_table`
(120) and `company_public_money` (127) are matviews over `agri_subsidies` and follow its OID
through a rename, so the table must keep its identity. Those two are also why a cloud agri
reload does not end at the loader — `agri_subsidies` is part of 127's money basis and of the
`/persons` money column, so after `db:load:agri:pg:cloud` re-run
`db:load:persons-browse:pg:cloud`, `db:load:person-search:pg:cloud` and
`db:load:graph:pg:cloud` (their sections above name any contracts/agri/funds reload as their
trigger). `agri_subsidies` is in
`sync_cloud.ts`'s `CRITICAL_TABLES`: its source cache is gitignored host state, so a dropped
table is only re-derivable from a machine that still holds the cache.
`kzk_decisions` (migration 130, `db:load:kzk-decisions:pg`) is the КЗК merits-outcome corpus —
the tier-2 half of the appeals pack, and the loader whose absence let that arm freeze for five
weeks unnoticed. It has **no automatic cloud path**:
```bash
npm run db:load:kzk-decisions:pg:cloud
```
Re-run it after every `scripts/procurement/kzk_decisions.ts --apply` crawl, and on a first
deploy before anything reads the table. It applies 005 + 130 itself, so it works on a cold
database. `db:refresh` runs the local equivalent; nothing runs the cloud side.
Two things make this loader unlike the rest of the list, both because — until the crawler's
`--backfill` is proven to re-derive it — **this table plus the gitignored
`data/procurement/kzk_decisions.json` are the only copies of a corpus with no committed
generator**. It is in `sync_cloud.ts`'s `CRITICAL_TABLES` despite being small, and its merge
refuses to run when the build would shrink the table by >5% (`--allow-shrink` to override) or
when more than 15% of source rows fail validation — an anti-join DELETE is the correct shape
for a derived table and a data-loss bug on this one. Note the appeals arm is the OPPOSITE
shape: `kzk_appeals` has no loader at all, so publishing it means re-crawling against the
cloud URL. `kzk_decisions.data.test.ts` fails on an empty table or a malformed act.
Riding immediately behind it — **the rejoin** (`npm run kzk:rejoin -- --apply`), which applies
migration 131 and folds `kzk_decisions` into `kzk_appeals.outcome`. Nothing else applies 131,
so without this the cloud database has no `decision_act_no` column at all:
```bash
npm run kzk:rejoin:cloud -- --apply
```
Re-run it after **every** `db:load:kzk-decisions:pg[:cloud]` and after any change to
`scripts/procurement/kzk_match.ts` — a matcher fix ships no data by itself. `db:refresh` runs
the local equivalent; nothing runs the cloud side, and skipping it is the "green locally,
stale on prod" class: local serves 3,014 outcomes while prod keeps 2,098, with nothing failing.
It refreshes every dependent through `scripts/procurement/kzk_dependents.ts` — including
`upheld_ocids`, which feeds the contract Corruption Risk Index, so a skipped rejoin makes
recently-appealed procedures grade cleaner than they are.
`--dry-run` is read-only and reports what it would write, including the hand-seeded rows it
refuses to touch. Provenance (131) is what makes the rejoin safe to re-run: `decision_act_no
IS NOT NULL` marks a row as machine-derived and re-derivable, NULL marks it as one of the
~2,098 irreplaceable hand-made ones, which no writer may overwrite.
**Migration ordering, 042 → 131.** `kzk_appeals_list` (042) SELECTs `decision_act_no`, whose
home is 131 — and 131 is applied ONLY by `kzk:rejoin`, which every path that applies 042 runs
_before_ (`db:refresh` orders `db:load:tenders:pg` ahead of `kzk:rejoin`; `load_tenders_pg`
and `apply_functions` never touch 131). 042 therefore carries an idempotent
`ADD COLUMN IF NOT EXISTS` for it. **Do not remove that line** thinking 131 owns the column:
`exec()` sends a migration as one implicit transaction, so a 42703 there rolls the whole file
back and aborts the tenders loader on any database that has not yet rejoined. It is invisible
on a machine that already has the column, which is exactly how it shipped once;
`kzk_decisions.data.test.ts` now asserts the ALTER is present and precedes the view.
**One-off, and Cloud SQL needs it by hand — `kzk_appeals.suspension`.** The column held a
stored `false` on 7,778 of 7,886 rows, which made 042's
`kzk_effective_suspension(suspension, status)` fallback unreachable: 1,501 appeals had
requested a temporary measure and at most 4 could ever display as suspended. **A re-crawl
cannot fix this** — the intake passes NULL into `COALESCE(existing, EXCLUDED)`, so the frozen
value is immovable — and there is no `db:load:kzk:pg:cloud` to carry a fix over. Prod stays
frozen until someone runs it there:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts 042_kzk_appeals.sql 044_procurement_ai.sql
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/procurement/kzk_unfreeze_suspension.ts --apply
```
**Order matters and the script enforces it.** 042 must land FIRST: before it,
`kzk_appeals_list` selected the RAW column, so releasing the column takes
`/procurement/appeals` from 4 suspended chips to 0 while every other surface still shows 4.
The script refuses to run if `kzk_effective_suspension()` is absent. If the one-off has been
deleted (it is scheduled to die when the определения arm lands), the equivalent is
`UPDATE kzk_appeals SET suspension = NULL WHERE suspension IS NOT NULL;` — but only while no
row is suspended without a `спрян` status, which is exactly what
`kzk_suspension.data.test.ts` asserts.
**`kzk_effective_outcome(outcome, status)` (042) — the ONE definition of whether an
appeal has a published ending.** `отказано производство` (КЗК refused to open
proceedings) is a DETERMINATE TERMINAL STATE, not a missing value: the filing was
denied and will never be reviewed on the merits. **1,661 of 7,998 appeals (20.8%)**
carry it, and every surface rendered them as a blank outcome until 2026-08-25.
⚠️⚠️ **IT IS DERIVED AT QUERY TIME AND MUST NEVER BE STORED, and the reason is not
obvious.** The obvious implementation — `UPDATE kzk_appeals SET outcome = … WHERE
status = 'отказано производство'` — **silently destroys the provenance guard on the
~2,098 irreplaceable hand-seeded rows.** 131's rule is the only test of provenance:
`decision_act_no IS NOT NULL` means machine-derived and overwritable, NULL means
hand-seeded and protected. A status-derived outcome has a THIRD provenance and no act
number, so it lands in the protected bucket. Measured: the guarded population goes
**2,098 → 3,759**, and `kzk_appeals_provenance.data.test.ts` **stays GREEN**, because
its floor is a `>=` — exactly the laundering hazard `kzk_baselines.ts`'s
`HAND_SEEDED_FLOOR` comment describes. It would also freeze those 1,661 rows against
every future matcher improvement, since the rule refuses to overwrite them.
Deriving stores nothing, so `decision_act_no IS NULL` keeps its meaning and
`kzk_baselines.outcomes` does not move — correct, because 1,661 outcomes appearing
from a re-reading of `status` is not the matcher getting better, and letting it raise
the ratchet would mask a later real regression.
**Cloud SQL needs it by hand, 042 BEFORE 044** (044's `kzk_appeals_summary()` calls
both effective-value functions, and `exec()` validates a `LANGUAGE sql` body at CREATE):
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts 042_kzk_appeals.sql 044_procurement_ai.sql
```
**Applied to Cloud SQL 2026-08-25** — 1.35 s, and the reader-visible window is real but
small: 042 DROPs `tenders_list`, `kzk_appeals_list CASCADE`, `appealed_ocids CASCADE`
and `upheld_ocids CASCADE` in one transaction, so those pages block on an
AccessExclusiveLock until it commits (the matviews are only 4,883 and 752 rows; 042
recreates `risk_upheld_ocid` itself). Measured either side: blank outcomes 4,920 →
**3,271**, `withOutcome` 3,078 → **4,727**, and everything that must not move did not —
hand-seeded 2,098, stored outcomes 3,078, stored `отказана` **0**, `upheld_ocids` 752,
`summary.upheld` 983.
Four things are easy to get backwards:
- **THE STORED VALUE WINS.** 12 appeals carry BOTH this status and a merits outcome
(9 `уважена`, 3 `отхвърлена`) — all 12 hand-seeded. Either the status is stale or the
seeding was wrong; either way `COALESCE` keeps the human's answer, the same precedence
`kzk_effective_suspension` uses.
- ⚠️ **WHAT MUST NOT USE IT.** `upheld_ocids` and `buyer_appeal_stats` filter
`outcome = 'уважена'` on the RAW column — a refusal is not an uphold, and that matview
feeds the contract Corruption Risk Index. `partitionByProvenance()` reasons about the
STORED value. And `kzk_appeals_summary()`'s `upheld` / `rejected` counters stay raw for
the same reason its `with_outcome` does not.
- ⚠️ **044 WAS MISSED ON THE FIRST PASS**, and the shape is worth knowing: it had already
adopted `kzk_effective_suspension` and left `with_outcome` on the raw column, so the AI
tool answered **3,078** while `/procurement/appeals` published **4,727** for the same
question — and quoted the lower figure as a grounded fact. It has a TypeScript twin,
`build_kzk_summary.ts`, that must move with it.
- **The code is `отказана`; the LABEL is „отказано производство".** The code is ours, not
a word the register prints, so `src/lib/kzkLabels.ts` carries a BG override — the only
entry in that map, because every other outcome IS a register term. **Until hosting ships
that bundle, prod renders the bare code** in both languages (the old `kzkOutcomeLabel`
falls through to the raw term), including in the `/procurement/appeals` outcome facet,
whose options ARE the column's distinct values. Harmless and visible; `npm run deploy`
closes it.
The gate is `scripts/db/tests/kzk_effective_outcome.data.test.ts` (6 tests) — the
never-stored invariant, that a stored outcome wins, that the CASE fills exactly the
refused population and nothing else, that the derived value cannot reach an uphold
count, plus a mutation check that the serving view really publishes it.
`contractor_rank` (migration 122) is the same first-deploy shape as `cpv_catalog`: the
`contractor_rankings` DbDataTable resource + the `/api/db/contractor-scope-kpis` route read
it and do NOT degrade a missing matview to an empty result, so on the FIRST cloud deploy the
loader must run **before** the `deploy:db` that ships them. `contractor_rank.data.test.ts`
fails on an empty or stale matview.
`procurement_settlement_payloads` (migration 123) is the deliberate OPPOSITE of those two on
first deploy. `/api/db/procurement-settlement` maps the requested window to a scope and reads
the matview, but **falls back to the live `procurement_settlement_detail()` when the
precompute cannot answer** — no stored row, or a narrow set of SQLSTATEs meaning the
matview is absent, unreadable or locked (a pool error still throws). Degrading is correct here in a
way it is not for `cpv_catalog`: it yields the RIGHT answer slowly (today's behaviour) rather
than a wrong one, so the route ships in any order, to any database. The cost is that every
reason the fast path was skipped is otherwise silent, which is why the route logs a
`psp:not-built` / `psp:read-failed` warning once per process — **that log, not latency, is the
signal that the cloud loader never ran.** `procurement_settlement_payloads.data.test.ts` fails
on a stale, partial or place-blank matview.
`procurement_payloads` (migration 124) is the same degrade-don't-fail shape as 123, applied to
the six `/api/db/procurement-*` dashboard routes — `overview`, `flow`, `rankings`,
`concentration`, `sectors`, `benchmarks` — via one shared `scopedPayload()` helper in
`functions/db_routes.js`. It exists because TWO of them exceeded the 10 s `statement_timeout`
and returned 500 on prod — `procurement-overview` on a windowed scope at 10.010 s and
`procurement-flow` with no window at 10.006 s. The other FOUR were the same shape and had
simply not been unlucky; `procurement_concentration` touches more pages than either
(411,245 vs 393,851 on the full corpus) and had no cache at all. (The third 500 in that Cloud
Run window was `/api/db/person-profile`, a different defect entirely — a point lookup that
full-scanned `person` — fixed by rewriting `person_by_name` in 082, not by a precompute.) Same log contract, different prefix: `pp:not-built` / `pp:read-failed`,
once per process per (kind, scope) — so an unbuilt `flow/all` and an unbuilt `overview/all` are
two separate lines, and a second request for either is silent. `pp:no-scope` is EXEMPT — a caller may legitimately ask for a window
that is not one of the thirty, and serving that live is designed behaviour, not a defect.
**Two SQLSTATE details on that fallback are easy to get backwards, and 123 had both wrong
until 124 was built:**
- **`55000` must be in the degrade set.** Reading a matview created `WITH NO DATA` does not
return zero rows — it raises `object_not_in_prerequisite_state`. That is exactly a database
where the DDL was applied and the REFRESH never ran, i.e. the first cloud deploy, which is
the case the orderless-deploy property is about. Without it that case is a 500.
- **`57014` must NOT be.** It looks like the "locked by a REFRESH" code; that is `55P03`.
`57014` is the pool's own `statement_timeout`, so the probe has already burned the full 10 s
budget and falling back to an aggregate touching 199k–411k buffers cannot finish either — it
turns a 10 s failure into a ~20 s one holding a pooled connection, under exactly the
saturation that caused the timeout. Degrading is only correct when it beats failing.
`procurement_payloads.data.test.ts` fails on a stale matview, on a `concentration` payload whose
stored `oblast` has drifted from `awarder_seats`, and — uniquely in this repo — on a
`SCOPED_MATVIEWS` entry whose declared `inputs` do not cover what the matview reads. It
deliberately does NOT skip when the matview is absent or unpopulated: those are the two states
it exists to catch, so they are assertions, not a green skip.
**124 RETIRED two older cache matviews**, `procurement_overview_cache` (025) and
`procurement_rankings_cache` (031). Each answered exactly ONE of the thirty windows — the full
corpus — which is why every parliament window fell through to the live aggregate and 500'd.
Their migrations now carry a tombstone `DROP` and no `CREATE`; `load_pg` no longer refreshes
them; the routes no longer read them.
**The tombstone only fires when the file is APPLIED, so the cloud side needs one command** —
`deploy:db` ships function code only, and the only other cloud path that applies 025/031 is a
~90-minute full contracts reload. Without this, prod keeps two orphan matviews for ever:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts 025_procurement_overview.sql 031_procurement_rankings.sql
```
Safe to run at any time and idempotent: both files `CREATE OR REPLACE` their function (they do
NOT `DROP` it — 124 depends on it), and the `DROP … IF EXISTS` is a no-op once the matview is
gone. **Do not add `122_contractor_rank.sql` to that command** — 122 DROPs and recreates
`contractor_rank` WITH NO DATA, and `/procurement/contractors` reads it WITHOUT degrading, so
applying it would blank that page until a multi-minute refresh finished.
Of the full-corpus cache matviews on this pattern, `procurement_by_settlement_cache` (030) is
the only procurement one left — its route serves a different shape from 119/123, so it still
has a job. (`procurement_risk_indexes_cache` (033), `dual_corpus_rankings_cache` (077) and the
044 cache are unrelated and unaffected.)
`db:load:pg` also re-REFRESHes all six (guarded on existence, `contractor_rank` before
`contractor_scope_kpis`), so a contracts reload cannot leave `/procurement/by-settlement`,
`/procurement/contractors`, the settlement pages or the `/procurement` dashboard serving the
previous corpus. `db:refresh` runs the local equivalent automatically; the cloud side does not.
**Skipping it does not fail — it blanks.** `db:resolve:persons` applies 117 with
`CREATE TABLE IF NOT EXISTS`, so a cloud database that never ran this loader gets an EMPTY
dimension, `person_by_slug()` still compiles, and all ~76.5k `mir`/`obshtina` roles publish
`placeLabel: null` — green locally, blank on prod, and baked into the prerendered `/person`
HTML. The ~2.7k judicial roles keep their label (they resolve via `judicial_body`), so the
damage looks partial rather than total and is easy to miss.
Last in the person chain, and the only loader with **two** independent triggers — the
persons browser matview (migration 120) behind `/persons`:
```bash
npm run db:load:persons-browse:pg:cloud
```
It folds six upstream datasets, so it must run **after all of them**: `db:resolve:persons`,
`db:load:declarations:pg:cloud -- --resolve` (net worth **and** `declaration.person_id`),
`db:load:official-candidate-links:pg:cloud`
(the non-MP photos), `db:load:judicial-bodies:pg:cloud` + `db:load:place-dim:pg:cloud` (every
place label and the magistrates' oblast), and the contracts corpus. Re-run it:
- whenever any of those is reloaded on the cloud side — the person half is the obvious case;
- **after a contracts reload**, which is the non-obvious one: `public_money_eur` is computed
from `contracts`, so a procurement refresh without this one lets the money column drift
away from `/procurement/contracts` with nothing failing;
- **after a `db:load:tr:pg[:cloud]`** — the matview reads `tr_officers` for its company
counts. Until 2026-08-10 this was invisible for the worst possible reason: 003's
`DROP TABLE … CASCADE` _deleted_ the matview on every TR load rather than staling it (see
the CASCADE note in "SQL functions and indexes" below). 003 no longer drops, so what is
left to close is ordinary staleness — and this is the loader that closes it.
Riding just behind it — `person_search` (migration 126, `db:load:person-search:pg`), the one ranked
index behind the combined-search route. It is a derived search index like `contractor_search`, but
UNLIKE `contractor_search` (rebuilt inside `db:load:pg`) it is a **standalone** loader with nothing
running it on the cloud side, so run:
```bash
npm run db:load:person-search:pg:cloud
```
after **each** of `db:load:persons-browse:pg:cloud` (its public arm), `db:load:tr:pg:cloud`, and any
contracts/agri/funds reload (its money arm). `db:refresh` sequences the local equivalent right after
persons-browse; nothing runs it on the cloud side. The `/api/db/person-search` route **degrades a
MISSING table to empty tiers** (so it never 500s on a first deploy before this loader runs), but a
**STALE** table still serves the previous vintage at a 200 — so this is the same "green locally,
stale on prod" trap as the loaders above. `person_search.data.test.ts` fails on an absent/empty table.
**`db:load:magistrates:pg[:cloud]` is a fourth trigger, and it is the one the list above reads as
somebody else's problem.** The other three name a table this loader reads DIRECTLY; a magistrate
roster reload reaches it only transitively — magistrates → `db:resolve:persons` → persons-browse →
here — so nothing in the chain looks like a person_search dependency. It is: since 070's
`magistrate_search()` was retired, `person_search` is the ONLY surface that makes a magistrate
findable by name, and the whole point of the roster retention (6af09bd0c6) is that a magistrate who
leaves the ИВСС register stays findable. Measured 2026-08-11, mid-change: **393 of the 460 retained
magistrates were missing from `person_search` while all 460 were already in `person_browse_table`** —
the retention's own purpose defeated, at a 200, with every row count reconciling. Re-run this loader
after any magistrate reload, on both sides.
`person_crypto_table` (migration 159) is the declared-CRYPTO register behind
`/declarations/crypto` and the `crypto_holdings` DbDataTable resource. It has **no loader of
its own** — it is a fifth victim of 090's `DROP MATERIALIZED VIEW person_wealth_year CASCADE`,
so `load_declarations_pg.ts` applies it in phase 2 right after 120, and its `CREATE … AS`
populates it. On the cloud side it therefore rides:
```bash
npm run db:load:declarations:pg:cloud -- --resolve
```
**First deploy is ORDERED, and getting it wrong is a 500 rather than a narrower answer.**
The registry engine reads the base relation unconditionally (same shape as `cpv_catalog` /
`contractor_rank` — there is no `missingMigration` degrade on a DbDataTable resource), so a
`deploy:db` that ships `crypto_holdings` before the loader has reached the target 500s every
request to `/declarations/crypto`. Loader first, then `deploy:db`, then `deploy`.
Three things about it are easy to get backwards:
- **It joins through `person_wealth_year`, and that is a CORRECTNESS property, not a
convenience.** A holding is re-declared on every filing that covers it — Борис Михайлов's
500,000 BUSD sits on both his 2023 годишна and his 2023 при-напускане — so summing the raw
`declaration_asset` rows reads **€1,960,489 against a true €1,649,180**, a 19% overstatement
on a page whose entire content is a number beside a person's name. 090 already picks ONE
declaration per (person, period_year); joining through it is what stops this register
becoming a fifth opinion about which filing counts.
- **The `scope` fan-out needs its `defaultScope`** (`latest`), exactly as `mp_cars` does. An
unscoped query is otherwise the UNION of the `latest` and `all` buckets, which serves that
double-count with the `count` and `sum` aggregates inflated to match and nothing erroring.
- **The classifier is `is_crypto_asset()` in 090, beside `asset_share_multiplier`, and it has
NO TypeScript twin — do not add one.** Every consumer is server-side. It classifies by RULE
("the declared unit is not money") rather than by a ticker allowlist, so a new coin
classifies itself; the price is that a new FIAT spelling would publish a bank balance as
crypto, which `declared_crypto.data.test.ts` closes by requiring every distinct non-fiat unit
in the corpus to be classified deliberately. The register is full of Cyrillic homoglyph
typos (`ЕUR`, `ВGN`, `УСД`) and hand-typed units (`шв. фр.`, `ФЖХ` — a BGN mistype, provable
because its €/unit is exactly the peg), so that gate is load-bearing. Note the precious-metal
carve-out matches on EXACT equality: `PAX Gold` is a gold-backed TOKEN and must stay crypto,
while `XAU` / `инвестиционно злато` must not.
Plan: `docs/plans/declared-crypto-v1.md`.
`person_abroad_table` + `person_abroad_overview()` (migration 169) are the „Пари в чужбина"
register behind `/declarations/abroad` and the `abroad_holdings` DbDataTable resource — the
corpus answer to „how much of the money declared by officials is held abroad". Like 159 it
has **no loader of its own**: it is the **SIXTH** victim of 090's `DROP MATERIALIZED VIEW
person_wealth_year CASCADE`, so `load_declarations_pg.ts` applies it in phase 2 after 159, and
its `CREATE … AS` populates it. Cloud side it therefore rides
`npm run db:load:declarations:pg:cloud -- --resolve`.
**First deploy is ORDERED, and getting it wrong is a 500 rather than a narrower answer** —
the registry engine reads the base relation unconditionally, exactly as for `crypto_holdings`.
Loader first, then `deploy:db`, then `deploy`.
Three things about it are easy to get backwards:
- **It joins `person_wealth_year`, and that is a CORRECTNESS property.** A holding is
re-declared on every filing that covers it: raw rows are 3,196 / €168.5m against a deduped
2,788 / €143.1m, a 17.8% overstatement. Same trap `person_crypto_table` documents.
- **Its `'latest'` anchor is NOT 159's.** 159 anchors on each person's most recent
CRYPTO-bearing period so it agrees with the block on that person's profile; 169 anchors on
the most recent FILING, because a corpus aggregate must drop holdings people have stopped
declaring. `person_abroad_overview()` shares that anchor — if the two ever diverge the
page's headline stops equalling the rows beneath it.
- **Coverage is narrower than „money", and every consumer must say so.** `held_scope` exists
only on tables 5 and 8, so the register spans €2.26bn of €5.67bn in declared assets and
magistrates are absent entirely. The same numerator is **5.9%** of bank+investment money,
**2.3%** of declared holdings and **0.8%** of the corpus — which is why
`person_abroad_overview()` returns `eurInScope` beside `eurAbroad` and nothing publishes a
bare percentage.
The gate is `scripts/db/tests/person_abroad.data.test.ts` (6 tests), plus
`AbroadRegistryScreen.test.tsx` and a degrade-contract test in `db_routes.person.test.js`.
Last of the person-layer standalone loaders — the connections graph (migrations 127 + 128 + 129,
`db:load:graph:pg`), the three `graph_*` tables (`graph_edge` / `graph_company_node` /
`graph_person_node`) + the down-sampled `graph_payloads` blob behind `/connections` and the re-pointed
`person_connections()` / `person_graph_ego()`. It APPLIES 127 (`company_public_money`, the broad
contracts∪subsidies∪funds money basis) + 128 + 129 and rebuilds the tables from `person_role`
(co-ownership) ∪ `company_politicians` (procurement) via a **stage merge** — all four are on a serving
path (084's `person_connections()` / `person_graph_ego()`, and `/api/db/connections-graph` for the
blob), so a `TRUNCATE`-and-rebuild would hold an AccessExclusiveLock for the whole load and 500 those
routes at the pool's `lock_timeout` (`person_reload_locks.data.test.ts` is the gate). Run:
```bash
npm run db:load:graph:pg:cloud
```
**AFTER `db:load:persons-browse:pg:cloud`** (it reads `person_browse_table` facets for the person
nodes) and after **each** of `db:resolve:persons:cloud`, `db:load:person-elections:pg:cloud` (the
`party`/`party_color` source for the party×party matrix, `person_election_stats`),
`db:load:tr:pg:cloud` (the `company_politicians` procurement arm), **`db:load:interreg:pg:cloud`**
(127 gained an Interreg arm and this loader is 127's only applier — see the Interreg section),
and **any contracts/agri/funds reload** (127's money basis).
`db:refresh` sequences the local equivalent right after `persons-browse`/`person-search`; nothing runs
it on the cloud side, so it is wired into the `update-persons` (last step) and `update-procurement`
(after `persons-browse`) watch skills so an orchestrated re-ingest re-derives the graph on prod.
**`tr:daily-refresh` runs this loader too**, between `cr-nkid` and `tr-company-place` — that
position is forced from both sides: it must FOLLOW `db:load:tr:pg` (which rebuilds
`company_politicians`, the procurement arm's input) and PRECEDE `db:load:tr-company-place:pg`
(which denormalizes `money_eur` from the `company_public_money` this loader rebuilds). It was
added 2026-08-20; before that the chain stopped at `tr-company-place`, so every TR refresh left
the LOCAL `graph_edge` a vintage behind its own input and `graph.data.test.ts`'s bridge arm
failed — measured 498 procurement edges against 673 `company_politicians` rows (74%, under the
gate's 80% floor), 643 (95.5%) after a reload.
⚠️ **`db:load:persons-browse:pg` is NOT in that chain, and deliberately so.** This loader reads
`person_browse_table` for each person node's `primary_facet` / `position_type`, which its own
comment marks DESCRIPTIVE ONLY — not load-bearing for money or closure — and both derive from the
person layer, which a TR refresh does not touch. What a TR refresh DOES stale in
`person_browse_table` is its company counts, which this loader never reads. Running the graph
without a persons-browse first is therefore safe HERE and nowhere else: the ordering rule above
still binds on any path that moved `person_role`.
⚠️ **AND THAT FAILURE IS EASY TO MISREAD AS A PROD PROBLEM.** `graph.data.test.ts` calls
`pinLocalDatabase()`, so it measures LOCAL no matter what `DATABASE_URL` says — running it with the
cloud URL to "compare the two" returns the local numbers twice and looks like corpus-wide breakage.
Measure a cloud bridge with SQL, not with that gate. When it did fail this way, prod was at 95.5% the
whole time and the two arms mapped 85/85 and 561/561.
⚠️ **The bridge's mp arm strips `:<ns>` from the `person_role` side, not the ref side.** `person_role`
stores an MP as one row per (mp_id, ns) — `1005:39`, `1005:40` — so the join is
`split_part(pr.ref, ':', 1) = substring(cp.ref from '^/candidate/mp-(.*)$')`. Stripping on the other
side instead makes every MP who served in more than one National Assembly look unmapped (17/85
against a true 85/85). The collapse is safe only while no mp_id is recycled across parliaments —
0 today, and the roll-call corpus has 26, so re-measure rather than assume.
**643 edges against 673 rows is not loss.** 26 (person, company) pairs are named by more than one
`company_politicians` row — 21 of them the same human reached via BOTH an mp ref and an officials ref
— and `SELECT DISTINCT (person_id, eik)` folds each into one edge, which is one relationship rather
than two. The gate's own comment predicts "~95%". It
carries its own **per-arm bridge preflight** (mp / official) that throws — inside the rebuild tx, so a
broken `company_politicians.ref → person_id` join (a roster re-slug) rolls back rather than shipping a
half graph, and a **non-empty blob guard** that rolls back rather than publishing a blank `/connections`
overview. `graph.data.test.ts` / `graph_payloads.data.test.ts` fail on a stale/partial graph, a
money-basis drift from 127, a broken procurement arm, or a private-facet leak into the public blob.
Being a DERIVED serving layer (like `person_search` / `contractor_search`), it takes **no**
`recent_updates` row and no standalone `data/data-changes.json` entry — the `/data/updates` feed is
stamped per-skill by `process-watch-report`, and the source skills that trigger this reload already
stamp it.
**`--resolve` is not optional, and skipping it is invisible to a row count.** Phase 2 is what
fills `declaration.person_id`; without it that column is NULL on every row, the table is still
present and full, and `person_browse_table` publishes `has_declaration = false` for all 56,801
people — the "с декларация" filter matches nobody and its KPI reads 0%, while net worth keeps
rendering from `person_wealth_year`, so the page looks healthy. This shipped to prod once; the
loader's preflight now checks join-key population, not just row counts.
Skipping it after a _place_ loader blanks the same way `/person` does — the loader itself
throws if any placed row lost its label, but only for the rows it can see at build time.
`db:refresh` runs the local equivalent automatically; nothing runs it on the cloud side.
**A COLUMN-TYPE change in 120 also needs this run on the cloud.** The file is DROP + CREATE,
so Cloud SQL keeps the previous column types until the loader executes there — and a type
change is the one edit whose staleness is invisible to a row count. `net_worth_eur` /
`public_money_eur` / `delta_pct` are `double precision` for a reason: node-postgres
serializes PG `numeric` as a STRING, so on a stale cloud matview the API response looks
perfectly correct while every money cell on `/persons` renders BLANK.
Last of all, and the only step here whose output is a COMMITTED FILE rather than a table —
the `/person` prerender + sitemap manifest (`data/person/prerender_slugs.json`):
```bash
npm run person:slugs:cloud
```
It must run **after** `db:resolve:persons:cloud`, after `db:load:declarations:pg:cloud --
--resolve` (which is what REFRESHes `officials_rankings_table` — the `prerender` set and
every `card` come from it), and after **`person:slug-redirects:cloud`**: the continuity half
of the selection calls `officials_person_slug()`, which falls through to `person_slug_retired`
(`106_officials_redirect.sql`), so with the redirect maps unloaded a re-slugged official
resolves to nothing and silently drops out of the prerender set. It is the one manifest that
MUST be minted from the SERVING database — not from local Postgres.
`person_slug_lock` accumulates per database and is never truncated, so two databases
re-resolved a different number of times hand the same people different slugs; measured
2026-07-31: 1,436 mention→slug locks disagreed and 640 person slugs existed only locally
(mostly `-2` collision suffixes). Every one of those 640 was in the committed manifest,
naming a person prod cannot serve.
**Re-measured 2026-08-20, and the shape is unchanged — a resolve CANNOT close it, so do
not spend one trying.** The resolver's SOURCE tables are byte-identical on both databases
(`tr_companies` 1,020,707 · `tr_officers` 872,202 · `tr_person_roles` 1,340,793 ·
`tr_name_fold_people` 456,398); what differs is resolve-run HISTORY, and `person_slug_lock`
is part of that history rather than an input — same 143,521 keys on both, but **3,115 (2.2%)
locked to a different slug**, up from the 1,436 measured on 2026-07-31. `person_role` at
`source='tr'` is 192,374 local against 192,369 cloud — 5 rows, 2 people, 3 companies, 5
graph edges — and `person_slug_retired` diverges in BOTH directions: 849 slugs only local
(28 of them `-N` collision artifacts that only ever existed on that machine), 40 only on
cloud, and 533 shared slugs pointing at DIFFERENT targets. Since
`db:resolve:persons:cloud` re-mints against prod's own accumulated lock table and cannot
import local's identity decisions, running it would churn `/person` URLs without converging
anything — at the cost of the resolve plus the declarations phase-1/phase-2 and council
re-attach chain, during which 090's CASCADE leaves `/persons`, `/officials/assets`,
`/mp-assets` and `/declarations/crypto` at 500. ⚠️ **The runtime half of that argument is
dead: the resolve is 7m35s, not the ~37 min this paragraph used to quote** (measured
2026-08-29 — see below). What survives is the CASCADE outage and the churn, not the cost. Prod is self-consistent on its
own terms, which is the bar that matters: `person_slug_retired` = 24,910 rows, **0** without
a target, **0** targets missing from `person`, **0** chains.
**Re-confirmed 2026-08-27, after a contracts + TR publish, with the decision re-run from
scratch — the answer is still no, and these are the numbers to reuse rather than re-derive.**
Prod had resolved the day before (every `person.created_at` is a single date, 2026-08-26 — the
resolver DELETEs and re-COPYs, so that column dates the last cloud resolve), and
`person_slug_lock` holds 144,166 rows against 134,295 persons, i.e. every person is locked and
a re-resolve would reuse its slug.
- **The net gap is tiny and the gross divergence is not the same thing.** persons 134,370 local
vs 134,295 cloud (net −75), but **1,502 slugs local-only and 1,427 cloud-only** — the same
people under different slugs. **1,402 of the 1,502 are `-N` collision artifacts** that only
ever existed on the local machine.
- **Nobody who matters is unreachable.** Of the 100 non-artifact local-only slugs, 79 are not
public figures (Tier-V private owners) and 21 are — and **all 21 resolve on prod by
`display_name` under prod's own slug. Zero genuinely absent.** A "local-only slug" is not a
missing person, and testing by slug rather than by name is how you would talk yourself into
running this.
- **The manifest is clean: 0 of 63,836 prerender slugs are unservable on prod** — because it was
minted from prod by `person:slugs:cloud`. That is the LATENT risk below, measured, and not
currently realised.
- **Attribution is at parity**: `declaration.person_id` 61,743 and `council_vote.person_id`
43,261 on BOTH databases.
So the whole cost of not resolving was ~75 private individuals and ~218 tr/ngo roles (0.06% /
0.11%).
⚠️ **`graph_edge` is the surface where that drift shows up as a scary-looking number, and it
is the SAME finding — decompose it before investigating.** Re-measured 2026-08-28: local
**202,638** vs cloud **202,352**, a 286-edge gap. It decomposes exactly and needs no new
hypothesis:
| arm | local | cloud | gap |
| --- | ----- | ----- | --- |
| `tr_owner` | 92,208 | 92,000 | 208 |
| `tr_role` | 109,486 | 109,416 | 70 |
| `procurement` | 944 | 936 | 8 |
208 + 70 = **278**, which is EXACTLY the `person_role` gap at `source IN ('tr','ngo')`
(201,694 vs 201,416) — a 1:1 correspondence, so the owner/role arms carry no independent
divergence. The procurement arm's 8 sits against a `company_politicians` gap of 9, consistent
with the documented 26-pair `SELECT DISTINCT (person_id, eik)` fold.
Three things keep it benign, and the first is the one worth checking rather than assuming:
- **It is NOT a strict superset.** Diffed by natural key (`person.slug`, `eik`, `kind`) rather
than by `person_id` — which is a positional ordinal and differs between databases — there are
**404 local-only and 126 cloud-only** edges, netting 278. That two-sided shape is slug drift,
not loss.
- **Nobody who matters is unreachable.** The 404 local-only edges span 254 people: **223 are
not public figures** (Tier-V private owners) and **31 are — and all 31 are present on cloud
under prod's own slug**, checked by `display_name`. Zero genuinely absent. Same result as the
2026-08-27 sweep above, which is why testing by slug rather than by name would mislead here.
- **No money consequence.** `company_public_money` — the ONE broad per-EIK money basis, which
is what `graph_company_node` denormalizes — is **81,474 rows on BOTH** databases. The
divergence is in the person↔company edge set from resolve history, never in the money.
⚠️ **THE WHOLE CHAIN WAS RE-MEASURED 2026-08-29 on `db-perf-optimized-N-2`** (61,743
filings, 1,023,014-company TR corpus), end to end in manifest order — **18m14s for all 13
steps**, against the ~37 min this section used to quote for the resolve ALONE. The old
figure was measured 2026-08-11 on the retired `db-g1-small`; it over-stated by ~4×, and it
was being used as a reason not to run the chain:
| step | 2026-08-29 |
| --- | --- |
| `db:resolve:persons:cloud` | **7m 35s** (was 1733 s / 28.9 min on `db-g1-small`) |
| `db:load:declarations:pg:cloud -- --resolve` | 4m 52s |
| `db:load:person-search:pg:cloud` | 1m 23s |
| `db:load:person-elections:pg:cloud` | 1m 18s |
| `db:load:persons-browse:pg:cloud` | 56s |
| `db:load:graph:pg:cloud` | 44s |
| `db:load:declarations:pg:cloud` (phase 1) | 39s |
| `db:load:tr-company-place:pg:cloud` | 28s |
| `db:load:tr-name-fold-people:pg:cloud` | 7s |
| `place-dim` / `judicial-bodies` / `official-candidate-links` / `person:slugs:cloud` | 2–3s each |
**The CASCADE outage is what survives as an argument, not the runtime** — and note it was
NOT re-measured: nothing polled the four pages during the run. What IS verified is that they
were healthy afterwards (`persons`, `crypto_holdings`, `abroad_holdings`,
`officials_rankings` all HTTP 200), that phase 1 carried `filed_position`/`filed_institution`
for **61,743/61,743** filings, and that phase 2 left **0/61,743** `person_id` NULL.
**Trigger to actually run it:** an input to the person layer moves (`cacbg_officials`/`cacbg_local`
— phase 1 FIRST, `ivss_declarations`, `cik_results`, `erik_campaign_financing`, or a curated
sanctions/ДС/regulator edit), or cloud `person_role` at tr/ngo drifts past ~1% of local (~2,000
roles; it was 218). A TR daily flip on its own is not a trigger.
**That is LATENT, not live** — worth stating so nobody re-derives a panic from it. Both
consumers (`buildPersonRoutes`, the sitemap's `enumeratePersons`) filter on `prerender`, and
that ~5,000-entry ex-officials set was identical between the local- and cloud-minted
manifests (0 churn); nothing reads `indexable` at runtime. It goes live the moment the
prerender set widens to the full G6 set, which `emit_prerender_slugs.ts` explicitly plans.
Do NOT read that as "the two manifests differ only in slug identity". The same measurement
found 185 entries that kept their slug and flipped `indexable` (161 false→true, 24 true→false)
— slug-lock drift cannot cause that, since the floor reads `declaration` and `person_role`,
so the two databases disagreed on content too. Whether that was purely temporal (the
committed manifest predated the cloud catch-up) is not established.
`emit_prerender_slugs.ts` now REFUSES to write when connected to the local docker Postgres
(`npm run person:slugs -- --local` overrides), so `db:refresh`'s `person:slugs` step and the
`person_prerender_set.data.test.ts` determinism gate — both of which used to mint the file
from the stale side — warn and skip instead. Nothing else regenerates it: this command is
the only way the manifest moves.
**The `/en` person titles carry a BUILD-TIME dependency on two committed data files**, the
same shape as the `/court/**` one above and quieter still. `scripts/prerender/placeNameEn.ts`
resolves every English place name from `data/municipalities.json` + `data/settlements.json` at
build time; a missing or unparseable file degrades to transliteration, which is valid Latin
and therefore passes the "no Cyrillic" gate — 455 places (436 settlements + 19 municipalities
whose curated `name_en` differs from the transliterated form) silently change spelling in
indexed titles with nothing red. It warns on stderr; that warning is the only signal.
That module is also a **second producer of a label `place_dim` already owns**, against the
explicit "keep it that way" in `scripts/person/places.ts`. It has to be — the prerender is a
Node build step with no database, while the runtime reads `place_dim.name_en` through
`082_person_api.sql` — so the SAME page resolves the English place name twice, once per side
of hydration. `placeNameEn.test.ts` fails if the two dictionaries disagree on any shared code.
The single-producer fix is `pd.name_en` on the prerender card (the locals query in
`emit_prerender_slugs.ts` already `LEFT JOIN place_dim`), and it takes effect only at the next
`person:slugs:cloud` mint — the manifest is committed, so a card field changes nothing until
the file is re-minted from the serving database.
### The ЕТ owner, and the three fields the TR ingest used to drop
**`PhysicalPersonTrader` (FieldIdent 00180) — the natural person an ЕТ IS — was not parsed
at all until 2026-08-26.** `PERSON_SECTION_TO_ROLE` in `parse_daily_filing.ts` is an
ALLOWLIST rather than a generic sweep over SubDeed keys, so the section was silently
DROPPED, not mis-parsed: every ЕТ row in `tr_person_roles` was present and complete and
simply had no person, which is indistinguishable from the ordinary pre-2021 genesis gap the
CR Deeds capture exists for. Measured over all 1,686 daily files: **8,000 of 30,252 ЕТ carry
one already on disk**, against 30,052 ЕТ rows with no person at all. The other ~22k are the
real genesis gap and only a CR capture can reach them.
The role is **`sole_trader`, never `sole_owner`** — an ЕТ has no капитал and no дял, because
the trader and the firm are ONE legal subject. So it is OUT of `tr_owner_share` and
`isOwnerRole` (both apportion a declared capital that does not exist here) and IN the `OWNS`
fold in `personParticipations.ts`, which asks the different question: owner or manager.
Three things about it are easy to get backwards:
- ⚠️ **A NEW `TrRole` MEMBER IS NEVER A PARSER-ONLY CHANGE.** The token is copied VERBATIM
to the UI — `load_tr_pg.ts` into `tr_person_roles.role`, `tr_officers.roles` and
`company_politicians.relations`; `resolve_persons.ts` into `person_role.role` — and all
four consumers fall back to PRINTING THE RAW ASCII CODE when it is unmapped
(`trRoleLabel`/`trRoleList`, `relationLabel` in TWO copies, `OWNS`). So it also needs
`tr_role_<name>` + `procurement_rel_<name>` in both locales and a
`ProcurementRelationKind` member. This has shipped twice before (`sole_owner`,
`declared_role`).
- **The flat `Person` record key is SCOPED to the role**, not added to the global
`PERSON_RECORD_KEYS`. Un-scoped, "cannot double-count" is a fact about today's feed rather
than about the code: if TR ever flattens another section, the loop emits two
`person_added` events for one human, and they do not collapse in the replay either
(the wrapper record carries its own RecordID, the flat one falls back to the group's).
- **CR's `CR_F_18_L` is the same field**, verified against the committed `et.json` fixture
(`{nameCode: "CR_F_18_L", fieldIdent: "00180"}`) rather than inferred from the
`CR_F_<n>_L` ↔ `00<n>0` pattern. It currently reaches nothing — 0 of the 29,777 captures
are ЕТ — so it is there for a tier-2/3 capture.
**`SubjectOfActivity` (00060) — предмет на дейност — was the second drop**, and a quieter
one: parsed out of CR Deeds since that ingest was written (`subjectOfActivity`) and
persisted by NOTHING, so it reached no table and no page. It now runs parser → state →
sqlite → `tr_companies.subject_of_activity` → `/api/db/company` → the company page.
⚠️ **CR fills it ONLY into a NULL, the opposite precedence from this projection's persons.**
The persons are additive because the CR body carries no history and so can only add; a
SCALAR has no record_id namespace to coexist in, and here the DAILY FEED is the fresher
source — it re-states the field on every change while a capture is frozen at its
`fetched_at`. Unconditional, the projection would roll ~30k companies back to their capture
vintage.
⚠️⚠️ **A NEW COLUMN ON `tr_companies` IS A DEPLOY-ORDER DEPENDENCY, under a SQLSTATE the
serving code does not otherwise handle.** `db_routes.js` guards its schema-dependent arms on
42883/42P01 — a missing FUNCTION or RELATION. A missing COLUMN is **42703**, which is in no
degrade helper in that file, and these tables are ancient enough that an arm reading them
has never needed a guard. The `/api/db/company` arm is #1 of a ~30-way `Promise.all`, so
that rejection is a 500 on the WHOLE payload — every `/company/:eik` AND every
`/awarder/:eik`, since `CompanyDbScreen` serves both — while the function-served
`/company/**` HEAD keeps working, which makes it present as an intermittent front-end bug
rather than a migration gap. `load_tr_pg.ts` is 003's only loader-side applier and is
~280 s on cloud (re-measured 2026-08-27; the 34.9 min this line used to quote was the retired
`db-g1-small`), so a `deploy:db` landing first is a site-wide outage with a ~5-minute
floor on the recovery.
The arm now falls back to the pre-003 column list on 42703 and logs `co:no-subject-column`
once per process, so the ordering is cosmetic rather than breaking. Apply it anyway — it
does NOT need the loader, since `ADD COLUMN … text` with no default is catalog-only in
PG11+:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts 003_tr_search.sql
```
Then the corpus, which is what actually makes either field appear — the change is INERT
until the state is re-derived, the same shape as 089's backfills:
```bash
npm run tr:daily-refresh # replay → state.sqlite (+ the CR projection)
npm run db:load:tr:pg # local
npm run db:load:tr:pg:cloud # the serving database — nothing runs this automatically
```
The gates are `scripts/db/tests/tr_sole_trader.data.test.ts` (which SKIPS with a distinct
reason on a corpus predating the fix — that must never read as "the rule is enforced"),
`functions/db_routes.company_subject.test.js` for the 42703 degrade, and the parser tests in
`scripts/declarations/tr/`.
**`Седалище` (CR_F_5_L / CR_F_5a_L) was the THIRD drop, and the largest** — same shape again
(parsed since the ingest was written, persisted by nothing), but it feeds the whole PLACE
layer rather than one field on one page. `tr_company_place` resolves `tr_companies.seat` to
an EKATTE, and that is where every governance „фирми, регистрирани тук" tile, the
`/companies` oblast facet and the home flyover's contractor arcs get their oblast from.
Measured 2026-09-06: **693,932 of 1,023,673 companies (67.8%) carry no seat at all**, because
the daily feed only re-states the field when a company files — so a firm that has not filed
since 2021 is invisible to every place surface on the site. The 29,777 CR captures answer for
**29,417** of them, **26,152** of which are gaps.
⚠️ **IT STORES A CANONICALISED FORM, NEVER THE RAW CR TEXT, and this is the half that fails
silently.** CR renders the seat as a labelled block — „Държава: БЪЛГАРИЯ Област: Благоевград,
Община: Разлог Населено място: гр. Разлог, п.к. 2760 бул./ул. … Телефон: … Адрес на
електронна поща: …" — while the daily feed writes „БЪЛГАРИЯ, гр. Разлог, 2760". `parseSeat`
(`scripts/db/load_tr_company_place_pg.ts`) splits the feed's string on commas and takes field
1, so on a raw block it reads „Община: Разлог Населено място: гр. Разлог", resolves nothing,
and leaves the company **unplaced with its seat column full and every row count reconciling**.
`crSeatToFeedForm` (`parse_cr_deeds.ts`) rewrites it; `parsed.seat` keeps the original for
provenance and nothing stores it. Two further reasons the canonical form is the right one to
store: the raw carries a phone, fax and e-mail (7,867 of 29,417 captures carry a non-empty
electronic-mail label; 7,899 carry an @ anywhere in the block) into a column that
is rendered on `/company/:eik` and is a `companies` browse column, and it would make one
field mean two different things depending on which source filled it.
⚠️ **Same fill-if-null precedence as `subject_of_activity`, and here the cost of getting it
wrong is MEASURED rather than argued.** On the 3,247 companies where both the capture and the
feed carry a seat, the two resolve to the same EKATTE for **3,216 (99.05%)** — and every one
of the 31 that differ is a company that **MOVED** (Несебър→Равда, Пазарджик→Септември,
София→Пловдив). A capture is frozen at its `fetched_at`; the feed re-states on every change.
Writing unconditionally would relocate those businesses backwards, to an address they have
left, on a page that names them. That 99.05% agreement is also the only real validation the
canonicaliser has — it is checked against an independent source rather than against its own
fixtures.
⚠️ **`db:load:tr:pg` is NOT the end of the chain.** `seat` reaching `tr_companies` places
nobody; `tr_company_place` is written by a different loader. Local:
```bash
npm run tr:daily-refresh # replay → state.sqlite → the CR projection fills the gaps
npm run db:load:tr:pg
npm run db:load:tr-company-place:pg # ⚠️ the step that turns a seat into a place
```
Cloud, and nothing runs it automatically — the documented pair, in this order:
`npm run db:load:tr:pg:cloud` (280 s) then `npm run db:load:tr-company-place:pg:cloud` (28 s).
Skipping the second is the usual silent shape: prod's seats are current and its placements
are a vintage behind, at a 200.
`db:load:tr-company-place:pg` also REFRESHES `company_browse_table` (188), which LEFT JOINs
it for `/companies`' settlement / obshtina / oblast columns — so skipping it leaves that
facet a vintage behind as well as the governance tiles. Since 2026-09-06 it VACUUMs the
matview afterwards too; see the visibility-map section for why a CONCURRENT refresh needs
that at all.
⚠️⚠️ **`reconstructState` WIPES THE CR PROJECTION, AND THE TELL IS THAT `tr_companies` DOES
NOT MOVE.** It rewrites `state.sqlite` from the daily-filings replay and nothing else, so a
bare reconstruct — rather than the full `npm run tr:daily-refresh` — drops every projected
row and never restores it. `project_cr_deeds.ts`'s header states the rule ("must re-run
after each reconstruct, before `db:load:tr:pg`"); what it does not give you is how to
RECOGNISE the state afterwards, which is the expensive half.
Measured 2026-09-06, when exactly this happened: `tr_companies` was byte-identical at
1,023,673 across the loss while `tr_officers` fell 895,095 → 858,545 and `tr_person_roles`
1,373,571 → 1,309,082. A `db:resolve:persons` then propagated it and `person` went
134,451 → 114,241, which is what the gates finally reported — three steps downstream of the
cause, and reading like a person-layer regression.
**The one-line discriminator:**
```sql
SELECT coalesce(persons_source,'(feed)'), count(*) FROM company_persons GROUP BY 1;
```
All rows `(feed)` with **zero `'cr'`** means the projection was wiped, not that the corpus
changed. 1,373,571 − 1,309,082 = 64,489 is the whole projected set, and it is also where the
4,383 „Заличено обстоятелство." placeholders and every седалище gap-fill live — so the
placeholder count going to 0 and `tr_company_place` falling back to ~328k are the same event,
not three. Re-running the reconstruct reproduces the feed-only figure exactly, which is how
you confirm the feed half was never at fault before spending time in the parser.
The repair is `npm run tr:daily-refresh` (reconstruct → projection → the four PG loads), then
`db:resolve:persons` **and its nine-step repair chain** — the person layer does not come back
on its own. Do not go looking for a code defect first: the parser and projection carry 61
unit tests and a green run here proves nothing about whether the projection was APPLIED.
The gate is `scripts/db/tests/cr_seat_projection.data.test.ts`, which asserts the three
distinct failures separately — no seat is stored as a raw block, a sample of seats still
resolves through the real `EkatteResolver` (≥95%; the projected set measures 99.67%), and
`tr_company_place` actually grew. It SKIPS with a distinct reason on a corpus predating the
projection, keyed on the seated count against the pre-projection 329,741.
### `tr_owner_share` — the ONE definition of who owns what percentage of a company
`tr_owner_share` (a VIEW in `003_tr_search.sql`) answers "what fraction of this company
does this person own". FOUR serving surfaces read it and nothing else may read the stored
`tr_person_roles.share`: `company_officers()` and `person_roles()` (008), the
`company_person_roles` matview (022) and `mp_tr_roles()` (150). The view definition and
`tr_owner_share.data.test.ts` are the current specification.
⚠️ **`erased_at IS NULL` DOES NOT MEAN CURRENT.** The TR daily feed re-lists the WHOLE
partner set on every capital change and never erases the prior vintage, so a denominator
built from "every non-erased owner row" sums a company's cap table once per filing it has
ever made. Since the euro changeover it also added лв and EUR as bare numbers — a
re-denomination is filed as a new vintage. БИЛЯНА ООД (104119056) published **26% + 8%**
against a real **75.5% + 24.5%**. Measured 2026-08-20 before the fix: 10,400 companies
understated (mean 50.4%), 777 overstated (mean 200.8%).
**The rule:** the current cap table is the LATEST ACTIVE OWNER VINTAGE, and each owner's
share is their amount over that vintage's total, normalised to EUR at the peg. Validated
against `tr_companies.funds_amount` on the 11,502 multi-vintage companies that carry one —
the view's denominator reconciles for **10,923 (95.0%)** against **130 (1.1%)** for the raw
all-active sum it replaces (that is the pair the gate asserts). 351,981 of 356,209 companies
get a percentage and every one sums to 100% within the `round(…, 4)` residue.
Five things about it are easy to get backwards:
- ⚠️ **003 MUST BE APPLIED BEFORE 008 AND 022.** `person_roles`/`company_officers` are
`LANGUAGE sql` and 008 does NOT set `check_function_bodies = off`, so both bodies are
validated at CREATE and raise 42P01 on a database whose 003 predates the view, rolling
the whole file back. 022 gets no cover from its own `SET` either — a matview resolves
its query at creation regardless. `db:load:tr:pg` applies 003 → 008 → 022 in order; a
standalone function fix must name them together:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts \
003_tr_search.sql 008_connections.sql 022_company_officers.sql \
148_person_company_basis.sql 150_mp_tr_roles.sql 192_person_bridge.sql
```
**192 is in that list because it is a FOURTH serving function riding `db:load:tr:pg`** —
`person_person_bridge` (see "applied, never loaded" below). It is not the 150 trap: this one
the TR loader DOES apply. It is here so an operator shipping a `tr_owner_share` fix by hand
does not leave it on its old body, which is the same "one function got missed" shape.
⚠️ **150 IS THE ONE `db:load:tr:pg` DOES NOT APPLY** — its only applier in the repo
is `db:resolve:persons` (`resolve_persons.ts`'s SCHEMA_FILES). So a TR publish alone
leaves `mp_tr_roles()` on the old body, and `/api/db/mp-management` keeps publishing
the defect at a 200 beside a corrected `/company/:eik`. 148 must precede it in the
same command: 150's `LANGUAGE sql` body SELECTs `person_company_bridge_a`, which
only 148 creates.
- ⚠️ **A VIEW change leaves `company_person_roles` STALE, and nothing says so.** The
matview is built FROM the view, so `/company/:eik/officers` keeps serving the previous
rule at a 200 until it is refreshed. Prefer the REFRESH to re-applying 022: 022 opens
with `DROP MATERIALIZED VIEW`, and the `company_person_roles` DbDataTable resource reads
the base relation with no `missingMigration` degrade, so that page **500s for the whole
rebuild**. The unique index on `key` exists, so the refresh can be concurrent:
`REFRESH MATERIALIZED VIEW CONCURRENTLY company_person_roles;`
- **A CONSUMER WITH NO CONSTANT `uic` MUST CORRELATE, NEVER `LEFT JOIN`.** The view is
cheap only when the planner can push a `uic` into it. `company_officers(eik)` has one;
`person_roles(q)` filters on `name_fold` and has none, so a plain join made the planner
build all 455k view rows to answer for one person — **200,666 buffers / 1,465 ms**, on a
route a crawler walks under a 10 s `statement_timeout`. Correlated it is low hundreds
(83–137 measured across fixtures and cache states; the gate's ceiling is 5,000). EXPLAIN
any new consumer — a slow one looks identical to a fast one in the result, and the gate
anchors on the view's OWN scan node because `person_roles` also joins `tr_companies` on
`uic`, so a bare `Index Cond: (uic =` regex matches even the regressed form.
- **The rule exists TWICE and the two must agree.** `owner_share.ts`
(`scripts/declarations/tr/`) is the TypeScript twin, because the SQLite corpus is written
offline before Postgres exists. **[2026-08-21] It has NO rendering consumer left, and the
⚠️ below is now the whole reason it is still live code.** There were two, both retired by
`docs/plans/company-page-consolidation-v1.md`: `companies-index.json` via `integrate.ts`,
read by `/mp/company/:slug` (Tier 5.2), and `data/officials/derived/company_links.json` via
`build_officials_company_links.ts`, which became `pep_connected` →
`company_politicians.relations` (Tier 6). `pep_connected` now READS `company_politicians`
instead of feeding it, and that table's `relations` are derived in SQL from the person layer
— so no committed artifact renders `share_percent` any more. ⚠️ A twin change is still **INERT**
until the corpus is rebuilt — `npm run tr:daily-refresh` then `db:load:tr:pg` — and the
gate SKIPS with a distinct reason until then, which must never read as "the twins agree".
Note `project_cr_deeds.ts` is a SECOND writer into the same table and runs AFTER
`sqlite_writer`, so it re-derives over the merged row set; without that the stored value
and the view partition over different rows and cannot agree.
⚠️ **THE TWIN AND THE VIEW DO NOT SHARE A COLUMN, AND THAT IS WHY THE GATE COMPARES TWO
IMPLEMENTATIONS RATHER THAN CHECKING ONE.** The twin writes `company_persons.share_percent`,
which `load_tr_pg.ts` COPYs into **`tr_person_roles.share`** — and `tr_owner_share` never
reads that column. The view re-derives from `share_amount` + `share_currency` through
`tr_share_eur`. So they are two outputs of one rule, not one value computed twice, and a
fix applied to either alone leaves the other publishing the old answer at a 200.
Consequence for any retirement: killing a RENDERING consumer does not make the twin dead
code — `tr_person_roles.share` still comes from it. **[2026-08-21] That is no longer a
forecast: BOTH renderers are now gone (above), so `tr_person_roles.share` is the twin's ONLY
consumer — and the rule is still live.**
- ⚠️ **„Заличено обстоятелство." IS NOT A PERSON.** It is the register's deleted-fact
placeholder — 4,356 owner rows carry it, not one has an amount — and counting it as an
owner makes a two-owner company out of a one-owner one, refusing 4,299 lone sole owners
their correct 100%. BOTH implementations exclude it; excluding on one side only makes
them disagree about who the owners are, which is worse than the defect it fixes.
**Cloud side.** `npm run db:load:tr:pg:cloud` applies 003 → 008 → 022 in order and
refreshes the matview — but **NOT 150**, so it must be followed by either
`npm run db:resolve:persons:cloud` (a multi-hour rebuild) or the `apply_functions.ts`
command above, which is the cheap path. Then `npm run deploy:db` (the
`share_eur` column on the `company_person_roles` registry resource in `db_table.js`) and
`npm run deploy` (`formatOwnerShare`, which renders one decimal and must never re-acquire
the client-side `sole_owner → 100%` fallback the server now refuses).
**The gate is `scripts/db/tests/tr_owner_share.data.test.ts`** (22 tests), plus 23 unit
tests on the TS twin and 12 on the formatter. It carries a mutation check that deliberately
does NOT use "the shares sum to 100%" — the OLD broken denominator summed to 100% too, and
that is exactly why the defect survived — reconciling the view's own denominator against
registered capital instead. Two reference companies, because neither catches both defects:
БИЛЯНА's vintages carry identical лв:EUR proportions so its percentages are invariant under
the currency fold, and МИТОТОПИЯ (208164555) is the one where folding decides which partner
is the majority owner.
### SQL functions and indexes — applied, never loaded
A serving FUNCTION or an INDEX carries no data, so no `db:load:*` ships it. The person functions
(082/083/084/085/106 …) are the largest family, but the rule is general — 007's query builders and
081's indexes are in exactly the same position. The only cloud path that applies them is `db:resolve:persons:cloud`, which is a
multi-hour rebuild — far too heavy for a one-function fix, and it re-resolves the whole
identity layer as a side effect. Ship a function-body change on its own:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts 084_person_connections.sql
```
Safe at any time and idempotent — every one of these files is `CREATE OR REPLACE`. **A
function-only change is invisible to every row count and every loader**: local is green, prod
keeps running the previous body indefinitely, and nothing reports a difference. `deploy:db`
does NOT carry it — that ships function _code_ in `functions/`, which is a different thing
from a Postgres function.
**Both appliers of a DROP…CASCADE chain now carry a collateral-drop guard, and the loader's
fires on ABORT.** `scripts/db/lib/collateral_drop.ts` snapshots public relations either side
of an apply and reports any that vanished while some schema file still CREATEs them (so a
tombstoned retirement is silent). `apply_functions.ts` runs it as a post-condition and exits
1; `load_declarations_pg.ts --resolve` runs it in a **`finally`** around its 090 → … → 159
chain, then rethrows.
The `finally` is the load-bearing part. 090's CASCADE takes five relations that are
recreated later IN THE SAME RUN, so an INTERRUPTED run is the failure mode — and `exec()`
sends each file as its own transaction, so the CASCADE has already committed and there is
nothing to roll back. Measured on Cloud SQL 2026-08-19: a resolve got through 090 and 097
and died before 100, leaving `/persons`, `/officials/assets`, `/mp-assets` and
`/declarations/crypto` at 500 with nothing logged. **The survivor set is what dates it** —
`person_cohort_wealth` present and everything from 100 onward gone means the run stopped
between them; reproduced exactly by injecting a failure into 100.
A post-condition that only fires on SUCCESS is blind to precisely that case, which is why
this one does not.
**A migration a LOADER applies may not DROP an object another migration reads in a stored
query, and CASCADE is never the way out.** `db:load:pg` applies 077 on every contracts load;
077 used to open with an unconditional `DROP MATERIALIZED VIEW IF EXISTS
dual_corpus_rankings_cache` (present only so the `DROP FUNCTION` beneath it could run). When
migration 145's `funds_hub_stats_cache` started selecting that matview directly, every
`db:load:pg` began aborting with **2BP01 — in the APPLY phase, BEFORE the COPY**, so
`contracts` silently kept serving the previous vintage while the ingest that produced the new
shards reported success. It blocked every procurement publish from 2026-08-09 (900e50dd4b) to
2026-08-10, on prod as well as locally, with nothing red anywhere.
CASCADE would have been worse than the bug. `db:refresh` self-heals it (db:load:pg at step 5,
db:load:funds-fit:pg recreating the dependent at step 11) — but the documented procurement
publish path is a **standalone `db:load:pg:cloud`**, which would drop the dependent on prod
with nothing there to recreate it, blanking the `/funds` hub tiles indefinitely. The fix was
to drop the DROPs (neither was needed — the matview is a fixed one-column wrapper over the
function, and `CREATE OR REPLACE` rewrites the body in place) and to expose
`dual_corpus_company_count()`, a **plpgsql** wrapper whose body is never parsed for
dependencies, as the only supported way for another migration to read that cache **in a stored
query** — a view, a matview, or a function body Postgres parses at definition time. An ad-hoc
query records no dependency and needs no wrapper, which is why `/api/db/dual-corpus-rankings`
selects from the matview directly and is fine. plpgsql specifically: a `LANGUAGE sql` string
body records no edge today either, but the `BEGIN ATOMIC` form (PG14+) does, so modernising
such a wrapper would silently restore the 2BP01 — and note that a `BEGIN ATOMIC` dependency is
recorded through `pg_proc`, NOT `pg_rewrite`, so any "who depends on this?" probe must cover
both classes or it is blind to exactly that vector.
`dual_corpus_dependents.data.test.ts` gates both halves, for both matviews a loader-applied
migration DROPs (077's and 145's).
**CASCADE is the same rule with the failure mode inverted, and it is the more dangerous
half.** A DROP without CASCADE REFUSES — 2BP01, loud, the loader aborts. A DROP _with_ CASCADE
SUCCEEDS: it deletes the dependent and the loader exits 0, with nothing in its output and no
row count reporting the loss, because the counts that would move belong to a relation that no
longer exists. `003_tr_search.sql` did exactly that from the start — `DROP TABLE IF EXISTS
tr_companies CASCADE` (and the same for `tr_officers` / `tr_person_roles`), applied by
`load_tr_pg.ts` on **every** run — so every `db:load:tr:pg` deleted three matviews owned by
other migrations:
- **`person_browse_table` (120)** — the ENTIRE `/persons` browser
- **`declaration_stake_company` (096)** — the conflict-of-interest surface
- **`company_officer_counts` (071)** — read by `magistrate_politician_links()` and by 099
Same self-healing accident as 077: `db:refresh` sequences `db:load:persons-browse:pg` after
`db:load:tr:pg`, so a full local refresh hid it. The exposure is the **standalone
`db:load:tr:pg:cloud`** — the documented routine TR publish, which the sections above tell you
to run after a contracts/agri/funds reload and which `update-persons` / `update-procurement`
both invoke — which would drop `person_browse_table` on Cloud SQL with nothing there to
recreate it, serving `/persons` empty until someone happened to run
`db:load:persons-browse:pg:cloud`. (Checked 2026-08-10: prod holds all three, populated, and
their OIDs sit above the last TR load's, so nothing is currently orphaned there.) The same
CASCADE also ran on every local `npm run tr:daily-refresh`.
The fix is the one 077 took, adapted: **003 no longer DROPs anything.** `load_tr_pg.ts`
replaces the four tables' CONTENTS instead (`replaceTable` — TRUNCATE inside the COPY's own
transaction) and drops/rebuilds the eleven secondary indexes itself, which is what the DROP
TABLE used to give it for free. Three consequences worth knowing:
- **That TRUNCATE is a MEASURED lock regression, not a neutral swap** — 50 of 180 concurrent
probes rejected with `55P03` across a live load. Do not read it as "readers keep the
previous vintage": they are rejected for the duration. The old scheme did NOT block readers
(`exec()` wraps a migration in one implicit transaction, so the DROP+CREATE committed
atomically and the COPYs then took only `RowExclusiveLock`) — what it served instead was an
EMPTY table at a 200 for the length of the load, i.e. search confidently answering "no such
company". An error a route can degrade on beats that, which is why the trade was made, but a
cloud TR publish is **not** reader-safe: it is **280 s** (re-measured 2026-08-27 on
`db-perf-optimized-N-2`; this line read 34.9 min, measured on the retired `db-g1-small`,
and that figure was being used to argue against running it at all). Paid off by Phase 4b in
`docs/plans/cloud-deploy-speed-v1.md` (F21), which also records that an INTERRUPTED load
used to leave the tables populated but unindexed — the index drops now live inside each
table's own transaction, so an abort rolls them back.
- **Every column in 003 is written twice, on purpose.** `CREATE TABLE IF NOT EXISTS` is a
no-op on a warm database, so on its own it would trade the loud data loss for a quiet schema
drift — a new column reaching a fresh clone and nothing else. The reconcile block at the
foot of 003 (`ADD COLUMN IF NOT EXISTS`) is what actually reaches a warm database, and
`tr_search_shape.test.ts` fails when the two lists disagree. A TYPE or GENERATED-expression
change still needs a hand-written ALTER there (142 has a worked example).
- **The dependents now go STALE rather than missing**, which is the point. The loader refreshes
`declaration_stake_company` and `company_officer_counts` itself (both guarded on existence);
`person_browse_table` is deliberately left to its own loader — so **`db:load:tr:pg[:cloud]`
joins the trigger list for `db:load:persons-browse:pg[:cloud]`** below.
`migration_drop_dependents.data.test.ts` is the generic gate for the whole class: it reads
every `DROP` in `scripts/db/schema/pg/*.sql`, resolves each surviving target's stored-query
dependents through BOTH the `pg_rewrite` and `pg_proc` arms, and fails on any dependent owned
by a different file. Three pairs are sanctioned there **with their reasons** — `person_wealth_year`
(090, whose four victims are all re-applied by `load_declarations_pg.ts` on the same path),
and `appealed_ocids` / `upheld_ocids` (042, whose `contracts_list` is rebuilt by 042 itself and
whose `risk_upheld_ocid` is recreated inside `rebuild_contract_risk_cache()`). Anything else is
a defect. It also asserts the CASCADE is genuinely silent, so the gate cannot go vacuous.
**The reason nothing caught it is worth generalising: `db:refresh`'s only verification is its
LAST step.** `test:data` — which includes `pg_roundtrip.data.test.ts`, whose row-count assert
compares Postgres against the shards and would have failed on exactly this drift — sits at the
end of a 57-link `&&` chain whose fifth link was the one aborting. An early loader failure
therefore leaves the whole suite unrun, so the corpus that a loader failed to update is never
checked. When a loader aborts, run `npm run test:data` before assuming only that loader's
table is affected.
**`asset_share_multiplier()` (090) is another, and it is the one whose absence is
INVISIBLE.** A declared property's price is the WHOLE property — Сметна палата filing
instructions, table 1 col 11: „Посочва се цената на придобиване на имота/правото В ЦЯЛОСТ
… БЕЗ ДА СЕ ДЕЛИ МЕЖДУ СЪСОБСТВЕНИЦИТЕ" — and col 8 requires each co-owner on a SEPARATE
row repeating that same price, with only household members getting one. So a bare
`SUM(value_eur)` counts a jointly-held home once per co-owner. The rule lives twice
because a route cannot import TS: `assetShareMultiplier()` in `src/lib/declarations.ts`
(the JSON rollups behind `/officials/assets`, `/mp-cars` and the MP pages) and
`asset_share_multiplier()` in `090_person_wealth.sql` (`person_wealth_year` and the four
matviews over it). `asset_share_multiplier.data.test.ts` runs both over EVERY
`(share, category)` literal in the corpus rather than a hand-picked list — the column is
free text with ~3,200 spellings.
Three things about it are easy to get backwards:
- **A one-time 13–20% drop is the FIX, not a regression.** Measured 2026-08-15: executive
−16.1%, municipal −19.9%, MPs −13.3%, `/mp-cars` €8,241,472 → €7,109,059. Both skills'
troubleshooting tables name it, because `update-officials` previously told an operator
that a >20% drop meant a category-filter regression.
- **`security` is NEVER weighted.** On the table-9/10 forms that cell is a COUNT of дялове
(„369 476"), not a fraction; weighting it would multiply a shareholding by its own share
count. Nor are `debt` rows — the multiplier is a constant 1 there and is deliberately
written out rather than applied for symmetry.
- **Anything not an unambiguous proper fraction returns 1**, which is the safe direction.
„СИО", „по 1/2", „1/2-1/2" and „1/2+1/2" each already state the household's WHOLE
holding on one row; a bare „50" is unreadable as either a percentage or an ideal part;
„0" would zero a real asset. ~19% of declarants divide the price among co-owners anyway,
against the instruction, and are therefore UNDER-stated — kept deliberately, since it is
undetectable on a single-row holding.
Related, and the same shape one layer down: **`perSqmAnchor()`
(`scripts/declarations/parse_declaration.ts`) is the one definition of what a
price-per-m² is measured against** — column 6 (сградата) first, column 5 (парцела) as the
fallback — shared by the parse-time separator-typo detector and
`check_suspicious_values.ts`. Two anchors is exactly how a 36m² villa on a 980m² plot hid:
423,558/m² of building, 15,559/m² of plot. It returns the first USABLE area, not the first
present one, because a column-6 cell sometimes holds an ideal part rather than an area
(75 are fraction-shaped corpus-wide, 2 of them in a table-1 built-area position) — and
committing to one would suppress the plot fallback instead of using it.
**`shlyo_query_fold()` (141) is one of these, with one difference: it is GENERATED.** It is the
shliokavitsa half of search — the Latin-side spellings a Bulgarian actually types (`6umen`,
`4erven`, `sofiq`), which `translit_bg_latin()` alone cannot reach, so before it „Jelqzkov"
returned 0 rows from `person_search` while „Jelyazkov" returned 2. The `/api/db/*-search`
routes compose it with `translit_bg_latin()` on the QUERY side only; nothing stores its output.
Never hand-edit `141_shlyo_query_fold.sql`. It is emitted from `src/lib/shlyoRules.ts` — the
same table the browser's client-side filter uses — by `npm run gen:shlyo-sql`, and
`gen_sql/shlyo_query_fold.test.ts` fails when the two drift. That is the whole point: a rule
copied by hand into SQL means the browser finds „6umen" and the server does not, with both
looking like they work.
`db:load:person-search:pg` applies it, so `db:refresh` and
`npm run db:load:person-search:pg:cloud` carry it. But that loader is only the cheapest applier
in the chain, **not the only interested party** — `procurement-search` rides the same fold and
reads none of `person_search`'s tables, so a database where that loader has never run raises
42883 on it. Ship a rule change on its own with the usual escape hatch:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts 141_shlyo_query_fold.sql
```
`person_person_bridge` (migration 192) is the SECOND-DEGREE half of the person page's
„Проверка на връзка" — one bridge person joining two names that share no company, behind
`/api/db/connection`'s `bridged` array. **`db:load:tr:pg[:cloud]` is its ONLY applier**, after
008 — which owns the `officer_name_counts` matview it reads, and against which a `LANGUAGE sql`
body is validated at CREATE. A body fix ships on its own with:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts \
003_tr_search.sql 008_connections.sql 192_person_bridge.sql
```
Three things about it are easy to get backwards:
- **`db:load:tr:pg` is a `REFRESH_EXCLUSIONS` member, so `db:refresh` never applies 192.** On a
machine with a loaded TR corpus, `npm run test:data` therefore FAILS until someone runs the
34.9-minute TR loader or the command above. The gate names the fix in its message, so it is
survivable — but it is a red test on a fresh checkout rather than a skip, which is the one
thing that distinguishes it from every other entry in this section.
- **The route DEGRADES, in two different ways, and only one of them is a missing migration.**
`42883`/`42P01` → `bridged: []` plus a `ppb:not-built:<code>` log; `57014` (the pool's own
10 s timeout) → `bridged: []` plus `bridgedTimedOut: true`, which the UI renders as „проверката
не завърши" rather than as „no indirect link". Everything else still 500s. Grep those two keys
to tell „nothing found" from „192 never landed here" from „the plan on this database is too
slow" — three states that look identical in the payload otherwise.
- ⚠️ **Its Cloud SQL cost is UNMEASURED.** Every figure in 192's header is local, on the table
family this repo has a 4h41m incident with. EXPLAIN it against the serving database — under
`PREPARE`, since a psql literal constant-folds and hides the generic plan the pooled route
actually gets.
**⚠️ The search FOLD is outstanding on Cloud SQL as of 2026-08-20, and it is a two-part
change that must not be split across deploys.** `translit_bg_latin()` (000) gained two
fixes — `unaccent` now runs BEFORE the Cyrillic→Latin translate (it folds `ё` into a plain
Cyrillic `е`, which used to re-enter the output afterwards), and the mapping now covers the
Cyrillic HOMOGLYPHS that are not Bulgarian letters (`і` U+0456 alone occurs 2,155,780 times,
because ЦАИС writes „Раздел І:" with it). Applied to LOCAL Postgres and to nothing else.
Both halves go together, in one window, per database:
```bash
DATABASE_URL=… npx tsx scripts/db/apply_functions.ts 000_search_fns.sql 176_translit_homoglyph_refold.sql
psql "$DATABASE_URL" -c "VACUUM (ANALYZE, PARALLEL 0) tr_person_roles, tr_officers, tr_companies, contracts, tenders, contractor_search, awarder_search, person_search;"
DATABASE_URL=… npx tsx scripts/db/load_tender_dossier_pg.ts --refold
```
Splitting them is a REGRESSION, not merely a delay: every consumer folds the QUERY with the
new function while the stored `*_fold` columns still hold the old output, so a reader typing
the Cyrillic `І` stops matching the stored `і` (the query now folds to `i`) and typing `i`
does not match it either. Measured locally: 176 is ~5 min over ~4.7M rows, the VACUUM ~1 min,
and `--refold` **20 min** for 50,283 dossier bodies (`--dry-run` reports in 4 s). The VACUUM
cannot live in the migration — `exec()` sends a file as one implicit transaction and Postgres
refuses VACUUM there — so it is the step most likely to be skipped, which is why the seven
tables it covers now have `reload_visibility_map` entries.
**Three more of these are outstanding as of 2026-08-04.** Two were found because data tests kept
timing out under load — the tests were the symptom, the serving path was the defect; the third is
the molecule-page widening below.
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg \
npx tsx scripts/db/apply_functions.ts 007_query_builders.sql 081_person_identity.sql \
066_nzok_drug_quarterly.sql 054_nzok_risk.sql
```
- **`nzok_drug_molecule_detail()` (054)** keyed entirely on `nzok_drug_overpay_by_inn`, which is a
**top-30 leaderboard** — so `/molecule/:inn` returned NULL and rendered its not-found branch for
**580 of the 610** reimbursed INNs. It now returns two tiers: `spend` (from
`nzok_drug_quarterly`, present for all 610) and `overpay` (the above-median analysis, ~30). NULL
is reserved for an INN in neither source. **066 MUST be in the same command and BEFORE it** — 054
calls `nzok_drug_quarterly_by_inn()` from 066, and because 054 sets `check_function_bodies =
false` the missing dependency does not fail the apply: it fails on the first CALL with 42883,
which `missingMigrationEmpty` degrades to `[]` — a truthy value that skips the not-found branch
and renders "no above-median prices, the normal case" on every molecule page at a 200.
`sector_search_landing.data.test.ts` fails if any findable INN is unservable.
- **`recent_updates()` (007)** was **13.61 s at the route's default `(days=1, limit=200)`** — the
route clamps `limit` to 1–1000, so the endpoint was over Cloud Run's 10 s `statement_timeout` at
its most common shape, not merely under load. (The SQL function's own default is `lim=1000`;
the route's is 200.) Its five UNION branches had no per-branch limit
(1,688,150 rows materialised to top-N a few hundred) and one branch joined `changelog_days` on
`first_seen_at::date`, an expression no index serves. Now 0.15 s. Note 007 also rides
`db:load:tr:pg:cloud`, so a TR load carries it — but do not wait for one.
- **`idx_person_role_ref` (081)** — `person_role` had only `(source, ref)`, which cannot serve
`WHERE ref = $1`, so `officials_person_slug()`'s anti-join scanned the whole index per probe:
23,916 probes × 3.1 ms = **74 s**. With the index, 104 ms. Otherwise 081 is applied only by
`db:resolve:persons:cloud` (a multi-hour rebuild) and `scripts/person/add_override.ts`.
`084_person_connections.sql` is the worked example: `/api/db/person-connections` reached
8.2–10.1 s on prod (one request over the 10 s `statement_timeout`) because
`person_connections` rebuilt a whole-corpus company→officer-count map on every request,
independent of the subject — 96.5% of its buffers, and paid in full even by a person with no
companies at all, which is the common case since the traffic is a crawler walking
`/person/{slug}`. Fixed by a query rewrite (per-eik lookups riding
`idx_person_role_source_ref`), no new object and no loader.
`person_connections.data.test.ts` holds the buffer ceiling and proves it still discriminates
by restoring the old body in a rolled-back transaction.
**`date_basis` (081) and the roles payload (082) ship TOGETHER, 081 first, in one command.**
This is the counter-example to the rule above — it looks function-only and is not. 082's
`person_by_slug` selects `r.date_basis`, and a `LANGUAGE sql` body is validated at CREATE
time, so applying 082 to a database without the column fails the whole file with `42703`:
```bash
DATABASE_URL=postgres://postgres@127.0.0.1:5434/electionsbg npx tsx scripts/db/apply_functions.ts \
081_person_identity.sql 082_person_api.sql
```
Applying 081 on the cloud also runs its backfill (`source='mp'` → `'term'`, 1,522 rows),
which is what makes the office dates appear on prod without a multi-hour re-resolve.
The column has **two** writers and needs both: 081's backfill carries a warm database across
the gap, and `resolve_persons.ts` sets it in the `copyRows` list that rebuilds `person_role`
from scratch. Dropping it from that list is the silent failure — the resolve DELETEs the
table, every basis comes back NULL, and the renderer shows _nothing_ rather than something
wrong, so no page errors and no count moves. `person_role_date_basis.data.test.ts` fails on
a dated role with no basis, on the mp count collapsing, and on `person_by_slug` returning an
MP's terms in a tie order (the profile keeps the first row of a deduped seat, so an
unordered `jsonb_agg` would let the term shown change between two resolves of the same data).
**`person_role.bridge` / `bridge_footprint` (081) carry the SAME `copyRows` hazard with a
worse consequence, and unlike `date_basis` there is NO backfill to soften it.** They record
WHICH of the three licences a tr/ngo role was attached under (`'A'` curated ∪ `'B'`
people-unique public fold ∪ `'V'` money-linked private owner) and the footprint the cap was
measured against AT ATTACH TIME. Three writers, all in `resolve_persons.ts`, and all three
must stay: the `roleRows` COPY stamps `'A'` (a tr/ngo mention reaches it only through the
curated `linkedEiks`), and the Bridge-B and Tier-V INSERTs stamp `'B'`/`'V'` with their own
footprint. Both INSERTs carry `ON CONFLICT DO NOTHING`, so **the COPY running first is the
precedence rule** — a pair a register already licensed keeps `'A'` rather than being
relabelled by a name-based bridge that also matches it.
⚠️ **`bridge_footprint` is one column over TWO base tables.** Bridge B counts
`tr_person_roles` (`bridgeB.ts`'s `hits` CTE); Tier V counts `tr_officers` (its mint's
`count(DISTINCT o.uic) <= 5`). They agree on all 68,662 Tier-V folds today, which is exactly
why writing the wrong one would be invisible.
⚠️ **NULL means "attached before the columns existed", never "unlicensed", and 081 ships no
backfill on purpose** — the corpus vintage the resolver saw is unrecoverable (`tr_officers`
is TRUNCATE-and-reload with no history, and `ingest_first_seen` records new COMPANIES so it
is blind to a fold gaining an officer row at one that already existed). So the columns stay
NULL until the next `db:resolve:persons`.
**Why they exist, and the two questions they separate.** `person_resolve.data.test.ts` used
to re-derive each licence from seven tables that reload independently of `person_role`, so it
asserted that two corpora were the same vintage rather than an invariant — and went red at
443 roles purely because `db:load:tr:pg` ran two days after the last resolve, with the
resolver blameless (no person in the whole layer has ever held more than `FOOTPRINT_CAP`
distinct EIKs). It now asks "was this licensed WHEN ATTACHED?" against the stored value, and
skips with its own reason on a corpus no resolve has stamped.
`person_role_bridge_freshness.data.test.ts` asks the other half — "do those licences still
rest on premises that hold?" — prints the drift on every run, and fails only past a share
argued from the measured distribution. It needs no stamped column (it falls back to the
attached footprint), so the signal survives the gap before the first resolve.
Four gates, and only one runs in CI: `person_resolve` (stored licence + identity class),
`person_role_bridge` (schema + constraint semantics, pinned by inserting each row shape — a
CHECK that fails open on NULL looks identical to a correct one in review),
`person_role_bridge_freshness` (drift), and `resolve_persons_bridge_columns.test.ts` — the
copyRows list, static over the source, and the only one that needs no Postgres.
### The visibility map a TRUNCATE-reload throws away — and the one cloud repair it needs
A loader that rebuilds a table with `TRUNCATE` + `INSERT`/`COPY` inside ONE transaction leaves
`relallvisible = 0` **permanently**, so Postgres cannot plan an index-only scan on it ever
again. TRUNCATE mints a new relfilenode with an empty map; every page is then written by a
transaction that has not committed, so nothing can be marked all-visible; and the
insert-threshold autovacuum that fires afterwards runs mid-`db:refresh` where a concurrent step
holds back the xmin horizon — it marks nothing, resets `n_ins_since_vacuum` to 0, and with
`n_dead_tup` also 0 never revisits the table. The fix is `vacuumAfterReload()`
(`scripts/db/lib/pg.ts`), called after the load's COMMIT — never inside `withTx`, since VACUUM
cannot run in a transaction block. The wired tables are listed in
`reload_visibility_map.data.test.ts` — which reads the loaders' own call sites and fails if any
vacuumed table is missing from that list, so the two cannot drift. `shipTable()` vacuums its
destination itself, so a new `TRUNCATE_SQL` entry is covered without touching its caller.
**That gate checks ONE direction, and the other one is the hole this keeps falling through.**
It asserts "every table a loader vacuums is listed here", so a loader that vacuums NOTHING
contributes no names and is invisible to it — which is exactly how `db:load:interreg:pg` went
unnoticed. Its file list is now derived from a glob over `scripts/db/load_*.ts` rather than
hand-maintained (that alone surfaced two more unlisted call sites, `budget_peer_band` and
`tr_name_fold_people`), but deriving it does not close the converse: asserting that needs an
independent source of "this table is bulk-reloaded", and a whole-database sweep for short maps
reports 22 tables today, mostly a few pages each where no index-only scan is worth planning. So
when adding a loader, check the call by hand — nothing will fail if you forget it.
**This is the rare defect that is invisible from every angle a reviewer normally checks**: row
counts reconcile, the corpus is correct, the migration is untouched, and the plan is still
_named_ an Index Only Scan — it just reports `Heap Fetches: <every row>`. Both instances found
so far were found by accident, and the first was initially read as a function-body regression in
a file nobody had edited. `contracts` is the counter-example that locates the cause: it is
stage-MERGEd rather than truncated, and its map survives a reload intact.
⚠️ **Do NOT read that counter-example as "stage-merged tables are safe" — they are not, and
`interreg_partners` is the proof.** All three Interreg tables are stage-merged, and the map
still ends up short: the merge's anti-join DELETE and its UPDATEs leave dead tuples, and
neither autovacuum threshold reaches them — the dead-tuple one is a 20% fraction a few hundred
rows out of twelve thousand never crosses, and the insert-threshold one fires mid-`db:refresh`
under a held-back xmin horizon, marks nothing, resets `n_ins_since_vacuum` and never returns.
Measured 2026-08-15 after an ordinary `db:load:interreg:pg`: `interreg_partners` at **130 of
474 pages (27%)**, 651 dead tuples, `last_vacuum` and `last_autovacuum` both NULL, while
`interreg_operations` happened to get autovacuumed and so looked fine. That broke a committed
gate rather than merely costing time — `funds_fit.data.test.ts`'s "the resolver stays cheap
enough to serve live" failed at **6,251 buffers against its 6,000 ceiling** — and the corpus had
SHRUNK across that reload (12,141 → 12,015 partnerships), so the extra buffers were bloat and a
stale map rather than data growth, which is why no row count reported it. What `contracts`
actually shows is that a big, continuously-autovacuumed table survives; size and traffic are
doing that work, not the merge shape. Wire the call regardless of shape.
⚠️ **AND A `REFRESH MATERIALIZED VIEW CONCURRENTLY` IS A THIRD SHAPE, not a fourth exemption.**
A concurrent refresh does not rewrite the heap — it DIFFS into the existing one — so it leaves
dead tuples exactly as a stage merge does, and the same two thresholds miss them. Measured
2026-09-06 on `company_browse_table` after an ordinary `tr:daily-refresh` chain: **8,921 of
24,791 pages**, with `/companies`' `contractor_total_eur` sort at `OFFSET 20000` reading
**644,916 buffers** against the ~20,000 the 193 indexes exist to give — a gate
(`company_browse.data.test.ts`) failing for a reason no row count could show. Four loaders
refresh that matview (`db:load:declarations:pg -- --resolve`, `db:load:pg`,
`db:load:graph:pg`, `db:load:tr-company-place:pg`) and all four now vacuum it.
⚠️ **The vacuum must be GUARDED on the refresh's return value.** `refreshMatviewConcurrently`
returns `false` and does nothing when the matview has never been applied — which is the whole
point of those call sites — while `vacuumAfterReload` has no such probe, and VACUUM on an
absent relation is **42P01**. Unconditional, it aborts precisely the databases the refresh is
written to survive: a cold `db:refresh` (188's only create path is step 55; `db:load:pg` is
step 6) and any standalone `:cloud` publish against a database without it.
**A bare `ANALYZE` is not half the fix — it is the disguise.** `db:load:graph:pg` ran
`ANALYZE graph_edge, graph_company_node, graph_person_node` after its merge, which stamps
`last_analyze` and never touches the visibility map, so `graph_company_node` sat at **20 of
1,174 pages (1.7%)** with 6,087 dead tuples and `last_autovacuum` NULL while looking freshly
maintained. Its two siblings were healthy (3,770/3,770 and 3,592/3,665) only because autovacuum
had happened to reach them — so the one table that needed it was the one that looked least
suspicious. Measured on the top-N by money the loader itself issues (`GLOBAL_COMPANY_CAP`, 150
rows): `Index Only Scan … Heap Fetches: 208`, 170 buffers, 7.8 ms → `Heap Fetches: 0`, 5
buffers, 0.18 ms. Use `vacuumAfterReload()`, never a bare ANALYZE, after a bulk rewrite.
**And plain VACUUM is not the whole fix either when a loader rewrites EVERY row — use
`compactAfterReload()`.** VACUUM makes dead space reusable but never returns pages to the
filesystem, so a merge that touches all rows settles at ~2x live size PERMANENTLY. It is
stable, so no growth alarm fires, `n_dead_tup` sits at 0 and the visibility map reads 100% —
every signal this section describes looks healthy. Measured on `council_resolution`, whose
upsert carries `last_seen_at` in the DO UPDATE arm so `stageUpsertSql`'s `IS DISTINCT FROM`
can never short-circuit: **224 pages compacted, 439 after ONE ordinary `db:load:council:pg`,
then 441 / 442 / 442** over three more runs. That cost `council_overview()` ~444 buffers of
its 1,500 ceiling — the gate still PASSED, at 7% headroom, which is why it went unnoticed;
after compaction it is 945.
⚠️ **Never do the `VACUUM FULL` on its own.** It rewrites the heap into a new relfilenode
whose visibility map is EMPTY, i.e. it trades the bloat for exactly the defect this section
is about. Measured doing only the FULL: `count(*) FROM council_vote` went **44 buffers → 790**,
`council_overview()` **953 → 2,502**, and `reload_visibility_map.data.test.ts` failed on both
council tables. `compactAfterReload()` exists so that cannot be got wrong — it is FULL followed
by `vacuumAfterReload()`, and it is capped at `COMPACT_MAX_BYTES` (32 MB, ~1 s of
AccessExclusiveLock at the measured ~33 ms/MB) so the lock can never grow with the corpus.
Above the cap it warns and leaves the slack.
Two things follow for anyone adding a loader. Compaction is deliberately UNCONDITIONAL below
the cap rather than gated on "did the heap grow this run" — at steady state it does NOT grow
(441 → 442 → 442), so that gate reads healthy on precisely the state worth fixing. And
`reload_visibility_map.data.test.ts`'s call-site scan matches **both** helper names: it
matched only `vacuumAfterReload` until 2026-08-20, so the day the council loader switched, its
four tables silently dropped out of the scan while the per-table assertions kept passing off
the hand-written `RELOADED` list — a gate going half-blind with nothing red.
**Cloud SQL carries the same exposure, and `tenders` is the one that costs something.** Migration
113 exists to make the `/procurement/tenders` browser's count+sum and its two facet GROUP BYs
Index-Only Scans over `idx_tenders_order`, and `db_table.js` routes them at the base table rather
than the `tenders_list` view as the other half of that fix. With an empty map the whole
optimisation is given back silently — measured locally 2026-08-11 on the default scope, **5,047
buffers with `Heap Fetches: 6088`, against 87 and `Heap Fetches: 0` after**. Prod is a
reading cold over the proxy under a 10 s `statement_timeout`, so it is worse there (that gap was
sized on the old db-g1-small; the box is bigger now, the direction is not).
Every `:cloud` loader run now vacuums — including the ones that publish by `shipTable()`, whose
`company_founded` was the one destination no caller covered and which no LOCAL gate can ever see,
because the local copy of that table is upserted rather than truncated. But a database loaded
before this shipped stays in the bad state until its next reload, and that reload is ~90 min for
contracts, ~20 for tenders. Repair it directly instead (safe any time, and the tenders one is
~2.5 s per 42k pages). The list is the one in `reload_visibility_map.data.test.ts`'s `RELOADED`
— that gate is what keeps it in step with the loaders' actual `vacuumAfterReload` calls, so a
new entry belongs in BOTH. ⚠️ **One absent relation aborts the whole statement** (42P01, nothing
vacuumed): several of these come from gitignored-input loaders that legitimately never ran on a
given database, so on a partial one drop the names it lacks rather than reading the error as a
failed repair. `company_founded` is the one member with no `RELOADED` entry — `shipTable()`
passes the table as a VARIABLE, which the gate's string-literal scan cannot see:
```bash
psql "$DATABASE_URL" -c "VACUUM (ANALYZE, PARALLEL 0) declaration_employer_link, grant_contract_link, tender_subcontracting, ted_notice, ted_coverage, adfi_inspection, aop_expert, aop_expert_area, isun_clean_contract, isun_clean_beneficiary, adfi_coverage, obshtina_population, fund_projects, fund_beneficiaries, company_founded, tenders, tender_normalcy_cache, procurement_normalcy_cache, procurement_annexes, tender_search_text, cprs_firm, cprs_licence, nzok_activities, nzok_activity_facility_periods, nzok_activity_proc_periods, nzok_activity_monthly, budget_fiscal_year, budget_fiscal_year_figure, budget_kfp_observation, budget_kfp_snapshot_section, budget_kfp_snapshot_line, budget_personnel, budget_admin_procurement, budget_muni_transfer, budget_muni_ipop_project, budget_muni_capital_project, budget_muni_execution, interreg_operations, interreg_partners, interreg_programmes, budget_peer_band, tr_name_fold_people, graph_edge, graph_company_node, graph_person_node, graph_payloads, council_muni, council_muni_code, council_resolution, council_vote, agri_subsidies, agri_payloads, agri_beneficiary, agri_beneficiary_year, agri_scheme_year, agri_hub_stats_cache, agri_political_link, agri_cross_programme, price_last_seen, price_current, vote_item, vote_cast, mp_seat, party_dim, mp_attendance, party_cohesion, mp_dissent, mp_similarity, mp_vote_norm, person_browse_table, company_browse_table, transport_project_link;"
```
`budget_admin_procurement` (157) is the odd one in that list: it is written by THREE
loaders — `db:load:budget:pg` (its own dimension), `db:load:pg` (the contracts corpus)
and `db:load:tr:pg` (`company_politicians`) — and its DELETE+INSERT rebuild runs inside
one transaction, so each of them has to vacuum it. All three do.
Two things about the repair are easy to get backwards:
- **`PARALLEL 0` — KEEP IT, but the reason has changed.** It was originally required: parallel
vacuum allocates one DSM segment up front, Docker's `/dev/shm` default is 64 MB, and
`VACUUM (ANALYZE) tenders` (14 indexes) died with `could not resize shared memory segment …
to 67145792 bytes`. **Since 2026-08-25 `docker-compose.yml` sets `shm_size: 1gb`, so that
specific failure is gone** — do not read an old "it dies without PARALLEL 0" note as current,
and do not "fix" a fresh clone by removing the flag either. It stays because nothing is lost
by it: VACUUM parallelises the index-vacuum phase only, and a freshly reloaded table has
`n_dead_tup = 0`, so that phase has no work to do. `vacuumAfterReload` passes it.
⚠ **The 64 MB default is a REPO-WIDE trap, not a vacuum one, and raising it is what actually
fixed the recurring symptom.** Postgres puts every parallel query's DSM segment in `/dev/shm`
too, and `npm run test:data` runs ~16 concurrent vitest workers over corpora in the hundreds of
thousands of rows. Measured 2026-08-25 on the full suite: two tests in
`official_companies.data.test.ts` failed with `could not resize shared memory segment … to
8388608 bytes: No space left on device`, and **both passed when the file was re-run alone** —
the [[test-data-flaky-under-load]] shape exactly. It reads as a data defect, it is
load-dependent, and it lands on whichever test was unlucky. A container created before that
commit still has 64 MB; `docker compose up -d` recreates it (the `pgdata` named volume
persists, so no reload is needed) and `docker exec electionsbg-pg df -h /dev/shm` confirms.
- **A VACUUM run while any long transaction is open marks NOTHING and still reports success.**
It is the same held-back-horizon mechanism that defeats autovacuum. Measured on a standalone
`db:load:nzok-activities:pg` with a concurrent `db:resolve:persons` 15 minutes into one
snapshot: `last_vacuum` stamped, `relallvisible` still 0, loader exit 0. `vacuumAfterReload`
therefore reads the map back and WARNS with the blocking pid rather than throwing — the load
has already committed and the data is fine, so the shortfall is worth reporting, not worth
aborting a `db:refresh` chain over. On Cloud SQL, which serves traffic continuously, expect
this to fire occasionally; re-run the VACUUM above when it does.
## Testing
Two layers: **Vitest** for unit + component tests (`npm run test:unit`), **Playwright** for E2E/SEO/perf smoke (`npm test`). Co-locate tests as `*.test.ts(x)` next to the module. Unit tests never touch the network (an unstubbed `fetch` throws in jsdom) or a live DB; the `scripts/db/tests/*.data.test.ts` Postgres gates are the exception and auto-skip when Postgres is down. The `functions/` package keeps its own `node --test` gate (`npm run functions:test`). Full convention — what to unit- vs component-test, fixtures, determinism, coverage, CI placement — is in [docs/testing-standards.md](docs/testing-standards.md).
**Two of those Vitest files are static-analysis gates over the SOURCES rather than
tests of a module, and both exist because the failure they catch is invisible in
review and expensive to catch any other way.** Each is seconds; the alternative is
a ~10-minute `vite build` that reports only that a number moved.
- **`src/entryGraph.test.ts`** — walks the static import graph from `main.tsx` and
fails if it reaches a sector registry, or anything a registry names. The rule it
encodes: **take a constant from an import-free module, never from a registry.**
`routes.tsx` imported one path string from `sectorPacks` and thereby put ~20
reference-data modules (~265 KB of source) into the entry chunk that every page
downloads. The byte budget in `tests/perf.spec.ts` caught the 587 B that pushed it
over; it could not say which edge did it. The forbidden set is DERIVED from the
registries' own closure, so a new pack is covered the day it is added.
- **`scripts/i18n/key_usage.test.ts`** + `npm run i18n:prune` — fails when the
corpora accumulate keys no call site can ask for. A key counts as reachable if it
appears as a literal, matches a built template (`` `pp_reg_seat_${seat}` ``),
matches a family prefix, or is a plural of a used base — and the scan covers
`scripts/` and `data/*.json`, because a key can reach the UI through a DATA
ARTIFACT rather than a call site. The prune deletes translated copy, so it is
dry-run by default and deliberately in no chain.
Both share `scripts/lib/strip_comments.ts`: prose that MENTIONS a pattern is not an
occurrence of it, and each gate has already been burned by a naive strip — in
opposite directions. Read that file's header before touching either.
## Architecture
### Tech Stack
- **React 19** + **TypeScript** (strict mode)
- **Vite 6** with SWC
- **React Router v7** — 58 routes, all wrapped in a `<Layout>` component
- **TanStack React Query v5** — all data fetching, `staleTime: Infinity`, no refetch on focus
- **Tailwind CSS** + CSS Modules — HSL color system via CSS variables
- **Radix UI** primitives with shadcn-style wrappers in `src/components/ui/`
- **Recharts**, **React D3 Library**, **Leaflet** — charts and maps
- **i18next** — English/Bulgarian, preference stored in localStorage
- **Firebase Hosting** — SPA rewrites configured in `firebase.json`
### Data Flow
1. **Raw data** lives in `/raw_data/` (CSVs, ZIPs from the electoral commission)
2. **Data pipeline** (`scripts/`) transforms raw data → static JSON files in `/public/YYYY_MM_DD/`
3. **Frontend** fetches JSON from `/public/YYYY_MM_DD/*.json` based on the election date selected in `ElectionContext`
### Key Source Areas
- `src/routes.tsx` — All route definitions
- `src/data/ElectionContext.tsx` — Central state: selected election date, aggregate stats. All data hooks read the selected date from this context.
- `src/data/macro/cabinetAnchorContext.tsx` — Global cabinet anchor mounted on the `/governments*` and `/indicators*` route group. URL-encoded via `?cabinet=<id>`; every quarterly/annual snapshot hook (`useElectionAsOf`, `useElectionYear`) consults the override and re-anchors to the cabinet's tenure end. Cleared via the header pill ×.
- `src/data/` subdirectories (`regions/`, `municipalities/`, `settlements/`, `sections/`, `parties/`, etc.) — React Query hooks per domain; each exports typed `useXxx()` hooks
- `src/screens/` — Page-level components matching the route structure
- `src/screens/components/` — Reusable components shared across screens
- `src/components/ui/` — Low-level UI primitives (22 components)
- `src/ux/` — UX utilities: data tables, tooltips, touch handling, media queries
- `src/locales/{bg,en}/` — the ONLY i18n corpora, one chunk per language via
`import()` in `src/i18n.ts`. There is no `public/locales/`; this line named one
until 2026-08-18, which would send anyone auditing or pruning the corpus to a
tree that does not exist — or make them conclude a served copy had been missed.
**It is ONE flat i18next namespace PARTITIONED across several files, not
several namespaces.** `translation.json` is the core corpus every page
downloads before it can paint; each of the others (`budget.json`,
`methodology.json` — the list is `src/locales/bundles.ts`) is a DEFERRED BUNDLE
that ships with the routes tagged `withBundle("<name>", …)` in `src/routes.tsx`
and is merged into the same namespace with `addResourceBundle`. So every call
site stays a plain `t("budget_hub_title")`, and there is no namespace for a
component to forget to declare.
Four things about it are easy to get backwards:
- **Read the UNION, never `translation.json` alone.** `loadCorpus()`
(`scripts/i18n/key_usage.ts`) does; so does the test-only
`src/locales/allKeys.ts`, which is what component tests import. Reading the
core file alone reports every bundled key as dead — and
`prune_translations.ts` would then delete them. That script writes back per
FILE for the same reason: writing the kept set to `translation.json` folds
every bundle into the core chunk, hands the byte budgets their whole overage
back, and every key count reconciles.
- **Which keys a bundle may hold is DERIVED, not curated.**
`scripts/i18n/bundles.ts` proves a key is named only by modules that no
route outside the bundle can statically reach; `split_bundles.ts --apply`
moves them (bidirectionally — a key that stops being exclusive comes back);
`bundle_reachability.test.ts` keeps it true. Adding a bundle is: name it in
`bundles.ts`, tag its routes, run the splitter, re-ratchet
`tests/perf.spec.ts`.
- **The failure is silent by construction.** A key whose bundle is not loaded
renders as its own identifier at a 200 — `budget_hub_title` where a heading
belongs. That is why the gate has no allowlist, why the analysis is biased
toward core in every ambiguous case (named outside `src/`, named by a data
artifact, reachable from the shell or from an untagged route → core), and why
`i18n.ts` heals a missing key by pulling every bundle rather than reporting.
- **A language switch must carry the loaded bundles over.** `changeLanguage`
re-merges everything this session has asked for BEFORE switching; without it
a visitor who switches language while reading `/budget/execution` watches the
page turn into identifiers, on the happy path, with nothing failing.
Measured 2026-08-22 at the split (brotli q11): core bg 147,845 → 114,785 and
en 129,418 → 99,559, i.e. −22% with 1,119 of 6,253 keys deferred. The corpus
grows ~1-2 KB br per language per day, so the budgets in `tests/perf.spec.ts`
will trip again; the next lever is another bundle (funds and procurement are
the two largest measured at ~7 KB br per language each), not a wider number.
### URL contract (cross-page state)
- `?elections=YYYY_MM_DD` — selected election (read by `ElectionContext`)
- `?cabinet=<id>` — global cabinet anchor on `/governments*` and `/indicators*` (read by `cabinetAnchorContext`)
- `?peers=RO,GR,HU,HR` — peer-country selection on `/indicators/compare` (read by `usePeerSelection`)
- `?pscope=all` / `?pscope=y:2024` — the shared time-scope on `/procurement*`, the sector views (water/defense/culture/judiciary), the farm-subsidy pages `/subsidies*`, and the sectors hub (all read by `useScope` in `src/data/scope/`; the param name stays `pscope`); default `ns` (the selected parliament's contract window) is omitted from the URL, `all` pivots to the full corpus, `y:<year>` to one calendar year. On `/culture` `ns` means "all years" (relabeled) and `y:<year>` re-aggregates the film KPIs / discipline split / concentration / awards to that year client-side from `films.json` — the time-spine (a historical trend) stays full-history (`scopeCultureOverview`). The procurement nav pills + the subsidies dashboard's tile links carry the current search forward (`useScopedHref` / a local `browseTo`) so the scope survives sub-page navigation. On `/subsidies` (which has no per-parliament slice) `ns` resolves to the latest financial year via `agriScopeToKey`, and each scope is served as its own precomputed `agri_payloads` overview blob (kind='overview', key=`<year>`|`all`|`''`); the `?pscope` year picker there lists only the CAP financial years present (`AGRI_FINANCIAL_YEARS`).
**A page narrower than the corpus MUST resolve the inbound scope.** `pscope` is in the `usePreserveParams` allowlist, so any in-app link carries a scope minted where it was valid onto a page that cannot serve it (`y:2026` on `/culture`, whose НФЦ register ends 2025; `y:2019` on `/subsidies`, a hole in the CAP corpus; anything past the last Доклад on `/sector/administration`). Pass the page's coverage to the hook — `useScope({ years, allowAll })` — and hand `<ScopeControl>` the SAME resolved value via `value`/`onChange`; then the pill and the numbers are one value. Two failure modes this closes, both of which look like real data: an unresolved year reaching the aggregation (`selYear` on `/administration` labelled the latest report's numbers "2026") and a Radix `<Select>` whose controlled value matches no `<SelectItem>`, which renders EMPTY — not the placeholder — leaving the whole widget reading as the page default. Falling back to `ns` is not the only honest option: `/subsidies` keeps the raw scope and NAMES the gap ("Няма данни за субсидии за 2019"), which is why `ScopeControl` displays an off-list year rather than inventing one. What no page may do is show one window and count another.
- `?q=<term>` — on `/procurement/contracts`, `/procurement/tenders` and `/persons`, seeds the DbDataTable free-text search (used by the combined-search "see all" deep links).
- `?facet` / `?pfacet` / `?role` / `?party` / `?oblast` / `?obshtina` / `?court` / `?decl` / `?held` — the `/persons` browser filters, all owned by `useUrlPersonFilters` (`src/data/persons/`); `?q` seeds its search box. Two distinctions are load-bearing and easy to get backwards. **`?facet` is MEMBERSHIP, `?pfacet` is the PRIMARY facet** — the first asks "is this person also a …" and filters a boolean flag, the second asks "what is this person mainly" and filters the single-valued `primary_facet` the mix bar partitions. And **the multi-valued dimensions filter a space-padded CODE SET, never the display scalar beside it**: `?party=gerb` means "ever affiliated" (keeping the 4,723 switchers), `?oblast=VAR` means "holds any role there" — matching the representative column instead would drop 1,851 people from an oblast they genuinely serve, which reads as "no such people" rather than as a narrowed view. `?obshtina` is the ONE exception: it filters the representative seat, because there is no obshtina code-set column. `?court` carries an institution NAME (the picker facets and filters the same `institution` column, so its counts are exact and no code→name dictionary is needed). Every value is validated on read.
- `?proc` / `?cpv` / `?single` / `?cancelled` / `?year` / `?grade` — the procurement browser filters, all owned by `useUrlProcurementFilters` (`src/data/procurement/`). `?proc` is a bucketed procedure, `?cpv` a division/prefix/comma-set, `?single|?cancelled` a boolean toggle (name differs per browser), `?year` the company/awarder page only. `?grade=D,E,F` is a validated A–F set filtering the **server-side** contract risk index (`risk_grade`, migration 112) — contracts browsers only, since tenders have no per-tender index. Every one is validated on read: unknown values are dropped rather than passed into a `DbColumnFilter`.
- `?cpv` / `?mp` — the `/procurement/contractors` ("Топ изпълнители") filters, owned by `useUrlContractorFilters` (a small local hook, NOT the shared one above, since the leaderboard has no `?proc/?year/?grade` analogue). Here `?cpv` is **single-valued and normalised to a 2-digit CPV division ON WRITE** — `contractor_rank` (122) is a `(scope_key × division)` rollup with an `'ALL'` sentinel, so picking a finer catalogue code stores `?cpv=45` and the division filter is ALWAYS sent (default `'ALL'`); a multi-value set would return N rows per contractor and double-count. `?mp=1` is the MP-tied toggle (`is_mp_tied`). `?q` seeds the search box and `?pscope` the scope, as elsewhere.
### Local-elections routes
Local cycles (`mi*`, `chmi*`) live alongside parliamentary but in their own data tree and URL space:
- `/local/:cycle` — cycle overview (council vote share, mayors-won, município list with SOF pinned)
- `/local/:cycle/:obshtinaCode` — per-município dashboard (section map + compact mayor-candidate & council-party tiles, council hemicycle, kmetstvo mayors, район mayors, top councillors, chmi history). The compact tiles' "see full results" links drill into the dedicated full-breakdown pages below.
- `/local/:cycle/:obshtinaCode/mayor` — full mayor candidate ranking (R1 + R2)
- `/local/:cycle/:obshtinaCode/council` — full council party-by-party breakdown (expandable elected-councillor lists) + hemicycle + top councillors
- `/local/chmi` — chronological feed of all extraordinary (partial + new) elections across cycles
- `/sverka` — national officials-vs-CIK reconciliation table
`<cycle>` is the raw-data folder name: `2023_10_29_mi`, `2019_10_27_mi`, `2024_06_23_chmi`, `2024_10_20_chmi_nov`, etc. The synthetic `SOF` obshtinaCode holds Sofia's city-wide bundle (Sofia districts are the 24 `S2***` shards). Partials never appear in the elections selector — they surface contextually via tile + `/local/chmi` only.
### Data Hook Pattern
```typescript
// All data hooks follow this pattern
const queryFn = async ({
queryKey,
}: QueryFunctionContext<[string, string]>): Promise<T> => {
const response = await fetch(`/${queryKey[1]}/resource.json`);
return response.json();
};
export const useResourceData = () => {
const { selected } = useElectionContext(); // selected = "YYYY_MM_DD"
return useQuery({ queryKey: ["resource", selected], queryFn });
};
```
### Data Pipeline (`/scripts`)
- `scripts/main.ts` — CLI entry point (cmd-ts)
- `scripts/parsers/` — Parse raw CSV/ZIP election data
- `scripts/parsers_local/` — Parse local-elections data (mi2023, mi2019, chmi partials). HTML-only ingest from `results.cik.bg/mi{YYYY}/tur1/rezultati/{oikCode}.html` via headed Playwright (CF Turnstile bypass — see `cik_fetch.ts`). One município bundle per shard under `data/<cycle>/municipalities/<obshtinaCode>.json` + national rollups in `index.json` + officials-vs-CIK reconciliation in `officials_diff.json` + per-município sidecars `officials_diff/<obshtinaCode>.json`. Aggregated cross-cycle chmi history at `data/local_chmi_history.json`. CLI: `npm run data -- --local-ingest <slug>` (where slug is `mi2023`, `mi2019`, or `chmi2024-2026/<YYYY-MM-DD>_chastichen` / `_nov`); `--local --local-date <folder>` re-parses already-downloaded raw data.
- `scripts/reports/` — Generate analytical reports (turnout, concentration, top gainers/losers, invalid ballots, recount metrics, machine flash memory)
- `scripts/stats/` — Aggregate statistics
- `scripts/search/` — Full-text search index generation
- `scripts/smetna_palata/` — Campaign financing parsing
### Path Aliases
`@/*` maps to `./src/*` (configured in `tsconfig.json` and `vite.config.ts`).
### Environment Variables
`.env.local` with `GEMINI_API_KEY` — injected into the frontend build via `vite.config.ts` as `process.env.API_KEY`.