update-connections · git:20260826.de6d6bb · 2026-08-26 · sha256 d7ff465a3ba8383d
update-connections git:20260826.de6d6bbA
Immutable. This exact content never changes and is served at /api/v1/blob/d7ff465a3ba8383d.
---
name: update-connections
description: Refresh the MP business-connections data — pulls property/interest declarations from register.cacbg.bg (Court of Audit) and Commerce Registry filings from data.egov.bg, then rebuilds the per-MP declaration files under data/parliament/ plus the assets rankings, car makes and provenance (which the person-identity layer resolves into the LIVE Postgres connections graph behind /connections). Also flags unrealistic-looking declared values (cars/apartments/assets) and walks the operator through adding a typo override. Use when the user asks to refresh declarations, update business connections, add a new declaration year (e.g. 2026 filings appear in spring), rebuild the Commerce Registry SQLite, or investigate a suspicious-looking declared value flagged by the typo checker. Also use after a fresh git clone if `data/parliament/declarations/` is empty. NOTE: three families this skill used to build are RETIRED — the static person↔person connections graph (connections*.json / mp-connections/), companies-index.json, and the officials↔company bridge (data/officials/derived/company_links.json). A missing companies-index.json is now the CORRECT state; see the banners below.
allowed-tools:
- Read
- Bash
- Edit
- Write
---
# MP business-connections data pipeline
> **⚠️ RETIREMENT NOTICE (connections-engine-v1 §P4.3).** The static person↔person MP-connections
> graph — `build_connections_graph.ts` (declarations phase 6) + `build_officials_connections.ts` and
> their outputs `connections*.json` / `mp-connections/` / `official-connections/` — is **RETIRED**.
> `/connections` and the `/person` "Свързани лица" tile now read the **live Postgres graph engine**
> (`graph_payloads` / `person_connections`, migrations 127-129 + 084 + loader `scripts/db/load_graph_pg.ts`),
> derived from `person_role` (co-ownership) ∪ `company_politicians` (procurement). The dashboard
> `MpConnectionsTile` was retired too. **Run `update-persons` (or its cloud chain, which re-derives
> the graph via `db:load:graph:pg`) after this skill to publish the refreshed connections.**
> Everything below that references `connections.json` / `connections-rankings.json` /
> `buildConnectionsGraph` / `mp-connections` is **historical** — those files are no longer built or served.
> **⚠️ SECOND RETIREMENT (2026-08-20 — `docs/plans/company-page-consolidation-v1.md`, Tier 5).**
> `data/parliament/companies-index.json` is **DELETED**, and with it phases 2, 2.5, 5 and 5a plus the
> `mpRoles` augment. **A missing companies-index.json is now the CORRECT state — do not run anything
> to "restore" it.** It was a 4.16 MB name-keyed company page whose registry arm attached a company
> UIC to an MP on a name-uniqueness check alone; migration 096 (`declaration_stake_company`) declines
> **1,751 of its 2,120 UICs** rather than grading them, and `/company/:eik` serves every one of those
> UICs from the registry identity instead. Deleted with it: `build_company_index.ts`,
> `augment_mp_roles.ts` (and its `MP_ROLES_SQL`), `tr/integrate.ts`, `useCompanyIndex.tsx`,
> `MpCompanyRedirect.tsx`, the `MpOwnershipStake.companySlug` field, and the `/mp/company/:slug`
> route (`firebase.json` 301s `/mp/company`, `/mp/company/**` and both `/en` mirrors to
> `/governance/companies`). **What this skill STILL does:** fetch + parse the declarations into the
> per-MP files, then rebuild the assets rankings, car makes and provenance — which the
> person-identity layer resolves. Everything below that references
> `companies-index.json` / `buildCompanyIndex` / `mpRoles` / `companySlug` / `/mp/company` is
> **historical**.
> **⚠️ THIRD RETIREMENT (2026-08-21 — `docs/plans/company-page-consolidation-v1.md`, Tier 6).**
> `data/officials/derived/company_links.json` is **DELETED**, with its builder
> `build_officials_company_links.ts` (declarations phase 4a) and the standalone runner
> `scripts/run-officials-links-only.ts`. It was 70,525 officials↔company links over 9,659
> officials and 22,960 UICs, **85.5% of them low-confidence**, graded on the discredited
> two-sided test its own header stated — "high only when the name is rare on BOTH sides:
> unique among officials AND mapped to a single TR company" — the one-company straitjacket
> migration 158's header calls wrong in both directions. The officials↔company set is now
> **`company_politicians` at `kind='official'`**, built by `db:load:tr:pg` from the gated
> person layer and read through `readOfficialLinkRows()` (`scripts/lib/mp_linkage.ts`). Its
> gate is migration 148's `tr_name_fold_people` people-per-name fold, which **REFUSES** a name
> the Commerce Registry says belongs to more than one human — and refuses an unmeasured fold
> too — rather than scoring it, so there is no high/medium/low grade left to filter on.
> ⚠️ **Nothing this skill runs rebuilds that set. The trigger is `db:load:tr:pg`, which needs a
> resolved person layer** — a different skill and a different cadence. Everything below that
> references `company_links.json`, `buildOfficialsCompanyLinks` / phase 4a,
> `run-officials-links-only.ts` or `namesakeCount` / `trNamesakeCount` is **historical**.
Builds the per-MP declaration files and their downstream rollups from two Bulgarian government
sources (the person layer turns them into the live `/connections` graph):
- **register.cacbg.bg** — annual property/interest declarations (Сметна палата). Provides MP-declared ownership stakes.
- **data.egov.bg** dataset `2df0c2af-e769-4397-be33-fcbe269806f3` — daily Commerce Registry (TR / Търговски регистър) filings. Provides company officers, owners, status, seat, and historical role changes.
Sitting MPs cannot legally hold management roles (ЗПК Art. 35), so the two sources are complementary: declarations give you the *ownership* side, TR gives you the *management* side plus all co-officer/co-owner relationships needed to build a real graph.
## When to use which command
The pipeline is a declarations fetch/parse (phase 1), a separate and much heavier TR-snapshot
path (phases 3-4), and a set of downstream builders. Pick the entry point based on what changed
upstream:
| Intent | Command | Time | Network |
|---|---|---|---|
| **Refresh declarations only** (new filing year, e.g. 2026 filings appear) | `npm run data -- --declarations` | ~3-15 min | per-MP XML, ~1 req/150 ms |
| **Refresh one filing year** (the usual incremental case) | `DECL_YEARS=2026 npm run data -- --declarations` | ~3-5 min | that folder only |
| **Refresh TR snapshot** (do this every few months — TR changes daily) | see "TR refresh playbook" below | ~30-60 min | one ~540 MB zip + replay |
| **Rebuild every artifact from disk** (after editing a build script — no upstream fetch) | `npx tsx scripts/declarations/rebuild_all_from_cache.ts` | ~2-4 min | none |
| **First-time bring-up** (fresh clone, no data) | TR bulk + reconstruct, then `--declarations` | ~1-2 h | full set |
**`npm run data -- --declarations` is the safe default.** It fetches register.cacbg.bg incrementally (XML files cached in `raw_data/declarations/{year}/`), then re-runs every downstream builder — assets rankings, car makes, provenance — on each run. None of them reads the TR SQLite any more, so a checkout without it builds the full set.
**`--prod` makes no difference to anything this skill writes, and that is deliberate.** Every
other builder in `scripts/` takes its JSON formatter from `main.ts`'s single `stringify`, which
pretty-prints unless `--prod` is passed. These artifacts are COMMITTED, so their format is part
of the file's identity in git — `scripts/declarations/formats.ts` fixes one per family
(compact for the parliament tree; **[2026-08-21] the pretty/2-space family is now EMPTY** — its
only member was the officials `company_links.json`, deleted at Tier 6) and no caller passes a
formatter in. Until 2026-08-15 the global one WAS threaded through, and because
the two families need different formats it could not be right for both at once: plain
`--declarations` pretty-printed the parliament tree (1,438 files, ~892k insertions of pure
whitespace) while `--declarations --prod` flipped `company_links.json` to compact (~928k
deletions, also pure whitespace). There was no invocation that produced a clean diff. Both forms
now produce byte-identical output; a refresh that changes nothing substantive touches only the
`generatedAt` stamps. `scripts/declarations/formats.test.ts` is the gate — do NOT re-add a
`stringify` parameter to a builder, and do NOT mass-reformat either family to unify them (each
would be a ~million-line no-op diff and a full re-upload of the trees that ship).
To rebuild the derived files without re-fetching declarations, use the runner rather than a
hand-rolled one-liner — it sequences the builders and writes through the fixed formatters:
```bash
npx tsx scripts/declarations/rebuild_all_from_cache.ts --skip-reparse
```
`--skip-reparse` drops the whole-corpus XML re-parse — use it when the edit is to a downstream
builder (`build_assets_rankings.ts`, `build_car_makes.ts`, `build_data_provenance.ts`) rather
than to the declaration parser. Drop the flag after editing `parse_declaration.ts`. Either way
the per-MP `declarations/{mpId}.json` files (the slow-to-fetch part) are never re-fetched.
**[2026-08-20] The runner's old ordering constraint is gone with the phases it protected.** It
used to insist on running `buildCompanyIndex` FIRST because the `mpRoles` augment *appended* to
`companies-index.json` and would duplicate roles against an already-augmented file. Both are
deleted, and the re-parse no longer carries `companySlug` across from the previous vintage — the
parser's stake array now stands as written. The runner prints phase numbers 1, 6, 7, 8 — 4a went
with the officials bridge on 2026-08-21 — and the gaps are real, labelled in its source, and do
NOT line up with `index.ts`'s.
## Inputs
- `public/parliament/index.json` — produced by the **parliament-scrape** skill. Required input. If missing, run that skill first; the declarations script will warn and exit otherwise.
- `register.cacbg.bg/{year}/list.xml` — directory of all declarants for that filing year. Walked under the "Народни представители" category only.
- `register.cacbg.bg/{year}/{xmlFile}` — per-MP declaration XML. Cached under `raw_data/declarations/{year}/`.
- `raw_data/tr/state.sqlite` (~120 MB) — reconstructed Commerce Registry. **Optional**: phases 5 + 6 degrade gracefully without it. See "TR refresh playbook".
- `data/postcode_ekatte.json` — BG Post postcode → EKATTE settlement map. Joined against the free-text `registeredOffice` field to assign each company a `ekatteHQ[]`, so the "Companies HQ'd here" tile (`/sofia` + every settlement page) can list MP-linked firms by HQ address. Regenerate from upstream with `npx tsx scripts/parliament/build_postcode_ekatte.ts` — triggered automatically by the watcher when `bgpost_postcodes` flips. Missing/empty file degrades to name-only matching: ~16 village ambiguities (Лозен, Лясково, …) get picked arbitrarily.
## Outputs
All under `public/parliament/`:
| Path | Size (raw / gzip / brotli) | Lifecycle |
|---|---|---|
| `declarations/{mpId}.json` × ~600 | ~3.6 MB total | One file per MP. Carries the **full** stake schema for `MpFinancialDeclarations`. |
| `companies-index.json` *(NO LONGER WRITTEN)* | — | Retired 2026-08-20 by company-page-consolidation-v1 Tier 5. A 4.16 MB company page keyed on a slug of the DECLARED NAME, whose registry arm attached a UIC on a name-uniqueness check alone; `/company/:eik` serves all 2,120 of those UICs from the registry identity, and the declared stakes come from `declaration_stake_company` (096), which REFUSES 1,751 of them. Its three builders (`build_company_index.ts`, `augment_mp_roles.ts`, `tr/integrate.ts`) are deleted |
| `mp-management/{mpId}.json` *(NO LONGER WRITTEN)* | Retired by mp-tr-edges-pg-v1 — `/api/db/mp-management` (migration 150) serves an MP's registry roles from the gated person layer instead, refusing names the Commerce Registry records for more than one person. `integrate.ts` phase 2, which wrote these, is deleted |
| `connections.json` | 2.5 MB / 218 KB / 136 KB | Cross-MP/company/person graph for `/connections`. Lazily fetched on that route only. |
| `mp-connections/{mpId}.json` × ~600 | ~4.2 MB total (median ~1.8 KB / max ~190 KB raw) | Per-MP 1-hop + co-officer-2-hop subgraph. Loaded on each candidate page (`MpConnectionsMini`). MPs with no neighbourhood get no file (fetch 404 → component renders nothing). |
| `connections-rankings.json` | 791 KB / 74 KB / 55 KB | Top-MPs / top-companies for the dashboard tile + `/connections` rankings card. **Loaded on every dashboard view** — keep it lean. |
| `companies-by-{ekatte,obshtina}/…` *(NO LONGER WRITTEN)* | Retired by mp-tr-edges-pg-v1 — `/api/db/place-mp-companies` (migration 151) serves these from the gated person layer, covering 1,332 settlements against the shards' 176. Both builders are deleted |
| `company-connections/{eik}.json` *(NO LONGER WRITTEN)* | Retired 2026-08-16 — `/api/db/company-connections` (migration 158 `company_political_links`) serves this from the gated `person_role` tr/ngo set. The builder is deleted. Its reader was never `/company/:eik` as this row long claimed, but the AI chat's `companyConnections` tool. NOT a port: the shards name-matched TR officers against a power roster and graded the result `medium`/`low`; 158 lets the registry's own people-count per name fold (`tr_name_fold_people`, 148) decide and refuses an unmeasured fold. Corpus-wide that is wider — 9,982 companies with a direct link vs 3,843, 26,047 answerable vs 19,232. ⚠️ The 19,232 gitignored local files are NOT deleted and nothing rewrites them; the `bucket_sync_paths.ts` exclusions must stay so they cannot re-upload |
The per-MP declaration files are append-only (one file per MP id; rewriting one file does not affect others). The aggregates the downstream builders still write — `assets-rankings.json`, `mp-assets/*`, `car-makes.json`, `mp-cars.json`, `data-provenance.json` — are **regenerated end-to-end on every run**.
`raw_data/tr/` is gitignored (~12 GB extracted). `raw_data/declarations/{year}/` is **not** gitignored but intentionally not committed — the per-MP XML cache exists on whoever ran the fetcher; CI / fresh clones just re-fetch them.
## Pipeline phases (what each step actually does)
```
register.cacbg.bg data.egov.bg dataset 2df0c2af-…
│ │
▼ ▼
[Phase 1] declarations/ [Phase 3] all-resources.json.zip
parseFinancialDeclarations fetchBulkZip / fetchDaily
│ │
▼ ▼
data/parliament/declarations/ [Phase 4] state.sqlite
{mpId}.json reconstructState
│ │
├─ buildAssetsRankings → assets-rankings.json, mp-assets/*
├─ buildCarMakes → car-makes.json, mp-cars.json
└─ buildDataProvenance → data-provenance.json
(state.sqlite feeds nothing on this side any more — phase 4a,
buildOfficialsCompanyLinks → company_links.json, was REMOVED
2026-08-21; see the third retirement banner. The officials↔company
set is company_politicians kind='official', built by db:load:tr:pg.
buildCompanyIndex, integrateTr and the mpRoles augment —
companies-index.json + mp-management/ — went 2026-08-20;
buildConnectionsGraph earlier, with connections-engine-v1;
and buildCompanyConnections → company-connections/{eik}.json
went 2026-08-16 — served from Postgres, migration 158.)
```
Phase 1 and the downstream builders chain inside `parseFinancialDeclarations` (`scripts/declarations/index.ts`). Phases 3, 4 are kept out of `npm run prod` because they take 30-60 min and produce a 12 GB intermediate — and since 2026-08-21 no builder in the declarations chain reads `state.sqlite` at all. It is still built here, for `db:load:tr:pg` and the other pipelines that read it.
⚠️ **Do not trust a phase NUMBER on the last three builders.** The retirements left gaps, and the two runners number what is left differently — `index.ts` calls the assets rollup "Phase 7" while `rebuild_all_from_cache.ts` calls it "phase 6". The builder names are the stable reference; the numbers are scar tissue.
## TR refresh playbook
The Commerce Registry changes every business day. Refresh schedule:
- **First time** or **after >6 months**: `--bulk` (full snapshot).
- **Catching up <6 months**: `--index --incremental` (daily filings only).
Both finish with `--reconstruct` to rebuild the SQLite.
> **Per-resource download outage (June 2026).** data.egov.bg's per-resource
> endpoint (`/resource/download/{uuid}/json`, the `--incremental` path) broke
> server-side: it 302-redirects to the portal HTML shell with a "Грешка при
> вземане на метаданни за ресурс" flash for **every** file resource. This is a
> backend metadata-fetch failure, not a CSRF/session issue a client can satisfy
> (a bad token still gives 419; a correct token still gives the redirect).
> `--incremental` now detects this, refuses to write the HTML shell as a filing
> (the old code wrote ~1100 stubs that made `--reconstruct` skip every day), and
> **auto-falls-back to the bulk zip** — so `cli.ts --incremental --reconstruct`
> recovers in one go. The dataset-level bulk-zip endpoint (the `--bulk` path) is
> separate and still works. Until egov restores per-resource downloads, prefer
> `--bulk` + `--reconstruct` directly.
```bash
# ~540 MB zip download to raw_data/tr/all-resources.json.zip — resumable (HTTP Range)
npx tsx scripts/declarations/tr/cli.ts --bulk
# OR for incremental:
# walks the data.egov.bg dataset listing → dataset-index.json
npx tsx scripts/declarations/tr/cli.ts --index
# fetches only daily filings not yet on disk
npx tsx scripts/declarations/tr/cli.ts --incremental
# Replay every daily filing through the TR parser → raw_data/tr/state.sqlite (~120 MB)
# Auto-detects zip mode vs raw_data/tr/daily/*.json
npx tsx scripts/declarations/tr/cli.ts --reconstruct
# Then rebuild every declarations aggregate from disk (NO upstream fetch):
# re-parse cached XML → rankings/car-makes/provenance.
# Use this whenever a builder's logic changed but cacbg/data.egov is
# unreachable.
npx tsx scripts/declarations/rebuild_all_from_cache.ts
```
> Formats are not the runner's problem — each builder owns its own
> (`scripts/declarations/formats.ts`), so no caller can churn a file by
> passing the wrong `stringify`. Note the data root is `./data`, not
> `./public`.
>
> (This replaced `scripts/run-connections-rebuild.ts`, which was deleted
> without updating this doc. The per-EIK `company-connections/` and the
> retired `companies-by-ekatte`/`-obshtina` shard builders are NOT in this
> runner, and both are now gone outright — `/company` political links are
> served by migration 158 and `/settlement/:id/companies` by migration 151.
> Neither is `buildCompanyIndex`, deleted 2026-08-20 with the file it wrote.)
Use `--limit N` on `--reconstruct` for a smoke test (replays N days only).
## Step-by-step: adding a new declaration year
When a new filing season opens (typically May for prior fiscal year):
1. **Verify the year is published** — register.cacbg.bg only exposes the listing once submissions open:
```bash
curl -sk -A "Mozilla/5.0" "https://register.cacbg.bg/2026/list.xml" | head -c 400
```
You should see `<?xml version="1.0"...><Categories>...`. A 404 means the year isn't open yet.
2. **Run the fetch** with the new year:
```bash
DECL_YEARS=2026 npm run data -- --declarations
```
Defaults to 2025 only. Set `DECL_YEARS=2024,2025,2026` to (re-)fetch multiple years; existing per-MP files are overwritten with the union.
3. **Watch the warnings** — every `[declarations] no MP match for "..."` is a declarant we couldn't link to an MP id. Common causes are listed under "Common pitfalls". A handful per year is normal; dozens means the parliament index is stale (re-run **parliament-scrape** first).
4. **Run the typo flagger** (mandatory). After the rebuild, scan the new
declarations for unrealistic-looking BGN values that survived parsing:
```bash
npx tsx scripts/declarations/check_suspicious_values.ts
```
Flagged rows fall into three categories — see "Typo and unrealistic-value
detection" below for the decision flow. Already-overridden typos do not
re-flag (the parser corrects them before the value lands in the JSON);
anything new that prints a "FLAG" line needs operator action.
5. **Spot-check the dashboard** at `/?_=` (cache-bust) — the "Бизнес връзки на депутатите" tile should show the new year reflected in any MP whose declarations grew. The tile filters by current parliament's NS folder; switch elections to verify older NSes still populate.
6. **Regenerate the declarations-hub blob** (mandatory whenever the declarations moved):
```bash
npm run db:gen-declarations-hub-stats
```
`data/governance/declarations_hub_stats.json` backs the whole `/governance/declarations`
tile grid. Nothing else regenerates it outside a full `db:refresh`, so a rebuild that stops
at step 5 leaves the tiles quoting the previous vintage. Caught that way 2026-08-11, when
the 2015-2020 MP backfill moved the companies figure 2,761 → 3,269 and the tile stayed at
2,761 — the mechanism is unchanged even though the relation it read is not.
⚠️ **[2026-08-20] EVERY figure in it now comes from Postgres, the companies tile
included — which changes when you run this.** It used to read `companies` / `companyMps`
straight out of `companies-index.json`, because that WAS what `/mp/companies` rendered and
the rule is that a tile quotes its DESTINATION's own relation. The destination is now
`/governance/companies` over `official_companies` (178), so the generator reads that
instead. So run this **after** `/update-persons`, never before: with the person layer
unreloaded the generator publishes the previous vintage of every figure rather than of
four. An absent `official_companies` (178 unapplied) makes the tile ship with NO figure
rather than a „0 организации" claim.
The gate is `scripts/db/tests/declarations_hub_stats.data.test.ts`. It **needs Postgres and
skips when the database is unreachable**, so a green run on a checkout with no database
proves nothing here. Every assertion in it is written against something the generator does
NOT use — the destination screen's own filter, the partition structure — because its first
version re-ran the generator's own SQL and pinned two bugs.
⚠️ **REGENERATING IT IS HALF THE JOB — IT IS A BUCKET-SERVED BLOB AND NOTHING HERE
UPLOADED IT.** `useDeclarationsHubStats` fetches `/governance/declarations_hub_stats.json`
from GCS, and until 2026-08-21 this skill had no sync step at all: it regenerated the file
in step 6, committed it in step 7, and stopped. So `/governance/declarations` was stale on
prod **by construction, on every run**. Measured that day: the served copy was from
2026-08-16 while local was 2026-08-20 — across the schema change in the ⚠️ note above, so
the deployed bundle was reading `organisations` / `organisationPeople` out of a blob that
still carried `companies` / `companyMps`. Publish it:
```bash
npm run bucket:sync:paths -- governance/declarations_hub_stats.json
```
Verify with `npm run db:check-generated`, which byte-compares all six
`REFRESH_GENERATORS` artifacts (`scripts/db/refresh_coverage.ts`) against the live bucket
and exits 1 on any that differ. The registry is where a NEW generated artifact declares
its publish path; do not re-add that knowledge as prose here.
7. **Commit**:
```bash
git add data/parliament/declarations \
data/parliament/assets-rankings.json data/parliament/mp-assets \
data/parliament/car-makes.json data/parliament/mp-cars.json \
data/parliament/data-provenance.json \
data/governance/declarations_hub_stats.json
git commit -m "Refresh declarations for 2026 filing year"
```
(The tree is `data/parliament/`, not `public/parliament/` — that path holds only `votes`,
so the older form of this block failed on its first argument.)
Three paths are deliberately NOT in that list, and every one of them would break the command
rather than be a no-op. `data/parliament/mp-management` is gitignored since
mp-tr-edges-pg-v1, so `git add` exits 1 on it ("paths are ignored") — and the failure mode
matters: the other pathspecs still stage, so the commit succeeds and only the non-zero exit
says anything went wrong. `/api/db/mp-management` serves that data now (migration 150).
`data/parliament/companies-index.json` is DELETED as of 2026-08-20, so naming it fails with
"did not match any files"; nothing writes it any more. And **[2026-08-21]
`data/officials/derived/company_links.json` joined that second class** — same failure, same
reason (Tier 6); the officials↔company set lives in `company_politicians` and is committed to
no tree at all.
## Data-integrity contract
This pipeline has two upstream stages — register.cacbg.bg (declarations XML) and data.egov.bg (Commerce Registry bulk JSON) — each with its own fail-loud surfaces. The shared rule: **never overwrite a committed aggregate (`assets-rankings.json`, `car-makes.json`, `data-provenance.json`) with a partial result when an upstream stage failed mid-run**.
Fail-loud surfaces (a run throws and the affected output is not written):
| Stage | Surface | Trigger |
|---|---|---|
| Declarations fetch | HTTP non-2xx on register.cacbg.bg | `GET <url> → <status>` |
| TR dataset-index fetch | HTTP non-2xx | `GET <url> → <status>` |
| TR bulk-zip prepare | HTTP non-2xx OR returned an HTML error stub | `prepare GET ... → <status>` / `prepare returned non-JSON body` |
| TR bulk-zip download | HTTP non-2xx OR empty body | `download GET → <status>` / `download returned empty body` |
| TR state reconstruction | Required SQLite schema missing | Thrown by `reconstruct_state.ts` |
Intentional non-fatal skips (logged with `[stage]` prefix, ingest continues):
| Surface | Behaviour | Why |
|---|---|---|
| `raw_data/declarations/<dir>` missing | Builder for that step warns `not found — skipping` | Allows partial pipeline runs (e.g. assets rebuild without re-fetching declarations) |
| Per-MP declaration parse returns null on a field | The field is omitted from that MP's record | Cell-level parser resilience — one bad row shouldn't reject a whole filing |
| Unmatched MP name (married, hyphen variant) | Counted in the "unmatched" tally at the end | Documented below — irreducible 1-2 per parliament |
| Slug collision across companies | First-wins with a warning | Documented below |
The per-stage summaries printed at the end of each `npm run data -- --connections` / `--declarations` / `--companies` run tell you the actual counts — if any of them suddenly drop by >10% vs. the previous run's commit, treat as a regression and investigate before committing.
## Common pitfalls
### register.cacbg.bg cert is not in Node's CA bundle
The Bulgarian government's root CA isn't trusted by default. The fetcher applies a one-off `Agent({ connect: { rejectUnauthorized: false } })` **only to register.cacbg.bg URLs** (`scripts/declarations/index.ts:34`). Don't disable globally and don't try to pin a cert — the chain rotates.
### Hyphenated surnames have spacing variants
register.cacbg.bg writes `"Бъчварова - Пиралкова"` (spaces around the hyphen); parliament.bg writes `"БЪЧВАРОВА-ПИРАЛКОВА"`. The declaration normaliser collapses `\s*-\s*` → `-` before lookup. If you see `no MP match` warnings on a hyphenated name, check the normalizer is still doing this.
### Married names cannot be matched
Same constraint as the parliament-scrape skill — `НЕБИЕ ИСМЕТ КАБАК` (CIK + register listing) and `НЕБИЕ ИСМЕТ ЦЪРЕНСКА` (parliament.bg, after marriage) won't link. Logged as `no MP match`. There is no fix in the data layer; manual override would need a name-alias table that doesn't exist yet.
### Empty `oldnsList` for some former MPs
parliament.bg's profile API returns `oldnsList: []` for some ex-MPs (e.g. Ивелин Михайлов, leader of Величие, served in NS 51). The connections rankings file backfills `nsFolders` for these MPs by parsing their declaration `institution` strings (`"51-во Народно събрание"` → `"51"`) — see `nsFoldersForMp` in `build_connections_graph.ts`. The frontend dashboard tile then filters on `row.nsFolders.includes(folder)` from the rankings JSON, **not** from `useMps()`. If you find an ex-MP missing from a per-election dashboard despite having declarations, check that their declarations file references the right NS in `institution`.
### The parliament index `nsFolders` field is NOT auto-updated by this pipeline
The backfill above lives in the rankings file only. `index.json` still reflects whatever parliament.bg's `oldnsList` returned. Don't add a "fix the index" step here — it would couple the connections pipeline to the parliament-scrape outputs and create a circular dependency. Leave the index as the canonical parliament view; treat the rankings nsFolders as the connections-aware view.
### Company names are grouped by a NAME FOLD, not by a slug
**[2026-08-20]** Slug collision used to be the hazard here: two companies differing only in
casing or quote style slugged to one string, `build_company_index.ts` disambiguated with
`-2` / `-3`, and `MpCompanyScreen` looked the slug up by exact match. All three are deleted
with `companies-index.json`, along with `MpOwnershipStake.companySlug`. Client-side stake
grouping now folds the raw declared name through `src/lib/companyNameFold.ts`
(`src/data/parliament/consolidateStakes.ts`), so there is no slug to collide and no index to
check against — but the underlying fact has not moved: **a declared company NAME is not an
identity.** `/company/:eik` is keyed on the registry EIK, and 096 refuses a UIC it cannot
confirm rather than guessing one.
### `"-"` placeholder values in declarations
register.cacbg.bg uses `"-"` as a "no value" sentinel in `itemType`, `companyName`, `holderName`, etc. Don't treat them as real strings (don't display one as a company name, don't let one become a group key). A name fold of `"-"` is empty, and an empty fold must not become a company — the deleted `build_company_index.ts` dropped exactly that case so the graph would not grow a placeholder company node, and any future consumer of the declared name owes the same guard.
### Decimal/thousand separator typos in declared values
A non-trivial fraction of declared BGN values are off by 100×–1000× because the declarant typed thousand-separators where the form expected decimals (or vice versa). Without intervention these dominate the assets ranking and the per-MP wealth pages. The pipeline handles them via narrow per-row overrides plus an automated flagger — see "Typo and unrealistic-value detection" below for the override tables, the heuristic thresholds, and the decision flow.
### The declarations chain no longer reads the TR SQLite at all
**[2026-08-21]** `raw_data/tr/state.sqlite` used to be an optional enrichment here: a missing one (fresh clone, before TR bulk runs) made `build_officials_company_links.ts` emit `company_links.json` from the declared-stake arm alone, and `pep_connected` (in `/update-procurement`) plus the funds political-economy join then got a thinner set. That builder is deleted (Tier 6) — and with `integrateTr` / `companies-index.json`'s `tr` field and the `mp-management/` shards already gone (2026-08-20, `/api/db/mp-management` serves those from migration 150), **no builder in this chain opens the SQLite**. A checkout with no TR snapshot now writes the complete set of committed aggregates.
Both downstream consumers read `company_politicians` at `kind='official'` instead, so what makes THEM thin is an unresolved or unloaded person layer, not a missing file: `db:resolve:persons` then `db:load:tr:pg`. Run the TR refresh playbook anyway — `db:load:tr:pg` reads the SQLite, so that set is only as current as the snapshot behind it.
### TR confidence model
TR-only matches are name-based. The integrator emits three tiers, of which only two ship:
- **high** = full normalized name match AND (TR seat city contains the MP's region OR another MP from the same party already declared a stake in this UIC)
- **medium** = full normalized name match only
- **low / surname-only** = suppressed entirely (Bulgarian common names like Иван Иванов explode into hundreds of false positives)
The `/connections` page has a "high confidence only" filter; the dashboard tile already counts `highConfDegree` only. When investigating an MP's ties seriously, prefer the high-only view.
**TR-namesake guard (name-collision fix).** Every officer→power-person name match needs a gate: a name spread across multiple TR companies is almost always several distinct people (common Bulgarian names recur thousands of times), so attributing all those companies to one MP/official is a false positive. **[2026-08-21] The old company-count proxy now survives NOWHERE — every successor counts PEOPLE instead:**
- ~~`build_officials_company_links.ts`~~ — **deleted 2026-08-21**, the last holder of the proxy (a TR link was `high` only when unique among officials AND `trNamesakeCount === 1`). Its successor is `company_politicians` at `kind='official'`, built by `db:load:tr:pg` from the gated person layer on the same `tr_name_fold_people` fold (148) as the two below — which REFUSES a name the registry says covers more than one human, and refuses an unmeasured fold, rather than grading either.
- ~~`tr/build_company_connections.ts`~~ — **deleted 2026-08-16.** Its successor, `company_political_links` (migration 158), reads `tr_name_fold_people` (148) — the registry's own count of how many PEOPLE a name fold covers — and refuses any fold it has not measured.
- ~~`build_connections_graph.ts` phase-3~~ — deleted with the static graph (see the first banner).
- ~~`buildTrNamesakeCounts` / `buildNamesakeFilteredLinkageMap` in the two cross-reference builders~~ — **deleted 2026-08-20.** `scripts/procurement/cross_reference.ts` and `scripts/funds/cross_reference.ts` now read the gated MP↔company link set through `scripts/lib/mp_linkage.ts`, which is gated on the same `tr_name_fold_people` fold.
A company count was only ever a proxy, and it erred in both directions: it dropped a rare-name official's whole set behind one busy registered agent, and passed a name held by two people with several companies each. Either way, the point is what it keeps out — a Горна Малина councillor off Софарма Трейдинг's billions, a Чирпан deputy-mayor off „Автомагистрали".
### One stake schema — `MpOwnershipStake`
**[2026-08-20]** There used to be two. The full `MpOwnershipStake` lives in
`data/parliament/declarations/{mpId}.json` — `MpFinancialDeclarations.tsx` renders all of
`itemType`, `companyName`, `registeredOffice`, `holderName`, `transfereeName` — and
`companies-index.json` shipped a slim `CompanyIndexStake` projection beside it. The index, the
projection and the `companySlug` field the two joined on are all deleted; the 556 committed
per-MP shards were rewritten without it. The rule that produced the split still binds any
future aggregate: **weigh a field's cost against how often the file is loaded**, and do not add
one back without checking it is actually rendered.
⚠️ **The Postgres column `declaration_stake.company_slug` is deliberately KEPT and no longer
written** — see its comment in `scripts/db/schema/pg/089_declarations.sql`. It is dropped from
the declarations loader's column list, so a fresh load leaves it NULL. Do not read it, and do
not "finish the job" by dropping it.
### Per-MP files vs aggregates
Per-MP files (`declarations/{mpId}.json`, `mp-assets/{…}.json`) are **lazy** — the candidate page fetches one. Aggregates (`assets-rankings.json`, `car-makes.json`) are **eager** in the routes that use them. When trimming output, weigh field cost against load frequency.
## Typo and unrealistic-value detection
Declarants occasionally enter the wrong number of zeros — a 33,000 BGN
apartment becomes 33,000,000, a 1999 VW Golf becomes 800,000 BGN. Left
alone these typos dominate every chart on the site (highest declared car,
top assets ranking, single-mp net worth) and silently corrupt the totals.
The pipeline handles them in two layers:
### Layer 1 — narrow overrides applied at parse time
Two tables in `scripts/declarations/parse_declaration.ts`:
- `REAL_ESTATE_VALUE_OVERRIDES` — Table 1 rows. Match key:
`sourceUrlContains` + `location` + `areaSqm` + raw value.
- `VEHICLE_VALUE_OVERRIDES` — Table 3 rows. Match key:
`sourceUrlContains` + `acquiredYear` + raw value, plus optional
`detailContains` (case-insensitive substring) to disambiguate when an MP
has multiple cars in the same filing.
Each entry corrects exactly one declared value; the parser swaps in the
`correctedValue` and the rest of the pipeline never sees the original. Both
match keys are intentionally narrow — heuristic clamps ("anything over 100k
BGN/m² must be wrong") would silently rewrite legitimate luxury holdings.
**The per-m² anchor is the BUILDING (column 6), not the plot (column 5).**
`perSqmAnchor` in `parse_declaration.ts` is the one definition, shared with Layer 2.
The filing instructions are explicit — „в колона 5 се посочва площта на парцела, а в
колона 6 - на сградата" — and anchoring on the plot was wrong in both directions: a
36m² вила on a 980m² Sofia plot reads 423,558/m² of building but only 15,559/m² of
plot, so it published at €15.2m and ranked #1 on `/officials/assets`; and an apartment
declares its plot as „0", so 662 valued building rows had no usable anchor at all.
It picks the first USABLE area, not the first present one — a column-6 cell sometimes
holds an ideal part („1/2") rather than an area (75 are fraction-shaped corpus-wide, 2 in
a table-1 built-area position), and committing to those would suppress the plot fallback.
`builtAreaFromCell` refuses the fraction shape at source.
⚠️ **A `/100` auto-correction is a rewrite of a published number, and Layer 2 cannot
see it** — the flagger reads the parsed shards, where the raw value is gone. Worse, the
two layers interact: a `/1000` typo that used to fail the per-m² test and get caught by
the `> 5M` absolute arm now fires, lands under 5M, and goes unflagged while still 10×
too high. Every entry point that runs the parser therefore prints an
`AUTO-CORRECTED (verify)` section (`reportAutoCorrections`). **Read it.** Where `/100`
is the wrong divisor, add a `REAL_ESTATE_VALUE_OVERRIDES` entry — Касчиев's villa is the
worked case, where the honest value is `/1000`.
**Persistent vs per-filing source URLs**: register.cacbg.bg URLs share a
UUID prefix that's the declarant's persistent identifier, with a 6-digit
trailing suffix per filing year. If the same MP files the same erroneous
row across multiple years, match by the **persistent prefix only** so one
override covers every year. Concrete example —
`D6FB7B43-A7B9-496A-BEA5-05040F3EB514` (Hakkı's prefix) covers his 2022,
2023, and 2024 filings of the same VW Golf row.
### Ideal-part weighting — why MP net worth dropped ~13%
Since 2026-08-15 every wealth total counts a co-owned holding at the declarant's
**идеална част**. The register records the WHOLE property's acquisition price on each
co-owner's row („БЕЗ ДА СЕ ДЕЛИ МЕЖДУ СЪСОБСТВЕНИЦИТЕ", table 1 col 11) and requires each
co-owner on a separate row (col 8), so summing raw counted a jointly-held home twice.
**A one-time drop against a pre-2026-08-15 baseline is the FIX, not a regression** —
measured: MPs −13.3%, executive −16.1%, municipal −19.9%. `/mp-cars` moved too
(€8,241,472 → €7,109,059): vehicles are weighted on the same rule.
The rule lives twice — `assetShareMultiplier()` in `src/lib/declarations.ts` and
`asset_share_multiplier()` in `090_person_wealth.sql` — because a route cannot import TS.
`scripts/db/tests/asset_share_multiplier.data.test.ts` runs both over every
`(share, category)` literal in the corpus. Two things it deliberately does NOT do:
`security` is never weighted (that cell is a COUNT of дялове, not a fraction), and
anything not an unambiguous proper fraction returns 1 — „СИО", „по 1/2" and „1/2-1/2"
each already state the household's whole holding on one row.
### Layer 2 — automated flagging of unhandled rows
```bash
npx tsx scripts/declarations/check_suspicious_values.ts
```
Walks every row in `public/parliament/declarations/{mpId}.json` and prints
"FLAG" lines for any whose declared BGN value passes a category-specific
heuristic threshold. Already-overridden rows don't re-flag because Layer 1
corrects them before they land in the per-MP JSON. Run after every
`npm run data -- --declarations` (also wired into the new-year refresh
playbook above).
Current thresholds (`THRESHOLDS` constant in the script — keep narrow):
| Category | Threshold |
|---|---|
| Real estate | > 5M BGN absolute, OR > 100k BGN/m² on the `perSqmAnchor` area (building first, plot as fallback — shared with Layer 1) |
| Vehicle | > 500k BGN absolute, OR > 150k BGN for cars > 15 years old |
| Bank / cash | > 50M BGN per row |
| Receivable | > 100M BGN per row (Peevski's 19M legitimate row sits well below) |
| Investment / security | > 50M BGN per row |
For each flagged row the operator decides:
1. **Real typo** — add an entry to the matching override table in
`parse_declaration.ts` keyed by sourceUrl (use the persistent prefix
when the same row appears across years), then re-run
`scripts/declarations/rebuild_all_from_cache.ts`. The flagger should
stop reporting the row on the next run.
2. **Legitimate large holding** — leave it alone. It will keep flagging
on every check; that's intentional. Don't widen the threshold to make
one row pass — that risks silencing the next typo at the same
magnitude.
3. **Wrong field, not wrong value** (e.g. a 3000 m² plot entered as
"3 m²" with the price intact) — neither override nor accept fixes
this. Flagging is correct; we don't currently support area overrides.
Note it on the spreadsheet of "known data-entry errors we live with"
and move on. The article §6 already calls this case out for the
Pavlov 2021 typo.
### Adding a new typo override — worked example
Operator sees this on the flagger:
```
▸ Стратсимир Илков Павлов — real_estate
33,383,100 BGN — real-estate value > 5,000,000 BGN
апартамент | гр.Варна | 71.14 m² | acquired 1999
declaration 2021: https://register.cacbg.bg/2021_nc/BA28CE20-4161-418F-A6A7-F02741296A4B125934.xml
```
The companion office (41 m², same year) on the same declaration is 27,169
BGN — the magnitude gap is the tell. Add to `REAL_ESTATE_VALUE_OVERRIDES`:
```ts
{
sourceUrlContains: "BA28CE20-4161-418F-A6A7-F02741296A4B125934",
location: "Варна",
areaSqm: 71.14,
rawValue: 33383100,
correctedValue: 33383,
note: "Corrected: declarant misplaced separator (source value 33,383,100 BGN for 71m² Varna apartment).",
}
```
Re-run `npx tsx scripts/declarations/rebuild_all_from_cache.ts`. The
flagger no longer reports the row, and the assets ranking / candidate
page now show the corrected value.
## Debug knobs
Three env vars on the declarations script for debugging without re-fetching everything:
```bash
DECL_YEARS=2025 npm run data -- --declarations # default
DECL_LIMIT=20 npm run data -- --declarations # first 20 declarations only
DECL_MP_NAME=ИВЕЛИН npm run data -- --declarations # only MPs whose normalized name contains "ИВЕЛИН"
```
`DECL_MP_NAME` does substring matching after normalization, so `ИВЕЛИН` matches `ИВЕЛИН ЛЮДМИЛОВ МИХАЙЛОВ`. Useful for debugging a single MP's data without burning 10 minutes on the full set.
For the TR side, `--limit N` on `--reconstruct` replays only the first N days (smoke test — verifies the parser without rebuilding the full state).
## What this skill does NOT do
- **Does not refresh `public/parliament/index.json`.** That's the parliament-scrape skill's job. This pipeline reads the index but never writes it. If you see lots of `no MP match` warnings, run parliament-scrape first.
- **Does not reconcile married names** to maiden names. There is no name-alias table. Affected MPs simply won't have declarations linked.
- **Does not pull historical TR snapshots.** The TR dataset is a stream of daily filings; `--reconstruct` replays them to a single "current state" SQLite. There is no "TR as of 2022-12-31" mode.
- **Does not run during `npm run prod`'s default flow.** Only `parseElections` runs by default. You must pass `--declarations` (and optionally refresh TR beforehand). This is intentional — the chain takes minutes and depends on external services.
## File map
| Path | Purpose |
|---|---|
| `scripts/declarations/index.ts` | Phase 1 entry. Walks register.cacbg.bg, writes per-MP JSON, then chains into the assets-rankings / car-makes / provenance builders. |
| `scripts/declarations/parse_declaration.ts` | XML → `MpDeclaration` (stakes + income tables). Also owns `REAL_ESTATE_VALUE_OVERRIDES` and `VEHICLE_VALUE_OVERRIDES` — narrow per-row corrections for declarant typos. |
| `scripts/declarations/check_suspicious_values.ts` | Flagger that prints any per-row BGN value above the heuristic thresholds. Run after every declarations refresh; informational exit code only. |
| `scripts/declarations/rebuild_all_from_cache.ts` | Re-parse every cached declaration XML and re-run every downstream builder, no network. Use after editing `parse_declaration.ts` (e.g. adding an override) so existing per-MP JSON files pick up the change. |
| `scripts/declarations/tr/cli.ts` | Phase 3 + 4 entry. Bulk + incremental + reconstruct subcommands. |
| `scripts/declarations/build_assets_rankings.ts` | Per-MP wealth rollups + the cross-MP rankings file. |
| `scripts/declarations/build_car_makes.ts` | `car-makes.json` + `mp-cars.json`. |
| `scripts/declarations/build_data_provenance.ts` | Per-NS declaration-year window + filing rate. |
| `src/lib/companyNameFold.ts` | The declared-company-name fold that `src/data/parliament/consolidateStakes.ts` groups on — it replaced the `companySlug` key (2026-08-20). |
| `src/data/parliament/useConnectionsGraph.tsx` | RQ hook for connections.json. |
| `src/data/parliament/useConnectionsRankings.tsx` | RQ hook for rankings (dashboard tile). |
| `src/data/parliament/useMpConnections.tsx` | RQ hook for one MP's neighbourhood. |
| `src/data/parliament/useMpManagement.tsx` | RQ hook for one MP's management roles. |
| `src/screens/ConnectionsScreen.tsx` | Full graph + rankings + path-finding. |
| `src/screens/dashboard/MpConnectionsTile.tsx` | Top-MPs / top-companies tile filtered by selected election. |
| `src/screens/components/candidates/MpConnectionsMini.tsx` | Per-MP graph on the candidate page. |
| `src/screens/components/candidates/MpFinancialDeclarations.tsx` | Per-MP declaration listing. Reads `declarations/{mpId}.json`. |
| `src/screens/components/candidates/MpManagementRoles.tsx` | Per-MP management roles. Reads `mp-management/{mpId}.json`. |
| `src/screens/components/candidates/MpAvatar.tsx` | Reusable MP avatar with party-coloured ring (used everywhere connections list MPs). |
## Frontend integration cheat-sheet
If you change a script's output schema, update these in lockstep:
- `MpOwnershipStake`, `MpDeclaration`, `MpIncomeRecord` in `src/data/dataTypes.ts` — per-MP file shape. **[2026-08-20] `MpOwnershipStake.companySlug` is gone**, from the type, from the 556 committed shards and from the declarations loader's column list; a stake is grouped client-side by folding its raw declared name through `src/lib/companyNameFold.ts`.
- `ConnectionsNode`, `ConnectionsEdge`, `ConnectionsTopMp`, `ConnectionsTopCompany`, `ConnectionsRankings`, `ConnectionsGraph` in `src/data/dataTypes.ts` — graph + rankings shape.
- `TrCompanyOfficer`, `TrCompanyEnrichment`, `MpManagementRole`, `MpManagementFile` in `src/data/dataTypes.ts` — TR-derived shapes.
The match key everywhere in this pipeline is the parliament-scrape normalizer with the extra hyphen-spacing collapse: `name.toUpperCase().replace(/\s*-\s*/g, "-").replace(/\s+/g, " ").trim()`. Don't introduce a second normalizer — it'll silently drift.