--- name: update-regional description: Refresh the sub-national (oblast / NUTS 3) indicators data (data/regional.json) — re-fetch Eurostat NUTS 3 annual series (GDP per capita, population, net migration, recorded-theft rate, active-enterprise density) and merge АЗ long-term unemployment plus four NSI JSON-stat open-data datasets (FDI per capita, museum visits, hospital beds, crude death rate). Use when the daily watch report flags new Eurostat regional releases ("Eurostat regional (BG): new release") or NSI regional open-data releases ("НСИ regional open-data (BG): new release"), when the user asks to refresh regional indicators, or when adding a new NUTS 3 / NSI oblast indicator. allowed-tools: - Read - Bash - Edit - Write --- # Update Regional skill Refreshes `data/regional.json` — the per-oblast indicator backdrop rendered on `/municipality/` drilldowns and the `/demographics` regional choropleth. Pulls Eurostat NUTS 3 annual series: GDP per capita (`nama_10r_3gdp`), population (`nama_10r_3popgdp`), net migration rate (`demo_r_gind3`), recorded-theft rate (`crim_gen_reg`), plus a derived active-enterprise density (`bd_size_r3` ÷ population); merges one АЗ oblast series — long-term unemployment share (`ltUnemployment`); and merges four NSI JSON-stat open-data datasets — cumulative FDI per capita (id=629), museum visits per 1000 (id=844), hospital beds per 1000 (id=1206), crude death rate per 1000 (id=1139 = deaths by district ÷ population; a descriptive age-dominated demographic outcome, NOT a spend-adjustable measure). Together these close the Regional-Profiles gap categories Security, Business environment, Investment, Culture and Health at oblast grain. ## When to run | Trigger | Action | | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Daily watcher reports `Eurostat regional (BG): new release · nama_10r_3gdp , ...` | Run Step 1 (both scripts) | | Daily watcher reports `AZ (Агенция по заетостта): new year(s)` | Run Step 1 — refreshes the `ltUnemployment` oblast series | | Daily watcher reports `НСИ regional open-data (BG): new release · …` | Run Step 1 — `fetch_nsi.ts` re-pulls the changed NSI dataset(s) | | User asks to "refresh regional indicators" or "update regional data" | Same | | Adding a new NUTS 3 indicator | Add an entry to `INDICATORS` in `scripts/regional/fetch_eurostat.ts` (Eurostat) or to `NSI_INDICATORS` in `scripts/regional/fetch_nsi.ts` (NSI open-data), extend the `RegionalIndicatorKey` union + `DELTA_KIND` + `formatRegionalValue` in `src/data/regional/useRegional.tsx`, then run | Eurostat publishes these series on an annual cadence (typically February-March for the prior year); АЗ publishes its annual review in Q1. Day-to-day there is normally nothing to do. ## Step 1 — Fetch ```bash npx tsx scripts/regional/fetch_eurostat.ts # Eurostat NUTS 3 series npx tsx scripts/regional/fetch_az_oblast.ts # merge АЗ ltUnemployment npx tsx scripts/regional/fetch_nsi.ts # merge NSI open-data (FDI, museums, hospital beds, death rate) ``` Run **all three, in order** — `fetch_eurostat.ts` rewrites `data/regional.json` from scratch (the Eurostat NUTS3 series + the derived `theftRate` crime and `enterpriseDensity` business indicators), then the two mergers add their indicators into the file it wrote. `fetch_az_oblast.ts` reads the cached АЗ XLSX (downloaded by `update-indicators`) for `ltUnemployment`; `fetch_nsi.ts` pulls four NSI JSON-stat open-data datasets (`fdiPerCapita` id=629, `museumVisitsPer1000` id=844, `hospitalBedsPer1000` id=1206, `deathRatePer1000` id=1139), normalising each against the population series the first script wrote. Running only the first no longer DESTROYS the other five indicators — `fetch_eurostat.ts` carries every foreign indicator key across its rewrite and names them on stdout (`carried forward from other writers: …`). It still leaves them at the previous vintage, so run all three when the mergers' own sources moved. Until 2026-08-11 the rewrite dropped all five from both the `indicators` and the `series` block with a valid file, a green run and no warning; `FOREIGN_INDICATORS` in `scripts/regional/fetch_eurostat.ts` is what prevents that now, and a new side-merged indicator MUST be added to it — `scripts/regional/regional_foreign_indicators.test.ts` fails when the committed artifact carries a key nothing declares. Expected output on a normal day: ``` NUTS3 → oblast mapping: 28 NUTS3 codes Loading gdpPerCapita (nama_10r_3gdp)... 31 oblasts, 620 points Loading population (nama_10r_3popgdp)... 31 oblasts, 620 points Loading netMigration (demo_r_gind3)... 31 oblasts, 620 points Loading theftRate (crim_gen_reg)... 31 oblasts, 248 points Deriving enterpriseDensity (bd_size_r3 V11910)... 31 oblasts Wrote /Users/.../data/regional.json carried forward from other writers: deathRatePer1000, fdiPerCapita, hospitalBedsPer1000, ltUnemployment, museumVisitsPer1000 ``` 31 oblasts = 27 administrative oblasts + 3 Sofia city МИР (S23/S24/S25, all sharing BG411) + the Plovdiv city МИР (PDV-00, sharing BG421 with the rural PDV). Most indicators auto-fail on a per-oblast point regression or a >10% drop vs. the prior committed file — investigate before re-running if you see `safety check: ...`. **Exception:** `enterpriseDensity` is a known-frozen supplementary indicator (`bd_size_r3` ended 2020) and degrades gracefully — if you see `! enterpriseDensity skipped — …`, Eurostat narrowed or retired the dataset; the rest of the refresh still completes. ## Step 2 — Verify ```bash node -e " const d = require('./data/regional.json'); for (const [k, byOblast] of Object.entries(d.series)) { const codes = Object.keys(byOblast); const sample = byOblast[codes[0]]; const last = sample[sample.length - 1]; console.log(k.padEnd(16), 'oblasts=' + codes.length, 'latest', last.year); } console.log('size:', require('fs').statSync('./data/regional.json').size, 'bytes'); " ``` Eyeball: - Each indicator should report 31 oblasts — except `ltUnemployment` (30; АЗ doesn't split one Sofia МИР) and any indicator whose source froze (`enterpriseDensity` latest 2020). - `latest` should be the prior calendar year (or two prior in Q1 if Eurostat hasn't released the latest annual figures yet). `enterpriseDensity` will show 2020 (upstream frozen). - File size ~115 KB raw / ~16 KB gzipped (10 indicators). `regional.json` is in the `bucket:gz` hot-files list, so prod serves it compressed — run `npm run bucket:sync:all` (sync **then** gzip) so it doesn't ship uncompressed. >180 KB raw would indicate a regression. - Spot-check a known-extreme oblast: Sofia city (S23) should have GDP/capita ~30,000+ EUR; Vidin (VID) ~8,000 EUR. If those flip, something is wrong with the NUTS 3 ↔ oblast mapping. ## Step 3 — Upload to bucket ```bash npm run bucket:sync ``` (Or `npm run bucket:sync:dry-run` first to preview.) This rsyncs `data/regional.json` (and any other changed data files) to `gs://data-electionsbg-com/`. ## Step 4 — Commit ```bash git add data/regional.json git commit -m "regional: refresh through " ``` If you added a new indicator, also commit the script + hook changes: ```bash git add scripts/regional/fetch_eurostat.ts src/data/regional/useRegional.tsx data/regional.json git commit -m "regional: add from Eurostat " ``` ## Adding a new indicator (advanced) Add an entry to `INDICATORS` in `scripts/regional/fetch_eurostat.ts`: ```ts { key: "newKey", // also add to RegionalIndicatorKey in src/data/regional/useRegional.tsx dataset: "dataset_code", // Eurostat code, e.g. "nama_10r_3gva" query: { unit: "EUR_HAB", freq: "A" }, // narrowing dim filters titleEn: "...", titleBg: "...", unitLabelEn: "...", unitLabelBg: "...", sourceUrl: "https://ec.europa.eu/eurostat/databrowser/view//default/table", }, ``` Then: 1. Add the key to `RegionalIndicatorKey` in `src/data/regional/useRegional.tsx`. 2. Add an entry to the `DELTA_KIND` map in the same file — `"percent"` for absolute counts (GDP, population), `"absolute"` for already-normalised rates (migration, unemployment rate). 3. Add the new code to the `DATASETS` array in `scripts/watch/sources/eurostat_regional.ts` so the watcher fingerprints it. 4. If the formatter needs a custom rule (anything that isn't "round to whole number with separators" or "signed rate with one decimal"), extend `formatRegionalValue` in `useRegional.tsx`. 5. Run this skill to regenerate `data/regional.json`. Before adding a new dataset, verify the query against the Eurostat REST API: ```bash curl -s 'https://ec.europa.eu/eurostat/api/dissemination/statistics/1.0/data/?format=JSON&lang=EN&&lastTimePeriod=4' | head -c 1500 ``` Confirm the response covers all 28 BG NUTS 3 oblasts. If it only returns aggregates (BG, BG3, BG31, ...) or returns empty `value: {}`, Eurostat does not publish that indicator at NUTS 3 granularity for Bulgaria — try a NUTS 2 dataset (one of `_2*`) or skip the indicator. ## Data-integrity contract The fetcher fails loud rather than write a partial `data/regional.json`: | Surface | Trigger | | ---------------------------------------------------------- | ----------------------------------------------------------------------------------- | | Eurostat HTTP non-2xx for any indicator | `Eurostat returned for ` | | Unexpected dimension order in the response | `Eurostat : unexpected dimension order ... — expected geo,time as last two` | | Per-oblast point floor breach | `safety check: weakest oblast has N points (floor 10)` | | 10% regression vs. prior `data/regional.json` total points | `safety check: total points dropped P → C (X% > 10%). Refusing to overwrite.` | | `process.exit(1)` from the top-level `.catch` | Any unhandled rejection | Two important quirks of this data set: 1. **The app's `data/municipalities.json` uses non-standard `nuts3` codes** (BG416/417/418 for Sofia МИР; BG421-1 for Plovdiv-minus-city). The fetcher does NOT derive the NUTS 3 → oblast mapping from that file. Instead, an explicit `EUROSTAT_NUTS3_TO_OBLAST` table at the top of `scripts/regional/fetch_eurostat.ts` maps Eurostat's real NUTS 3 codes to app oblast codes (with Sofia stolitsa BG411 fanning out to S23/S24/S25). 2. **Sofia stolitsa is one Eurostat entity, three app entities.** The same series value is duplicated to S23, S24, and S25. The drilldown tile and choropleth render a footnote noting this. ## What this skill does NOT do - **Does not refresh macro indicators.** That's `update-macro` — separate watcher source, separate skill. - **Does not scrape NSI or other Bulgarian government portals.** NUTS 3 (oblast-level) data comes from Eurostat. Sub-municipal (LAU 2 / per-обштина) data — registered unemployment per ОНС, DZI scores, EU funds absorption — would need a separate ingest path and is currently out of scope. - **Does not change the SPA UI.** Adding a new indicator key requires hand-editing the type union and (if needed) the formatter in `useRegional.tsx`. The skill only refreshes data. ## File map | Path | Purpose | | --------------------------------------------------------------- | --------------------------------------------------------------- | | `scripts/regional/fetch_eurostat.ts` | CLI entry — fetch all NUTS 3 series, write `data/regional.json` | | `scripts/watch/sources/eurostat_regional.ts` | Daily watcher — fingerprints the 3 NUTS 3 datasets | | `data/regional.json` | Generated payload (~50 KB, minified) — committed | | `src/data/regional/useRegional.tsx` | React Query hook + types + `formatRegionalValue` helper | | `src/screens/dashboard/RegionalIndicatorsTile.tsx` | Drilldown tile (geography section of `/municipality/`) | | `src/screens/components/regional/RegionalChoroplethMap.tsx` | Choropleth on `/demographics` | | `src/screens/components/regional/RegionalIndicatorSelector.tsx` | Indicator dropdown for the choropleth |