CLAUDE.md git:20260919.c129a05B

Immutable. This exact content is served forever at /api/v1/blob/c3017b94782535d5.

# claude-gateway — n8n Workflow Project

## Context

This repository manages three agentic subsystems for Example Corp Network:
- **ChatOps** — infrastructure alerts (LibreNMS, Prometheus) → `#infra-nl-prod`, `#infra-gr-prod`
- **ChatSecOps** — security alerts (CrowdSec, vulnerability scanners) → same infra rooms
- **ChatDevOps** — development tasks (CI/CD, features) → `#cubeos`, `#meshsat`

All share the same orchestration: n8n workflows bridge external triggers to Claude Code sessions.

- **n8n instance:** https://n8n.example.net
- **GitLab:** https://gitlab.example.net/n8n/claude-gateway (project ID: 30)
- **Claude Code host (NL):** `nl-claude01` — SSH as `app-user`
- **Claude Code host (GR):** `grclaude01` (10.0.X.X) — SSH as `app-user`, oversight agent for NL maintenance
- **Claude Code workspace:** `/app/cubeos`
- **Matrix server:** matrix.example.net
- **Matrix rooms:** `#chatops`, `#cubeos`, `#meshsat`, `#infra-nl-prod`, `#infra-gr-prod` (routed by project prefix; `#claude-gateway` decommissioned)
- **LibreNMS (NL):** https://nl-nms01.example.net (API key in .env, self-signed cert)
- **LibreNMS (GR):** https://gr-nms01.example.net (dedicated GR instance, self-signed cert)
- **Observability host (NL):** `nlopenobserve01` (10.0.181.X) — OpenObserve OTLP traces (:5080), **Healthchecks.io** dead-man/ping monitoring (:8000), **Langfuse v2** LLM/agent trace observability (:3000). Creds in `.env` (`HEALTHCHECKS_*`, `LANGFUSE_*`). OpenObserve image is **pinned to v0.91.1** (2026-07-03, after watchtower pulled a v0.92.0-rc1 briefly published as `latest`, migrated the DB, and upstream reverted → 846-restart crash-loop; bump the pin manually on real releases).
- **IaC repo (NL):** `/app/infrastructure/nl/production`
- **IaC repo (GR):** `/app/infrastructure/gr/production`
- **GR GitLab:** https://gr-gitlab.example.net/ (project ID: 5)
- **GR AWX:** https://gr-awx.example.net
- **Matrix bot:** `@claude:matrix.example.net` (MXID; homeserver is `matrix.example.net`)
- **Mattermost:** mattermost.example.net
- **YouTrack:** https://youtrack.example.net

---

## Available MCP Tools

These MCP servers are configured at user scope and available in every Claude Code session:

| MCP | What to use it for |
|-----|-------------------|
| `netbox` | Query devices, VMs, IPs, VLANs, cables, interfaces, changelogs from NetBox CMDB (310 devices/VMs, 421 IPs, 39 VLANs across 6 sites). **Prefer over LibreNMS for device identification.** |
| `n8n-mcp` | Build and manage n8n workflows — **use this to create, update, activate, and test workflows directly on the n8n instance**. Do NOT just generate JSON files; use the MCP to push and verify. |
| `gitlab-mcp` | Create MRs, check pipeline status, commit workflow exports — use instead of curl for all GitLab operations. |
| `youtrack` | Read issue context at session start, post completion comments at session end. |
| `proxmox` | Proxmox VE API — list nodes/VMs/LXC, get configs and status, start/stop/reboot guests. 15 tools, uses API tokens (no SSH). Lifecycle ops gated by `PVE_ALLOW_LIFECYCLE` env var. |
| `codegraph` | CodeGraphContext — code graph database (KuzuDB). Query function callers/callees, call chains, dependencies, dead code. Indexed repos: CubeOS (355K lines), MeshSat. Venv at `/home/app-user/.cgc-venv/`. |
| `opentofu` | OpenTofu Registry — provider docs, resource schemas, module metadata. Use when writing/editing `.tf` files to get correct argument names and types. |
| `tfmcp` | Terraform/OpenTofu local analysis — module dependency graph, resource dependencies, module health scoring. Use for K8s module dependency analysis. Experimental (v0.1.9). |
| `kubernetes` | kubectl/helm against the NL + GR clusters — get/describe/apply/patch, logs, exec, rollout, port-forward, helm install/upgrade, node cordon/drain. Use for live K8s state instead of SSHing control-plane nodes. |

---

## Master workflow skill (read first on every infra/security session)

Every session triggered by an alert, YouTrack state change, or Matrix command must load `/chatops-workflow` first. It contains the Phase 0→6 choreography (triage → drift-check → context → propose → approve → execute → post-incident), the Debugging Protocol (Reproduce → Localize → Fix one → Verify → Guard), the Proving-Your-Work directive (pair every CONFIDENCE ≥ 0.8 with visible evidence — enforced by `scripts/classify-session-risk.py`'s `evidence_missing` signal), the general Shortcuts-to-Resist list, and the operator-vocabulary map. Re-read at every phase boundary; context compaction may drop skill bodies. Source: `.claude/skills/chatops-workflow/SKILL.md`. Pure-dev sessions (code-explorer / CI-debugger / workflow-validator) do NOT need this skill.

---

## 03_Lab Reference Library

Path: `/app/reference-library/` (~10 GB, ~5,200 files, synced via Syncthing to nl-claude01; previously also to nl-openclaw01, that LXC was destroyed 2026-04-29).

Supplementary reference library covering everything that NetBox (CMDB) and GitLab (IaC) don't store: hardware documentation, physical wiring, change history, firmware, topology diagrams, ISP records, and per-host operational notes. You can read any file directly.

Structure: `NL/Servers|Inventory|Changes|Firmware/`, `GR/gr/`, `Cross-Site/network_info.xlsx|Designs/`, `Research/`.

Queryable via `./openclaw/skills/lab-lookup/lab-lookup.sh`: `port-map`, `nic-config`, `vlan-devices`, `switch-ports`, `docs`, `ups-pdu`. Read files directly: `Read /app/reference-library/NL/Servers/<host>/`.

### Data trust hierarchy (ALWAYS follow this order):
1. **Running config on the live device** — SSH and check (`show run`, `ip a`, `pct config`, `kubectl get`). This is the ONLY 100% truth.
2. **LibreNMS** — active monitoring, real-time status. What's happening NOW.
3. **NetBox** — CMDB inventory (devices, IPs, VLANs). Accurate but manually maintained — can drift.
4. **03_Lab, GitLab IaC, backups** — supplementary reference. Useful context but can be stale.

**If 03_Lab contradicts a live device, the live device wins. Always.** Never modify 03_Lab files.

---

## Codebase-understanding dashboard (Understand-Anything plugin)

The `understand-anything` Claude Code plugin (v2.8.2) is installed at user scope on `nl-claude01` — it turns any repo into an interactive knowledge-graph dashboard (`/understand` to build `<repo>/.understand-anything/knowledge-graph.json`, then a vite dashboard). A **complete graph of THIS repo** (1521 nodes / 8 LLM-semantic layers / 13-step tour) is served at **http://10.0.181.X:5173/?token=gateway2026dashboard** (0.0.0.0-bound, token-gated, dev server — dies on reboot; `pkill -f 'vite/bin/vite.js'` to stop). `.understand-anything/` is gitignored. Full detail + security audit + the resume-from-stall technique are in auto-memory `understand_anything_repo_audit_20260705` (+ `understand_resume_phase2_20260706`). Note: the plugin's analyzer sub-agents inherit Bash/Write and auto-run on hostile repos — safe here (own code), but don't `/understand` an untrusted repo.

---

## Syslog-ng Central Logging

Both sites have syslog-ng servers collecting logs from all hosts:

| Site | Server | Base Path |
|------|--------|-----------|
| NL | `nlsyslogng01` | `/mnt/logs/syslog-ng/` |
| GR | `grsyslogng01` | `/mnt/logs/syslog-ng/` |

Log path: `{base}/{hostname}/{year}/{month}/{hostname}-{date}.log`

**Terminal session logging:** All hosts forward terminal commands to syslog-ng, tagged `terminal-session:`. Format:
```
terminal-session: user=root tty=/dev/pts/3 pwd=/root ssh=10.0.181.X cmd=claude
```

To query terminal sessions for a host (answers "what did someone do on this host?"):
```bash
ssh -i ~/.ssh/one_key root@nlsyslogng01 \
  "cat /mnt/logs/syslog-ng/<hostname>/2026/04/<hostname>-2026-04-08.log | grep 'terminal-session:' | grep -v 'message repeated' | tail -15"
```

OpenClaw's `fetch_terminal_sessions()` in `site-config.sh` does this automatically during infra-triage (Step 2b).

---

## n8n Node Schemas

n8n-as-code is vendored as a monorepo at `node_modules/n8n-as-code-monorepo` (github:EtienneLescot/n8n-as-code — not published to npm as `n8n-as-code`, so invoke the `search`/`schema`/`templates` commands below via the monorepo's package, not bare `npx n8n-as-code`). It contains offline schemas for 537 n8n nodes, 10,209 properties, and 7,702 workflow templates.

**MANDATORY:** Before configuring any n8n node, look up its exact schema first:
- Search nodes: `npx n8n-as-code search <node-name>`
- Get full schema: `npx n8n-as-code schema <node-name>`
- Find templates: `npx n8n-as-code templates <keyword>`

Never guess n8n node parameter names, field paths, or valid values. Every gateway bug so far (sessionExpired false positive, wrong routing, missing subcommands) was caused by hallucinated node configs. The schemas are available — use them.

---

## Architecture

```
YouTrack issue state → In Progress
          ↓
    n8n webhook trigger
          ↓
    n8n SSH: Launch Claude in background with nohup
    runs: claude -p "prompt" --output-format stream-json --verbose
    returns PID + session_id immediately
          ↓
    n8n fires Progress Poller workflow (async)
    Poller polls JSONL log every 30s, posts tool activity to Matrix as m.notice
          ↓
    n8n SSH: Wait for Claude (polls PID every 5s)
    extracts final result from JSONL when PID exits
          ↓
    n8n → Matrix room message
          ↓
    User replies in Matrix
          ↓
    n8n SSH Execute Command node (resume session)
    runs: claude -r <session-id> -p "message" --output-format json
          ↓
    Loop until done
          ↓
    n8n posts completion comment to YouTrack issue
```

### Key Design Decisions

- **Background launch + progress polling** — Claude runs via `nohup ... &` with `--output-format stream-json --verbose`, writing JSONL to `/tmp/claude-run-<ISSUE>.jsonl`. A separate Poller workflow reads new lines every 30s and posts tool activity to Matrix as `m.notice`. A Wait node polls the PID every 5s and extracts the final result when Claude exits. This gives users real-time visibility into what Claude is doing during 5-15 minute runs.
- **Session continuity via `-r` flag** — first invocation creates a session and returns a `session_id`. Subsequent invocations use `claude -r <session-id> -p "message" --output-format stream-json` to resume the conversation. The `session_id` is stored in SQLite between workflow executions.
- **Matrix as the human-in-the-loop interface** — Claude Code response posted to Matrix, user replies trigger next workflow execution which resumes the session.
- **YouTrack as the trigger and sink** — webhook starts the session, completion comment ends it.

---

## LLM Usage Tracking

Per-model token/cost tracking across 3 tiers — `llm_usage` table is the single source of truth, exposed via Prometheus (`write-model-metrics.sh`, Cronicle `*/5`). 4 live writers feed it (Runner per-session, Claude CLI poller via `poll-claude-usage.sh`, LLM judge, local Ollama); the OpenClaw OAuth poller (`poll-openclaw-usage.sh`) and OpenAI poller (`poll-openai-usage.sh`) are retired (OpenClaw LXC destroyed 2026-04-29). Tier 2 cost = $0 for Max subscription (interactive CLI), API-equivalent ~$16,420 total. Three portfolio stats APIs serve live data to Hugo: `/webhook/agentic-stats` (`ncUp08mWsdrtBwMA`), `/webhook/lab-stats` (`B90NqTknqhInVLYP`), `/webhook/mesh-stats` (`PrcigdZNWvTj9YaL`). Local-first judge + synth defaults (gemma3:12b, qwen2.5:7b fallback) since 2026-04-19.

Full reference: [`docs/llm-usage-tracking.md`](docs/llm-usage-tracking.md). Calibration baseline: [`docs/judge-calibration-2026-04-19.md`](docs/judge-calibration-2026-04-19.md).

---

## Model Orchestration (centralized, 2026-06-28; MRs !116–!120)

Model selection is centralized across two planes (full provenance: [`docs/model-provenance.md`](docs/model-provenance.md)):

- **Claude-Code plane (subscription, flat-rate):** every `claude` invocation (dispatched remediation, `agent_as_tool`, `mr-review`, `parallel-dev`, interactive) is routed by ONE switch — [`scripts/claude-provider.sh`](scripts/claude-provider.sh) `{zai|anthropic|status}`, which edits the `env` block of `~/.claude/settings.json`. Live default = **Z.ai** (`glm-5.2` Opus-equiv for `--model opus`, `glm-4.7` Sonnet-equiv for sonnet/haiku). `anthropic` reverts to the Max OAuth subscription. Subscription auth can't proxy through LiteLLM, hence the direct settings.json route.
- **API plane (per-token, paid):** the eval layer (frontier cross-check, judge max-effort, RAGAS, judge-haiku) routes via the **shared LiteLLM** (`nllitellm01`, `:4000`, v1.85.0 — reused, NOT gateway-owned) to **Mistral** (`mistral-large-latest`) + **DeepSeek** (`deepseek-v4-pro`), with local-Ollama fallback (never Anthropic). Per-component spend via `x-litellm-tag`. Per operator directive, **Mistral + DeepSeek are the ONLY paid per-token APIs** (Anthropic = 0 per-token spend).
- **Local plane ($0):** judge/synth/rewrite/embed/rerank + teacher on Ollama (`gemma3:12b`, `qwen2.5:7b`, `nomic-embed-text`, `bge-reranker-v2-m3`, fast-path `llama3.2:1b`).

**Single source of truth:** [`config/model-routing.json`](config/model-routing.json) (component→provider/model) resolved by [`scripts/lib/model_routing.py`](scripts/lib/model_routing.py) (`--list`/`--resolve`/`--providers`). Provision the LiteLLM models+key idempotently: [`scripts/litellm-gateway-setup.sh`](scripts/litellm-gateway-setup.sh). Provider keys in `.env`: `ZAI_API_KEY`, `MISTRAL_API_KEY`, `DEEPSEEK_API_KEY`, `LITELLM_GATEWAY_KEY` (all gitignored). **Gotcha:** `deepseek-v4-pro` is a reasoning model → returns `[thinking,text]` blocks; parsers join `type=='text'`. To answer "which model on which component now": `python3 scripts/lib/model_routing.py --list` + `bash scripts/claude-provider.sh status`. **Note:** `model_routing.py --list` shows the *intended default* catalog (it reads `model-routing.json`, not the live toggle); for the **live** Claude-Code provider, `claude-provider.sh status` is authoritative (it reflects the `settings.json` env block). Known follow-up: make the resolver reflect the live toggle for the `dispatched-session` entry.

---

## MemPalace Integration (2026-04-09)

8 patterns ported from [mempalace](https://github.com/milla-jovovich/mempalace). New tables: `session_transcripts` (verbatim chunks + embeddings), `agent_diary` (persistent per-agent memory). Temporal KG via `incident_knowledge.valid_until`. Hooks: Stop (auto-save every 15 msgs) + PreCompact (emergency save). RAG upgraded to **5-signal RRF** (`semantic + keyword + wiki + 0.3*transcript + 0.25*chaos_baselines`). See [`docs/mempalace-details.md`](docs/mempalace-details.md) for full details.

---

## Compiled Knowledge Base (Karpathy-Style Wiki)

[Karpathy-style](https://x.com/karpathy/status/2039805659525644595) wiki at `wiki/` — 78 articles compiled from 7+ sources (memories, CLAUDE.md files, incidents, OpenClaw, docs, 03_Lab, Grafana). Compiler: `scripts/wiki-compile.py` (SHA-256 incremental, daily 04:30 UTC cron + `/wiki-compile` skill). All articles embedded into `wiki_articles` table as 3rd RRF signal. Health: `wiki-compile.py --health`. See [`docs/compiled-wiki-details.md`](docs/compiled-wiki-details.md) for source mapping and CLI usage.

---

## Operational runbooks

- **Rerank service (bge-reranker-v2-m3 at nl-gpu01:11436)** — [`docs/runbooks/rerank-service.md`](docs/runbooks/rerank-service.md). Rollback via `RERANK_BACKEND=ollama` env, container restart, model cache rebuild. Prometheus alert: `RAGRerankServiceDown`. **§ Dynamic VRAM (2026-08-26):** the shared RTX 3090 Ti budget — rerank + embed-bge-m3 `empty_cache`/idle-unload, **Ollama split: `:11434` embed-only / `ollama-gen :11441` gateway generates**, judge pre-flight/lock/timeouts, selfheal 3-strike; sources `scripts/gpu01/`.
- **Infragraph (causal infra dependency graph + model-based invariant, epic IFRNLLEI01PRD-1029 — LIVE 2026-06-09)** — [`docs/runbooks/infragraph.md`](docs/runbooks/infragraph.md). Query: `scripts/infragraph-query.py {blast-radius,deps,cascade,predict,explain,health}`. Two lanes, two fail modes: advisory triage Step 2-graph + risk-classifier signal + Build Prompt context **fail OPEN**; the remediation lane **fails CLOSED** — the Runner commits a plan_hash-keyed prediction before any approval poll (Prepare Result default-DENIES unpredicted [POLL]s; `INFRAGRAPH_DISABLED=1` = analysis-only mode), and `lib.action_verdict()`/`infragraph-verify.py` write the ONLY mechanical match/partial/deviation verdicts (deviation = never auto-resolve). Phase C proposals via `infragraph-propose-blast-radius.py` (--scan hourly / --bootstrap / **--approve per rule = the operator grant**; close the control issue = instant deactivation; first approved rule: IFRNLLEI01PRD-1046). **Core-infra declared edges built out 2026-07-09 (MR !179): `docs/host-blast-radius.md` grew 5→102 declared edges (all 6 PVE + their critical guests + network backbone), adversarially live-verified — corrected GR core = gr-sw02 not gr-sw01; 12 active fold rules (6 PVE + 3 switches + fw01/rtr01/gr-fw01), all tagged `infragraph-control` + titled `[INFRAGRAPH-ACTIVE]`, open-issue = the on-switch. `cmd_approve` now preserves `parent_host` so `--scan` won't re-propose already-folded parents (MR !181). Detail: [`memory/infragraph_coverage_gap_20260708.md`](memory/infragraph_coverage_gap_20260708.md).** 6 crons (seed 04:10, learn :25, metrics */5, eval :40, scorecard Mon 05:10, propose :45). Alerts: `Infragraph{MetricsExporterStale,SeedStale,PrecisionDrop}`; holistic §39; weekly audit invariant section. Gate IFRNLLEI01PRD-1040 = continuous revocation review via `test-results/infragraph-scorecard.json` (baseline frozen: 0.4156 per-incident auto-resolve, 30d). Full build memory: [`memory/infragraph_epic_buildout_20260609.md`](memory/infragraph_epic_buildout_20260609.md).
- **n8n Code-node safety (post 14h-outage)** — [`docs/runbooks/n8n-code-node-safety.md`](docs/runbooks/n8n-code-node-safety.md). Mandatory pre-push validator (`scripts/validate-n8n-code-nodes.sh`) for any Code-node edit. Required sequence: fetch → snapshot → edit → `--check` → splice → validate → PUT → re-fetch → re-validate → test-fire → commit.
- **Risk-based auto-approval integration** — [`docs/runbooks/risk-based-auto-approval.md`](docs/runbooks/risk-based-auto-approval.md). How the Classify Risk SSH → Build Prompt risk-section → Bridge `[AUTO-RESOLVE]` chain is wired. Weekly audit at `scripts/audit-risk-decisions.sh` enforces the no-false-positive invariant. **As of 2026-06-16 the runbook's § Autonomy-forward gate (IFRNLLEI01PRD-1102) is the live model** — 3 bands (AUTO / AUTO_NOTICE / POLL_PAUSE), enabled via `~/gateway.autonomy_forward`; `rm` to kill instantly. **Per-session approval SMS RETIRED 2026-07-09** (operator didn't react + paid Twilio) → replaced by a free daily Matrix HITL digest (`scripts/session-digest.py`, Cronicle `emrddrnzz0b` 07:30 UTC to `#infra-nl-prod`); `~/gateway.autonomy_session_sms` removed (banding untouched; platform tier-1 dead-man SMS still on). See [`memory/session_sms_retired_matrix_digest_20260709.md`](memory/session_sms_retired_matrix_digest_20260709.md).
- **Memory promotion pipeline** — [`docs/runbooks/memory-promotion-pipeline.md`](docs/runbooks/memory-promotion-pipeline.md). Operator workflow for running `scripts/memory-audit.py` and distilling clusters.
- **Skill versioning (SKILL.md semver)** — [`docs/runbooks/skill-versioning.md`](docs/runbooks/skill-versioning.md). When to bump patch/minor/MAJOR on a SKILL.md. `scripts/audit-skill-versions.sh` is advisory stale-skill detection via git history; `scripts/audit-skill-requires.sh` checks the declared `requires.bins` + `requires.env` against host state; Prometheus alerts `SkillPrereqMissing` + `SkillMetricsExporterStale` in `prometheus/alert-rules/agentic-health.yml`.
- **SeaweedFS cross-site replication recovery** — [`docs/runbooks/seaweedfs-cross-site-replication.md`](docs/runbooks/seaweedfs-cross-site-replication.md). Two stale-checkpoint failure modes (cross-site `filer.sync` + intra-cluster `meta_aggregator`) sharing one shape: persisted offset → GC'd change-log volume → permanent retry-loop. **Diagnostic gotcha:** read state per-pod, not via the cluster service — round-robin hides per-pod metadata divergence.
- **HAHA + FISHA reliability hardening (2026-04-30)** — closed IFRNLLEI01PRD-704/-801..-805/-815 in one session after the 2026-04-27→04-30 ~66h HAHA outage. Components live: `monitor_cmd` on all 5 OCF docker resources, raised start/stop timeouts, `nfs-stale-fh-exporter.py` + `exportfs-flush-webhook.py` on nlcl01file01/02, Pacemaker alert chain to ARP refresh, `alertmanager-twilio-bridge.py` Tier-1 SMS, 7 PrometheusRules, `fence_pve` TypeError patched. 14-test chaos catalog ran end-to-end; 12/14 confidence rows now >0.90 detection AND recovery. Full memory: [`memory/incident_haha_nfs_stale_fh_20260430.md`](memory/incident_haha_nfs_stale_fh_20260430.md), [`memory/haha_reliability_hardening_20260430.md`](memory/haha_reliability_hardening_20260430.md), [`memory/haha_chaos_engineering_20260430.md`](memory/haha_chaos_engineering_20260430.md).
- **Public-BGP upstream health (2026-05-17, AS64512)** — [`docs/runbooks/upstream-bgp-failure.md`](docs/runbooks/upstream-bgp-failure.md). Detection chain: `scripts/write-bgp-upstream-metrics.py` (cron `*/5` on nl-claude01) → textfile collector → Prometheus → 4 alerts (`AS64512UpstreamMissing` critical / `AS64512UpstreamCountLow` critical / `AS64512VisibilityLow` warning / `AS64512BGPMetricsExporterStale` warning). PrometheusRule deployed via IaC at `infrastructure/nl/production/k8s/namespaces/monitoring/bgp-upstream-alerts.tf`. The script reuses `get_ripe_bgp()` from `scripts/vpn-mesh-stats.py` so the alert definition stays in lockstep with the kyriakos.papadopoulos.tech/status/ diagram. Decision tree in runbook covers single-upstream loss, both-upstream loss, and RIS visibility blips (which usually self-resolve in <15 min).
- **Control-plane dead-man's-switch (IFRNLLEI01PRD-1152, LIVE 2026-06-21)** — [`docs/runbooks/gateway-watchdog-deadman.md`](docs/runbooks/gateway-watchdog-deadman.md). `gateway-watchdog.sh` (cron `*/5`) emits `gateway_watchdog_heartbeat_timestamp_seconds` + `gateway_workflow_active`/`gateway_n8n_healthy` via a `trap … EXIT` (fires on every path). Alerts `GatewayWatchdogHeartbeatStale` (with **`absent()`** clause — closes "no-data=no-alert") + `GatewayWorkflowInactive`, both tier=1+critical → **Twilio SMS** (the watchdog's own Matrix alerts are muted). Who-watches-the-watchdog gap from the months-dark failure class. Holistic §38 `watchdog-deadman`. Memory: [`memory/watchdog_deadman_20260621.md`](memory/watchdog_deadman_20260621.md).
- **Synthetic-incident canary (IFRNLLEI01PRD-1154, LIVE 2026-06-21)** — [`docs/runbooks/synthetic-incident-canary.md`](docs/runbooks/synthetic-incident-canary.md). `synthetic-incident-canary.sh` (cron `37 2 * * *`) probes the classify→predict spine against an **isolated `mktemp` DB** (never the live gateway.db) — structurally cannot pollute tables, collide a real fail-closed gate, or trigger real remediation. Metrics `synthetic_incident_canary_stages_passed`/`_live_db_leak`; alert `SyntheticCanaryLeak` (tier=1 SMS) + `SyntheticCanaryFailing`/`Stale` (warning). Holistic §38 `synthetic-canary`. Memory: [`memory/synthetic_canary_20260621.md`](memory/synthetic_canary_20260621.md).
- **Scheduled-reboot suppression (self-learning, LIVE 2026-06-29)** — [`docs/runbooks/scheduled-reboot-suppression.md`](docs/runbooks/scheduled-reboot-suppression.md). Tier-1 phase SR (`scripts/lib/tier1_suppression.py`, between blast-radius + knowledge-pattern) suppresses an on-schedule reboot on a host with a live/unkilled/unexpired registered schedule, before YT-create + Claude session; **two-phase verify** reopens + pages if the boot wasn't a clean `systemd-reboot`. Self-learning: `discover-scheduled-reboots.py` + `classify-reboot-alert.py` register `observing`; `promote-scheduled-reboots.py` → `live` after ≥2 in-window boots. Safety floor: env/sentinel `~/gateway.sched_reboot` (dark off) · critical-never · reboot-rule allowlist · observe-before-live · kill_switch/valid_until in SQL · strict DST-correct window (vendored croniter) · fail-open. Holistic §43; 5 Cronicle jobs; alerts `ScheduledReboot{Misclassified,MetricsStale,PromotionStuck}` (tf-twin deferred). Activate: `touch ~/gateway.sched_reboot`; kill: `rm`. Memory: [`memory/scheduled_reboot_suppression_build_20260629.md`](memory/scheduled_reboot_suppression_build_20260629.md).
- **Orchestrator control-plane (epic IFRNLLEI01PRD-1421, LIVE 2026-06-26)** — the governing liveness/coordination layer over the agentic federation (~10 subsystems, inventoried as 320 components), the answer to the 2026-06-25 dark-component class (MemPalace/self-audit ran dark for weeks because nothing owned their liveness as a set). 3 self-monitoring bricks: **Registry** (`scripts/registry-{seed,curate,check}.py` + [`config/component-registry.json`](config/component-registry.json), 320 components = 176 cronicle-job + 61 prom-writer + 56 n8n-workflow + 27 db-table — 12 critical / 0 critical-dark / 37 known-dark, `RegistryCriticalDark` tier1; crons `*/30` + daily); **Interaction Graph** (`scripts/interaction-graph.py` + `config/interaction-graph.json`, static-analyzes 247 scripts → 22 multi-writer CONFLICTs / 0 orphan GAPs / 0 CRON-CLASHes; daily); **Orchestration Benchmark** (`scripts/orchestration-benchmark.py` + `config/orchestration-scorecard.json`, score 1.0, 4/4 invariants incl. I1 irreversible-never-auto-resolved over a synthetic stream; weekly). Decision plane: 3-band autonomy-forward gate + fail-closed prediction gate over 559 decisions (76% auto-approved) on a SHA-256 tamper-evident hash-chained log (`GovernanceChainBroken` tier1). Observability: unified logging → self-hosted OpenObserve + Langfuse traces + OTLP fresh-push; 1530 metric series / 63 writers / 35 agentic alerts + dead-man heartbeat + synthetic canary + repo-deploy-drift guard. Realtime view: **"Orchestrator Control-Plane — Realtime Overview"** at [grafana.example.net/d/orchestrator-ctrl-plane](https://grafana.example.net/d/orchestrator-ctrl-plane) (31 panels / 6 sections, source [`grafana/orchestrator-control-plane.json`](grafana/orchestrator-control-plane.json)). 5 PrometheusRules in-cluster (IaC !347/!348), fault-injection-proven. QA `scripts/qa/suites/test-1421-orchestrator.sh`. Benchmark: **B+ (3.48/5)** across 11 dims vs industry standards ([`docs/orchestrator-plane-benchmark-2026-06-26.md`](docs/orchestrator-plane-benchmark-2026-06-26.md)). Findings: [`docs/orchestration-findings-2026-06-26.md`](docs/orchestration-findings-2026-06-26.md). Memory: [`memory/orchestrator_control_plane_20260626.md`](memory/orchestrator_control_plane_20260626.md).
- **Plane-A platform controller — the orchestrator's self-healing ACTUATOR (IFRNLLEI01PRD-1421 extension, LIVE + ARMED 2026-06-26)** — [`docs/runbooks/platform-controller.md`](docs/runbooks/platform-controller.md). The 3 bricks above OBSERVE; this ACTS. A k8s-style reconcile loop (`scripts/platform-controller.py`, `*/5` Cronicle job) that keeps the **platform** alive — **NEVER its mission** (no VM-resize / host-reboot / incident-resolve; that stays in the autonomy-forward + fail-closed-prediction lane — like k8s keeping pods alive vs deciding app logic). Heals: n8n critical-workflow reactivation (monitors all 58) + failed **SAFE-LIST** gateway Cronicle-job re-run (idempotent regenerators only) + Cronicle restart + the consolidated `gateway-watchdog.sh --heals-only` heal-library (n8n-restart / Bridge-bounce / zombie + stale-lock). k8s guardrails: per-target heal cap 3/hr → CrashLoopBackOff → `platform_controller_escalations` → tier-1 **SMS**. GATED, ships dark via `~/gateway.platform_controller_armed` (`rm` = analysis-only). Registered `prom:platform_controller` CRITICAL = its own dead-man — it **consolidated** the gateway-watchdog dead-man + heals into one operator (watchdog standalone Cronicle job `emqurqydu5t` DISABLED = the rollback path). **Honest scope: COMPLETE oversight of the scheduler (per-job run history + failure-naming + OpenObserve logs + alerts), but DELIBERATELY BOUNDED control — quarantine / idempotent-safe-re-run / restart only; it never re-runs non-idempotent jobs or fixes a job's root-cause bug, it escalates.** Alerts `PlatformController{Escalation,Stale}` (infra MR !351). Memory: [`memory/platform_controller_20260626.md`](memory/platform_controller_20260626.md).
- **Scheduler is now native CRONICLE, not raw crontab (LIVE 2026-06-26)** — 180 cron jobs migrated off crontab to native Cronicle on nl-claude01 (107 gateway + 72 agora-quant), the platform-controller's heal/oversight surface. Per-job run history + failure-naming + dead-man monitoring. Memory: [`memory/cronicle_migration_20260626.md`](memory/cronicle_migration_20260626.md).
- **Intersite tunnel auto-heal — 3 layers, LIVE + ARMED 2026-08-14 (IFRNLLEI01PRD-1833, gateway MRs !211/!212, infra !463)** — [`docs/runbooks/intersite-tunnel-autoheal.md`](docs/runbooks/intersite-tunnel-autoheal.md). Auto-recovery for the wedged-SA NL↔GR partition class (5 incidents): **L0** `vti-{freedom,budget}-recovery.sh` (Cronicle `*/3`, light `clear crypto ipsec sa peer`) now pass global MUTATIONS=OFF shadow via the **audited intersite exemption lane** (`~/gateway.mutations_intersite_allow`, `mutation_shadow_exempt` in `scripts/lib/suppression-gates.sh` — first lane user of the TG-pattern); **L1** on-device nlrtr01 `ip sla 10`→`track 10` (down 90 s)→EEM `INTERSITE-BUDGET-LEG-HEAL` clears the crypto session (~90 s, edge-triggered = storm-proof, survives nl-claude01 down); **L2** [`scripts/intersite-tunnel-heal.py`](scripts/intersite-tunnel-heal.py) invoked by `bgp-mesh-watchdog.sh` each `*/5` cycle — ≥2-run down-streak + maintenance/chaos + shadow-exemption + arming sentinel + exponential backoff/3-strike-escalate + wedge signature (GR public pingable AND decaps frozen ×2 samples or no-SA-data) → **full both-ends runbook re-key via netmiko** (GR through the `:2222` stone), verify, Matrix, metrics. Kill: `rm ~/gateway.intersite_autoheal_armed` (L2) / `rm ~/gateway.mutations_intersite_allow` (lane). Both sentinels in master-switch `ARMING_SENTINELS`. Alerts `IntersiteAutoheal{Failed,Escalated,Stale}` in-cluster. Live-drilled both legs day one (EEM fire→rebuild <60 s; `--drill-force` 58 s/leg). QA `test-intersite-autoheal.sh` 11/11. Memory: [`memory/intersite_autoheal_build_20260814.md`](memory/intersite_autoheal_build_20260814.md).

- **Headless Intel AMT console (OOB KVM without a browser)** — [`docs/runbooks/amt-headless-console.md`](docs/runbooks/amt-headless-console.md). How to get keyboard+screen on an off-network PVE host from nl-claude01. **AMT 16+ (MS-01) REMOVED the standard VNC-5900 feature** (WS-Man Put returns "feature is not supported") — the only transport is Intel redirection on 16994/16995, bridged to a local VNC port via `nomis/intel-amt` + `vncdotool` (legacy-TLS ctx patch required; a fresh session eats the first keystrokes and the first capture is stale). MeshCentral-via-Playwright is the fallback (⛔ never type passwords onto the canvas — use the Type dialog). Also covers power ops (per-host opcode support differs), and the **post-emergency half-init trap** (`systemctl list-units --state=running | wc -l` vs a healthy peer → reboot; `pvesh create /nodes/localhost/startall` since pve-guests refuses manual start). Proven on gr-pve01 2026-08-25 (IFRGRSKG01PRD-312).

- **Reviving a wedged/vanished NVMe with a cold AC drain (PROVEN 2026-08-27 gr-pve01 · 2026-09-16 nl-pve01)** — [`docs/runbooks/pve-nvme-cold-drain-recovery.md`](docs/runbooks/pve-nvme-cold-drain-recovery.md). **Two triggers: (a) the disk vanished from PCIe; (b) it is still enumerated but stuck — `Unable to change power state from D3cold to D0, device inaccessible` (variant (b) can take out SEVERAL drives at once, incl. both halves of a mirror, and leaves the host a RAM-zombie: ICMP up + corosync member but SSH dead at banner).** ⭐⭐ **Absence from PCIe enumeration is NOT proof of hardware death** — a hung NVMe controller can fail to train the link and still be fully recoverable; a warm reboot only asserts PERST# (link reset, not controller reset) and **S5 was not sufficient either**. Only full AC removal forces a POR. Recovered `S7NUNE0X619155` on gr-pve01 after 2 days absent (`power_on_hours` returned **identical** to the value at death = proof it was truly dead; 0 media errors, ext4 clean). Covers: identifying an already-gone disk from **LibreNMS per-serial RRDs**, the ⛔ outlet-name-lies guard + verified GR PDU map, the peer-reboot hard gate, and why **AMT goes dark in S5 ⇒ no remote wake** (board auto-resume on AC is the only path back). Host downtime ~10 min.
- **PVE host Prometheus exporters (pve-exporter :9221 + node_exporter :9100, LIVE 2026-08-25; per-site scraping 2026-08-26)** — [`docs/runbooks/pve-host-exporters.md`](docs/runbooks/pve-host-exporters.md). Native on all 5 live PVE hosts (nl-pve02 excluded, off by design), installer/drift-check `scripts/pve-host-exporters-install.sh`, identity `prometheus@pve!pve-exporter` (PVEAuditor). Scraping is per-site via canonical `var.pve_hosts` jobs in `k8s/namespaces/monitoring/main.tf` (all 3 repos, NL !521 / GR !133 / NO !32); alerts split: site-local liveness/pressure in canonical `pve-host-alerts.tf` (incl. the `/ on (instance)` PVELoadHigh fix — the April-era expr could never match) / cluster-view group NL-only in `host-pressure-alerts.tf` (infra !517); Grafana dashboard `pve-exporter-10347`. Key fact: pve-exporter 3.x serves the WHOLE cluster view from every host (`?cluster=1`) — rules dedup `max by (id)`; per-host scrape duration = that host's API/pmxcfs canary. Test copy `prometheus/alert-rules/pve-host-health.yml` (+fixtures, QA test-726). Both alert paths e2e-drilled (NL: YT -314; GR: YT IFRGRSKG01PRD-316).

Other ops docs: [`docs/rag-architecture-current.md`](docs/rag-architecture-current.md) (auto-refreshed), [`docs/rag-metrics-reference.md`](docs/rag-metrics-reference.md), [`docs/crontab-reference.md`](docs/crontab-reference.md), [`docs/network-addresses.md`](docs/network-addresses.md), [`docs/host-blast-radius.md`](docs/host-blast-radius.md), [`docs/scorecard-post-agents-cli-adoption.md`](docs/scorecard-post-agents-cli-adoption.md).

---

## Maintenance Mode

When `/home/app-user/gateway.maintenance` exists (JSON with `started`, `reason`, `eta_minutes`, `operator`), alert processing is suppressed across all receivers, watchdog, and OpenClaw triage. Created/removed by AWX playbook or manually. 15-minute post-maintenance cooldown tags alerts as `post-maintenance-recovery`.

**Key facts:**
- **ASA weekly reboot: DISABLED (2026-04-10).** EEM watchdog applets removed from both ASAs after weekly reboots caused VTI tunnel instability and cascading cross-site outages. `asa-reboot-watch.sh` cron commented out. Manual reloads use the maintenance companion (`/maintenance`).
- **Freedom ISP:** dual WAN with SLA failover + QoS toggle (with ping fallback) + SMS alerts via Twilio.
- **PVE kernel:** AWX cross-site automation (GR from NL template 69, NL from GR template 21).
- **Inter-site routing:** Full BGP via direct peering over VTI. No static inter-site routes (2026-04-10). 8 FRR instances (4 RRs + 2 VPS + 2 ASAs as RR clients).

Full BGP/FRR/VPS-peering technical detail in [`.claude/rules/infrastructure.md`](.claude/rules/infrastructure.md). Maintenance flow + ASA reboot suppression detail in [`docs/maintenance-mode-details.md`](docs/maintenance-mode-details.md).

---

## Known Host: nl-pve01 — storage-loss (D3cold) AND pmxcfs-wedge failure modes

**⛔⛔ 2026-09-19 — 3rd D3cold (this time the survivor `7VS00ZJ8`) → cold drain #3 restored storage, but BOTH copper SFP-10G-T bond legs are dead → the host runs on a TEMPORARY 1 G uplink (IFRNLLEI01PRD-2871).** The outage went ~5.5 h unnoticed because nltg01 (primary alert intake) died with the host (IFRNLLEI01PRD-2868). Current state: nl-sw01 **Te1/0/37** (the AMT port) is a trunk with native VLAN 10 (**not saved**), and `enp109s0` is in vmbr0 with bond0 detached (**runtime only**). ⛔ **A reboot of nl-pve01 means no network again** until the modules are replaced and the uplink reverted (steps in -2871 and `.claude/rules/infrastructure.md` § Known Host: nl-pve01). ⛔ Every AC drain has killed one copper leg (09-16 Te1/1/3, 09-19 Te1/1/4), and software resets (flap, shut/no shut, i40e reload, warm reboot) are proven useless. Before any future drain, have hands on site or the 1 G fallback ready ([`docs/runbooks/pve-nvme-cold-drain-recovery.md`](docs/runbooks/pve-nvme-cold-drain-recovery.md) § 1 G fallback). Lyrion (LMS) runs in nlservarr01 on this host; its 7 pCP speakers reboot-loop while it is down.

**⛔⛔ 2026-09-17/18 — BOTH rpool FireCuda 530s are FAULTY (controller stall); ALL P0 EVACUATED; host is write-fragile until re-drived (IFRNLLEI01PRD-2864).** The leg lost on 09-17 (`7VS00YN2` @ 02:00.0) was revived by a 2nd PDU cold drain but returned at ~1 write/s → **OFFLINE by choice**; the survivor `7VS00ZJ8` then stalled three times in 45 min under ~5 MB/s of ordinary writes (400-1200 ms w_await, 40-100 D-state, self-releasing after 1-5 min, no kernel NVMe error). Power management + thermal are **eliminated by measurement** (ASPM/APST params live since 2026-06-23, `runtime_suspended_time` 0 ms, 78.8 °C at stall). **Standing rules until both drives are replaced:** zero P0 here (freeipa01 npm01 matrix01 smtp-gpg01 smtp-dkim01 vaultwarden01 oas01 wg01 → nl-pve03; pihole01 netbox01 protonmail-bridge01 oas02 → nlpve04); **k8s-node01 stays CORDONED; GitLab runner 10 stays PAUSED; NO inbound migrations/copies of any size** (a throttled 80 MB/s stream stalled the leg at 2 GB); on a stall **wait ~8 min** before any PDU action — they self-clear. Seagate RMA **`IR60507401`** is open for `7VS00YN2` (standard replacement; erase on the operator's laptop before shipping — procedure in `.claude/rules/infrastructure.md` § Known Host: nl-pve01). `cpu: host` VMs cannot live-migrate nlpve04 (AMD) ↔ pve01/pve03 (Intel). HAHA + FISHA fence via `fence_pve` → this host's API, so a pve01 outage takes Home Assistant + NFS down until it returns (IFRNLLEI01PRD-2860). Full record: [`memory/pve01_p0_evacuation_and_cold_drain_inflight_20260917.md`](memory/pve01_p0_evacuation_and_cold_drain_inflight_20260917.md); guest map: [`docs/nl-guest-criticality.md`](docs/nl-guest-criticality.md). **⚠ 2026-09-18:** nltg01 (territory-grounder = primary alert intake) still runs here — de-facto P0, move pending (IFRNLLEI01PRD-2868); the uplink bond is single-leg too (Te1/1/3 dead since the 09-16 drain, IFRNLLEI01PRD-2865).


**⛔⛔ 2026-09-16 — BOTH rpool NVMe lost to a D3cold power-state event (IFRNLLEI01PRD-2853).** `nvme 0000:59:00.0` + `nvme 0000:02:00.0` → `Unable to change power state from D3cold to D0, device inaccessible`, 13 s apart. Those are the **two FireCuda 530s forming `rpool` mirror-0**, so a *system-level* PM event beat the mirror — redundancy does not protect against this. 65 guests dark ~40 min. Fixed by an operator-authorised **PDU cold drain** (`nlpdu01` outlet 5, 5 m 35 s AC removal): both drives back, **rpool ONLINE 0 errors, no data loss**, host down ~8 min, etcd quorum never lost. ⚠ **ROOT CAUSE UNFIXED (ASPM/APST?) → expect recurrence.**

**⛔⛔ Do NOT confuse the two failure modes.** Storage-loss presents as a **RAM-zombie**: ICMP UP at normal latency, corosync still a voting member, node `status: unknown` — but **SSH dies at banner exchange** and SNMP is dead, with **normal load**. The pmxcfs wedge instead shows **load 100+ with CPU idle** and D-state procs at `wchan=filename_create`. The wedge remedy (`systemctl restart pve-cluster`) is useless for storage loss and needs a shell you do not have. ⭐ **Get the AMT console early** — on 2026-09-16 the session called it as the wedge class first, and only the console screenshot corrected it. Power-work facts (PDU outlet 5, **Master Control = outlet 17**, AMT shares the outlet so there is no wake path while off, zero quorum margin with nl-pve02 down, `/dev/nvme*` renumbers on cold boot): [`.claude/rules/infrastructure.md`](.claude/rules/infrastructure.md) § Known Host: nl-pve01 + [`docs/runbooks/pve-nvme-cold-drain-recovery.md`](docs/runbooks/pve-nvme-cold-drain-recovery.md) § variant (b).

**CORRECTED 2026-06-30:** n8n LXC (`nl-n8n01`, CT VMID_REDACTED) is on **nlpve04**, confirmed live via `pvesh get /cluster/resources` (vmid VMID_REDACTED → node nlpve04). The prior "n8n lives on nl-pve01" was stale VMID-node-digit decode drift — n8n was NOT affected by the 2026-06-30 nl-pve01 power-cycle. **What IS on nl-pve01 (and pressured):** ~40 onboot LXCs/QEMUs incl. matrix (`nl-matrix01`, CT VMID_REDACTED), NPM (`nlnpm01`), FreeIPA, Pi-hole, scanner (`nlsec01`), NetBox, HAProxy, code-server, and NFS server (`nlcl01file01`). nl-pve01 has repeatedly **wedged its pmxcfs** under load (2026-06-23/-27/-30): the signature is `load-avg very high (100+) + CPU ~idle + guest status=unknown + D-state procs wchan=filename_create on /etc/pve` = hung pmxcfs (NOT CPU/IO). Wedge AMPLIFIER found 2026-06-30: `scripts/lab-stats.py` pinned its `pvesh /cluster/resources` to pve01 with no server-side timeout → each call during a stall strands a permanent D-state orphan (D-state ignores SIGKILL) → 134 piled → pmxcfs deadlock. Fixed: lab-stats now queries a healthy node first (pve03/04/02, pve01 last) + `timeout 20 pvesh` (claude-gateway MR !130, merged). Fix WITHOUT reboot (proven 2026-06-27 pve04): `systemctl restart pve-cluster` FIRST (FUSE teardown releases D-states) THEN `reset-failed pvestatd && restart pvestatd`. **Dedicated wedge detection LIVE 2026-06-30 (IFRNLLEI01PRD-1501):** the generic `NodeSaturation` alert mis-reads the wedge as CPU, and until 2026-08-25 NO PVE host was a node_exporter/Prometheus target (the `PVELoadHigh`/`PVEMemoryPressure*` rules in `host-pressure-alerts.tf` were silently inert). **SUPERSEDED IN PART 2026-08-25/26:** all 5 live PVE hosts now run native `prometheus-pve-exporter` (:9221) + `prometheus-node-exporter` (:9100), scraped per-site (canonical `var.pve_hosts` jobs, 2026-08-26), so the pressure rules are LIVE plus a `pve-exporter` cluster-view alert group — see [`docs/runbooks/pve-host-exporters.md`](docs/runbooks/pve-host-exporters.md); the wedge chain below stays as the complementary D-state/pmxcfs canary. New chain: `scripts/write-pve-wedge-metrics.sh` on nl-claude01 (Cronicle `pve-wedge-metrics` `*/2`, id `emr0p04dnkl`) SSHes **all 6 PVE hosts** (`PVE_WEDGE_HOSTS` default = nl-pve01-04 + gr-pve01/02; extended 2026-07-09 after **nl-pve02 ALSO wedged** — MR !180) and emits per-host `pve_wedge_*` (dstate_procs / pmxcfs_probe_seconds / pmxcfs_probe_ok / guests_status_unknown / collector_up / collector_last_run); 3 alerts in `host-pressure-alerts.tf` group `pve-pmxcfs-wedge` (infra MR !354, Atlantis-applied + merged, PrometheusRule gen 2): `PVEPmxcfsWedgeForming` (warn), `PVEPmxcfsWedged` (critical/tier1 → Twilio SMS), `PVEWedgeCollectorStale` (dead-man). See [`memory/pve01_pmxcfs_wedge_lab_stats_amplifier_20260630.md`](memory/pve01_pmxcfs_wedge_lab_stats_amplifier_20260630.md) + [`memory/feedback_pve_mgmt_wedge_pmxcfs_restart.md`](memory/feedback_pve_mgmt_wedge_pmxcfs_restart.md) + [`memory/pve02_pmxcfs_wedge_20260709.md`](memory/pve02_pmxcfs_wedge_20260709.md) (2nd host to wedge; collector + `PVEPmxcfsWedge*` alerts now cover all 6 PVE). **Since 2026-08-24 the collector covers 5 PVE hosts:** nl-pve02 is excluded via a `PVE_WEDGE_HOSTS` env export in the Cronicle job script (see next paragraph); `PVEPmxcfsWedged` tier-1 SMS was commented out 2026-08-01 (operator SMS triage).

## Known Host: nl-pve02 — POWERED OFF (since 2026-08-24)

nl-pve02 (a nested PVE node = VM on nl-nas01's VMM) was **evacuated and powered off 2026-08-24** — its memory footprint starved the NAS (kswapd thrash → iSCSI stalls → estate ext4-ro epidemic). Cluster `eu-nlgr-pvecl01` quorate **5/6 = the healthy baseline** (operator directive; LibreNMS `cororings --rings 5` both sites, device `ignore=1`, wedge collector trimmed, NetBox VM 200 Offline, VMM autostart disabled). Its 7 LXCs live on nl-pve01 (k8s-ctrl02, cl01garbd01) + nlpve04 (k8s-openbao02, redis02, semaphore01, meshcentral01, gitea01). Stays off until the revival-vs-deletion decision — **IFRNLLEI01PRD-2646 carries the full revert list**. Detail + the nested-bridge migration rule (untagged NIC on trunk vmbr0 = boots-fine-but-network-dead): [`.claude/rules/infrastructure.md`](.claude/rules/infrastructure.md) § Known Host: nl-pve02 + memory `pve02_evacuation_bridge_retag_completion_20260824`.

Full detail (remediation steps, IFRNLLEI01PRD-622/-692/-704 history, prior failure modes, and the 2026-09-16 D3cold storage-loss event): [`.claude/rules/infrastructure.md`](.claude/rules/infrastructure.md) § "Known Host: nl-pve01" (section created 2026-09-16 — this pointer was previously dangling).

---

## Operating Modes

> **2026-06-28 status:** the mode abstraction is **vestigial and slated for retirement.** Only `cc-cc` is live. OpenClaw was retired 2026-04-29 and its LXC (`VMID_REDACTED` / `nl-openclaw01`) has since been **destroyed** ("not found on any node") — the `pct start VMID_REDACTED` restore path below is **broken**. All 9 alert receivers SSH-direct to `scripts/run-triage.sh`. `~/gateway.mode` was corrected to `cc-cc` on 2026-07-03 (had been stale at `oc-cc` since 2026-03-11); it is cosmetic either way (dispatch is hardwired, not read from the file). Model selection is now centralized via the [Model Orchestration](#model-orchestration-centralized-2026-06-28-mrs-116120) layer, which supersedes the frontend/backend-pairing concept these modes encoded.

The file `/home/app-user/gateway.mode` historically controlled which frontend/backend pair is active.

| Mode | Frontend | Backend | Status |
|------|----------|---------|--------|
| `cc-cc` | n8n/Claude Code | Claude Code (direct SSH dispatch) | **LIVE — the only functional mode** |
| `oc-cc` | OpenClaw | Claude Code via n8n | **Dead** (OpenClaw LXC destroyed) |
| `oc-oc` | OpenClaw | OpenClaw/Sonnet 4.6 (self-contained, OAuth Max) | **Dead** (OpenClaw LXC destroyed) |
| `cc-oc` | n8n session mgmt | OpenClaw as backend (via docker exec) | **Dead** (OpenClaw LXC destroyed) |

Switch modes with the `!mode <mode>` command in any Matrix room where OpenClaw is present. ⚠ Restoring `oc-cc`/`oc-oc`/`cc-oc` is **no longer possible** — it required `pct start VMID_REDACTED` on `nl-pve03`, but that LXC no longer exists; it would have to be rebuilt from scratch. Do not rely on this path.

**Mode change history** (1-line summaries; full memory pointers):
- **cc-cc migration (2026-04-29, commit `484f5da`):** 9 alert receivers SSH directly to nl-claude01 and invoke `scripts/run-triage.sh` instead of routing via OpenClaw. Driven by Anthropic April-4 OAuth-for-third-party ban + OpenClaw 2026.4.26 MCP-bind regression. Durable structural check: `holistic-agentic-health.sh §38 cc-cc-receiver-wiring`. Full memory: [`memory/cc_cc_migration_complete_20260429.md`](memory/cc_cc_migration_complete_20260429.md), [`docs/openclaw-retirement-complete-2026-04-29.md`](docs/openclaw-retirement-complete-2026-04-29.md).
- **Tier 1 model migration (2026-04-28, IFRNLLEI01PRD-746):** OpenClaw Tier 1 switched from `openai/gpt-5.1` (paid) to `claude-cli/claude-sonnet-4-6` (Max-subscription OAuth, $0). Fallback ladder: Opus 4.6 → Opus 4.5 → Sonnet 4.5 → Haiku 4.5. Full reference: [`docs/model-provenance.md`](docs/model-provenance.md).

---

## Conventions

### Stable rules

- Branch naming: `feature/description` or `fix/description`
- Create MRs, don't push directly to main
- Workflow names prefixed with `"NL - "`
- Export workflow JSON after every change via n8n-mcp, save to `workflows/`
- Workflow JSON filenames: `claude-gateway-{workflow-slug}.json`
- n8n node versions: use httpRequest v4.2, webhook v2
- n8n version: **2.11.3** (community edition)
- n8n-mcp version: 2.61.0
- **Switch V3.2 known issue (n8n 2.41.3):** Rules created via API/MCP omit `conditions.options` block, causing `extractValue` crash. ALWAYS include `conditions.options: {version: 2, caseSensitive: true, typeValidation: "strict"}` in each rule's conditions when creating Switch V3.2 nodes programmatically. Compare with LibreNMS receiver "Repeat Action" node for reference. After any workflow update via API, toggle deactivate→activate to reload webhook listeners.

### Agent-output verification (codified 2026-06-26)

When a sub-agent or workflow produces docs, audits, counts, or findings, **verify every concrete claim**
(section #, count, file path, metric, git status, "X is untracked/broken/dark") against the LIVE source
before acting on it or committing it. Sub-agents fabricate plausible-but-false specifics that read as
authoritative and are invisible in the diff — e.g. a 2026-06-26 cross-audit's headline "git-untracked
config" finding was simply false (it ran `git status` on a diverged working branch, not `main`). Tell
doc-agents to read the owning file for counts, not the prompt. Source: `memory/feedback_verify_agent_generated_doc_claims.md`.

### [P0] Operator-anger rules — must stay verbatim

- **[P0] Full hostnames, no exceptions:** ALWAYS use full site-prefixed hostnames (nl-pve01 not pve01, nlcl01iot01 not iot01, nlcl01file02 not file02, gr-pve01 not pve01). Never use generic role labels ("the ASA", "the router", "the active node") as a substitute. Applies to all output: playbooks, comments, memory, YT, Matrix messages, tables, diagram labels, filenames. Reinforced 2026-04-30 after multiple session slips.
- **[P0] VLAN naming — never use the subnet third octet as a VLAN tag:** `10.0.181.X/24` is **inside_mgmt VLAN 10**, not "VLAN 181". `10.0.X.X/27` is the storage subnet, not "VLAN 88". `10.0.X.X/28` is **VLAN 12 (CCTV)**, not "VLAN 183". The third IP octet is not the 802.1Q tag. Refer to subnets by name (inside_mgmt), tag (VLAN 10), or CIDR (10.0.181.X/24) — never by octet-as-tag.

### Workflow-edit gates (1-line; full detail in runbooks)

- **Code-node edits require validator (post-14h-outage gate):** Before any `curl -X PUT /api/v1/workflows/<id>` that modifies a Code node's `jsCode`, run `scripts/validate-n8n-code-nodes.sh` and confirm **VALIDATION PASSED**. Full runbook: [`docs/runbooks/n8n-code-node-safety.md`](docs/runbooks/n8n-code-node-safety.md).
- **Risk-based auto-approval (IFRNLLEI01PRD-632, 2026-04-19):** Runner classifies session risk, Build Prompt injects directive, Bridge parses `[AUTO-RESOLVE]`. Full runbook: [`docs/runbooks/risk-based-auto-approval.md`](docs/runbooks/risk-based-auto-approval.md).

### Platform features (full detail in `.claude/rules/platform-features.md`)

The following systemic platform features are documented in [`.claude/rules/platform-features.md`](.claude/rules/platform-features.md) under their dated section headers — load that file when working on any of these surfaces:

- RAG Circuit Breakers (2026-04-19)
- Schema Versioning (2026-04-20) — bump `CURRENT_SCHEMA_VERSION[table]` + add to `SCHEMA_VERSION_SUMMARIES[table]` whenever any payload column's JSON shape changes
- OpenAI SDK Adoption Batch (2026-04-20, IFRNLLEI01PRD-635..-643)
- QA Suite (2026-04-20, expanded 2026-04-23) — `scripts/qa/run-qa-suite.sh`, **run after any change to the adoption-batch surfaces or the patcher**
- Teacher-Agent — All 5 Tiers (2026-04-20, hardened 2026-04-23)
- CLI-Session RAG Capture (2026-04-20)
- Preference-Iterating Prompt Patcher (2026-04-20)
- NVIDIA DLI Cross-Audit + P0+P1 (2026-04-29)

### Recent incident records (1-line; full detail in memory)

- **nl-pve01 3rd D3cold → drain #3 → both copper SFP legs dead → TEMP 1 G uplink (2026-09-18/19, IFRNLLEI01PRD-2871):** the survivor `7VS00ZJ8` went `D3cold → D0 inaccessible` at 09-18 ~20:50 NL. **No page for ~5.5 h**, because TG (nltg01, primary intake) died with the host (-2868). Found only when the operator asked why the Lyrion speakers were red (LMS = nlservarr01 on pve01). The AMT console confirmed it; operator-authorised drain at 02:25:56 NL (outlet 5, 5 m 52 s) → rpool ONLINE, 0 errors. Both X710 ↔ nl-sw01 Te1/1/3/Te1/1/4 copper SFP-10G-T legs stayed dark through 5 remote resets → temporary uplink via Te1/0/37 (AMT port, trunk native 10, unsaved) + `enp109s0` in vmbr0 (runtime) → quorum 5 at 03:27 NL, 36/36 guests by 03:37 NL, all P0-in-practice and P1 services functionally verified. HAHA held via SBD (fence_pve failed again, -2860). Memory: [`memory/incident_pve01_down_again_tg_intake_blind_20260918.md`](memory/incident_pve01_down_again_tg_intake_blind_20260918.md), rule [[feedback_pve01_cold_drain_kills_copper_sfp_leg]].
- **nl-nas01 Volume 1 at 91 % → PBS retention + backup-coverage fix (2026-09-18, IFRNLLEI01PRD-2867/-2868/-2865):** culprit = nlpbs01 `ds01` never pruned anything (`keep-all=1` everywhere, no prune job, **GC yearly**) → +0.5 TB/week, 15,388 snapshots back to 2023. Operator-chosen policy (last 3 · daily 7 · weekly 8 · monthly 12) as PBS prune job `ds01-retention` + weekly GC; first prune removed 8,515 snapshots; GR grpbs02 aligned to the same policy. **Then a full-volume inventory** (the first answer named only the fastest grower): removed 1.34 TB of junk (orphan qcow2 of non-existent VMID 104, dead `syslog/opensearch`, 9.5k stale NZBGet downloads, fio files, orphan LUN) + the dead **Bacula** volumes (2.81 TB); added syslog gzip/expiry (`syslog-archive.sh`, nl-fw01 ≈ 4 GB/day at 14.5:1), Syncthing `cleanoutDays=30`, shorter Frigate retention → **6.3 → 12 TiB free**. k8s synology-csi LUNs (~3 TB) have UNMAP off — k8s-side deletes never free NAS space. Space owners + open decisions: `.claude/rules/infrastructure.md` § nl-nas01 Volume 1. Same day: **NL vzdump jobs are node-pinned VMID lists**, so the 09-17 moves had silently dropped 26 running guests (every moved P0, k8s-ctrl02 unbacked since 08-11) out of backups — fixed. Also: 19 guests stopped + `onboot=0` (2026-09-19: nlandroidsdk01 restarted at 8 GiB, nlgitlabrunner02 stopped in its place), nlfrigate01 → nlpve04 (software decode, infra MR !573) to relieve pve03 RAM, LibreNMS noise cleared (ignore/ack; rule 23 "TG Transport Selftest" is TG's own feed — keep). Memories: [`memory/syno01_volume1_growth_pbs_keep_all_20260918.md`](memory/syno01_volume1_growth_pbs_keep_all_20260918.md), [`memory/vzdump_node_pinned_jobs_migration_gap_20260918.md`](memory/vzdump_node_pinned_jobs_migration_gap_20260918.md).
- **nl-pve01 rpool leg lost AGAIN → P0 evacuation + 2nd cold drain + BOTH-drives verdict (2026-09-17/18, IFRNLLEI01PRD-2853/-2860/-2861/-2864, gateway MR !228 + close-out MR):** 18:53Z the same `7VS00YN2` dropped (single leg this time; host survived on the mirror). Found by root's `zfs-zed` mail — **ZFS pool state is exported (`node_zfs_zpool_state{state="degraded"}`) but no alert rule exists** (-2861). Evening: all 7 P0 + 5 P1 migrated off pve01 in a RAM-balanced swap (operator rules: stop nothing, balanced, permanent); the survivor stalled pool-wide under the inbound copies → operator-pre-authorised autonomous PDU cold drain 22:49Z (AMT dark in 5 s = proof), back 22:57Z, leg re-enumerated **but at ~1 write/s → offlined**; Home Assistant was down ~15 min because HAHA/FISHA fence through pve01's API (-2860); protonmail-bridge01 reclassified P0 and moved; then the survivor stalled 3× under trivial load → **both FireCudas faulty, RMA both, different model** (-2864; Seagate warranty to 2028-03-23; RMA `IR60507401` opened 09-18 for `7VS00YN2`). 16 P2 stopped+disabled by operator. New standing rules: [[feedback_cpu_host_vms_cannot_live_migrate_amd_intel]], [[feedback_git_push_token_url_lands_in_shared_config]], node_exporter binds the mgmt IP not localhost, `pkill -f` self-matches the invoking shell. Memory: [`memory/pve01_p0_evacuation_and_cold_drain_inflight_20260917.md`](memory/pve01_p0_evacuation_and_cold_drain_inflight_20260917.md), [`memory/incident_pve01_nvme_d3cold_single_leg_20260917.md`](memory/incident_pve01_nvme_d3cold_single_leg_20260917.md), [`memory/nl_guest_criticality_map_20260917.md`](memory/nl_guest_criticality_map_20260917.md), [`memory/nl_guest_purposes_20260918.md`](memory/nl_guest_purposes_20260918.md).
- **nl-pve01 lost BOTH rpool NVMe to a D3cold event — recovered by cold drain (2026-09-16, IFRNLLEI01PRD-2853):** at 01:55:47Z `nvme 0000:59:00.0` and `0000:02:00.0` both returned `Unable to change power state from D3cold to D0, device inaccessible` 13 s apart — the two FireCuda 530s that form `rpool` mirror-0 — so the host lost **all** storage while its kernel stayed alive in RAM. **65 guests** affected (nlfreeipa01, nl-pihole01, nlnpm01, nl-matrix01, nlnetbox01, nlcl01file01 NFS, 5× k8s). ⭐ It presents as a network fault and was **initially misdiagnosed as the pmxcfs-wedge class**; the operator's AMT console screenshot is what corrected it — **ask for the OOB console early**. Recovery: operator-authorised PDU cold drain (`nlpdu01` outlet 5, guarded on the live outlet name, 5 m 35 s AC removal) → both drives re-enumerated, **rpool ONLINE, 0 errors, no data loss**, host down ~8 min, NL k8s etcd quorum never lost. Post-recovery sweep of all 65 guests: inventory reconciles, **fs integrity clean** (6/7 QEMU + 40/40 LXC all `rw`, 0 ext4/IO errors); only damage was a **stale pidfile** killing `oxidized.service`. Side-finds: **IFRNLLEI01PRD-2854** (nlghostfolio01 rootfs 25G/25G full — latent disk-full is invisible until a restart) and **IFRNLLEI01PRD-2855** (nlemailagent01 unverifiable: no guest agent, no DNS, no lease). ⚠ **Root cause (ASPM/APST) UNFIXED — this will recur.** ⚠ `/dev/nvme*` renumbered (nvme0n1 is now the Kingston); ZFS unaffected (`nvme-eui.*` keyed). Memory: [`memory/incident_pve01_nvme_d3cold_both_drives_20260916.md`](memory/incident_pve01_nvme_d3cold_both_drives_20260916.md).
- **GR cold-drain window — vanished NVMe REVIVED (2026-08-27, IFRGRSKG01PRD-312):** operator-approved maintenance window on gr-pve01 to test whether the disk lost on 08-25 was dead or merely wedged. **It revived.** `S7NUNE0X619155` (PM9F1 Samsung 2048GB) re-enumerated after a 5-min full AC drain via `grpdu02` outlet 2 — `power_on_hours` came back **byte-identical to the reading at death** (proof it was genuinely dead to the host), 0 media errors, ext4 `clean`. ⭐⭐ **The lesson: absence from PCIe enumeration is not proof of hardware death** (odds were put at ~25% precisely because no root port presented a link — that inference was wrong). Sequence: cordon → 39 guests down → **peer gr-pve02 rebooted first as a hard gate** → target `shutdown -h now` → AC off 5 min → on → **board auto-resumed** → disk present at first boot. GR outage ~27 min, host down ~10 min, **zero data loss**, 36/36 guests back, quorum never below 4/6. ⛔ **Disk deliberately NOT trusted**: fstab still commented, **swap still zero**, storage still `inactive` — needs rw `e2fsck` + burn-in first. Bonus: the chronic GR etcd/apiserver crash-loop **stopped** (p99 WAL fsync ~1000 ms → **~43 ms**, `slow fdatasync` → 0, apiserver restarts frozen) — **not declared fixed**, 16-min sample on a fresh ARC with the compactor still parked; root causes (128K raw files on `rpool/data`, 2 GiB ARC cap) unchanged, re-measure over days. New defects filed: **IFRNLLEI01PRD-2807** (node_exporter loses a boot race on every PVE reboot — confirmed 2/2 rebooted hosts, latent on the 3 NL) and **IFRNLLEI01PRD-2808** (`maintenance-companion.sh` unusable for the GR site). Runbook: [`docs/runbooks/pve-nvme-cold-drain-recovery.md`](docs/runbooks/pve-nvme-cold-drain-recovery.md). Memory: [`memory/gr_cold_drain_window_inflight_20260827.md`](memory/gr_cold_drain_window_inflight_20260827.md).
- **Estate-wide backup outage + BX500 root-cause + existing-hardware mitigation (2026-08-25, IFRGRSKG01PRD-313, GR MRs !130/!131):** all 3 velero BSLs Unavailable same day (NL filers stranded on the OFF nlk8s-node01; GR workers ext4-`emergency_ro`; NO collateral because **notrf01 backs up to `nl-s3` by design**). GR true root cause: gr-pve02 `ssd-pool` = **Crucial BX500 DRAM-less QLC** — SLC-cache exhaustion → multi-minute latency collapse → LIO ABORT_TASK → worker latches. Same-evening mitigation, fully GitOps + e2e-tested: **2nd democratic-csi backend on gr-pve01 NVMe** (`iscsi-nvme-{retain,delete}`) + 600 s SCSI/iscsid timeouts on all 3 GR workers; loki/prometheus×2/thanos-store/compactor recycled onto it (no data migration — PVC-delete-per-ordinal, warm-cache bounces). Gates: 20G soak with pve02 flat + abort-0, latch sweep 0/0/0, read canary 82 s, proof backup Completed 29 items, 3/3 BSLs Available, gr-prometheus HTTP 200. ⛔ **thanos-compactor parked at replicas=0 — since 2026-08-26 DECLARATIVELY via the `thanos_compactor_replicas` tfvar (NL=1/GR=0/NO=1; MRs NL !525/GR !134/NO !33), so an `atlantis apply` no longer re-arms it** — its backlog churn crash-loops all 3 GR apiservers (etcd co-tenancy: all 6 GR k8s VMs on pve01 rpool); the pre-fix trap fired twice (same-night concurrent apply + the 08-26 scrape-handover restore, canary 173/206/223 vs 41/70/78); **re-enable = flip the GR tfvar to 1 in a watched window, apiserver restarts = canary (baseline 177/209/225)**. ⛔ bare `kubectl get backup` resolves to CNPG's CRD — use `backups.velero.io`. Host detail: [`.claude/rules/infrastructure.md`](.claude/rules/infrastructure.md) § Known Host gr-pve02 + § gr-pve01. Memory: [`memory/estate_sweep_seaweedfs_backups_broken_20260825.md`](memory/estate_sweep_seaweedfs_backups_broken_20260825.md).
- **gr-pve01 crash + emergency-boot loop (2026-08-25, IFRGRSKG01PRD-312):** the ext4 disk holding the host's **only swapfiles physically vanished** → `Bad swap file entry` + **pmxcfs SIGBUS** + tmpfs I/O errors (looks exactly like bad RAM — it wasn't; 0 MCE/EDAC on the clean boot), then the same missing device trapped every reboot in emergency mode. All 45 GR guests down ~4 h, **zero data loss**; recovered via AMT console (fstab fix → corosync start → clean reboot), 36/36 onboot guests back, rpool ONLINE, 5/6 quorate. **Paged only as a Gatus "HA-down" SMS** because grnpm01 is one of two A-records for `homeassistant.example.net` — HAHA was healthy throughout; label fixed estate-wide to `HA-ext-URL-via-NPM-slow-or-down-from-<site>` (NL !516 / NO !30 / GR !129, byte-identical 3-repo mirror). Collateral fixes: dead `dir:` storage was reporting *active on rpool* → `is_mountpoint yes`; backup **fleecing disabled** on both jobs (the dead disk was their fleecing target — would have failed 12 QEMU VMs incl. all 6 GR k8s nodes; ⛔ do NOT repoint at rpool); **gr-awx restored** (both ingress-nginx replicas had stacked on one node post-restart, starving it). New standing rules: [[feedback_reboot_after_emergency_mode_boot]], [[feedback_pve_dir_storage_needs_is_mountpoint]], [[feedback_check_soft_antiaffinity_stacking_after_restart]]. Host detail: [`.claude/rules/infrastructure.md`](.claude/rules/infrastructure.md) § Known Host gr-pve01. Memory: [`memory/gr-pve01_hang_gatus_ha_down_20260825.md`](memory/gr-pve01_hang_gatus_ha_down_20260825.md).
- **nlpve04 host OOM killed nlk8s-node01 (2026-08-25):** kernel global OOM at 13:15:30Z killed VM VMID_REDACTED (16 GiB); NL k8s left on 3 workers, node01 **deliberately left off** — post-pve02-evacuation NL has no host with headroom for it (pve04 ~9 GiB avail even after the kill, pve03 ~11, pve01 ~21). First real casualty of the 5/6 baseline; feeds IFRNLLEI01PRD-2646. NL Gatus was down 13:16→~15:00Z as a result. Memory: [`memory/pve04_oom_killed_k8s_node01_20260825.md`](memory/pve04_oom_killed_k8s_node01_20260825.md).

- **Backlog reclamation #2 + MTA-STS estate enforce (2026-08-14 afternoon, infra `3a7ebc0`/`8b49e64`/`d6e3e5a`):** NL YT unresolved 106→53 / GR 5→3 via 55 live-verified operator-approved closes (Thanos firing set + live LibreNMS + off-net probes; 15 = dup-closes onto rolling trackers). ⭐ **Pile-up root cause: `scripts/alert-yt-autoclose.py` ARMED since 07-18 but every close MUTATIONS=OFF-shadow-suppressed — hourly correct close-lists discarded for a month; exemption-lane-vs-unshadow decision = IFRNLLEI01PRD-2359.** Same session (operator-ordered, TLSRPT gate waived): **ALL 15 mail domains → MTA-STS `enforce`** (9 estate 11:47Z id `2026081401` + ellizg.com follow-up 12:23Z id `2026081402`; -2244 Done; TLSRPT rua-path residual = IFRNLLEI01PRD-2358; procedure + state in infra `native/smtp/CLAUDE.md` § MTA-STS). Cloudflare Workers+DNS token now standing access (`CLOUDFLARE_WORKERS_API_TOKEN` in .env + memory `reference_cloudflare_api`). Side-finds: nl-nas02 actually DOWN since 08-06 (-2282 reframed retire-vs-fix), GR intake dropping firing criticals (dedup json 0-bytes since 08-08), velero genuinely sick (BackupStale critical + KubeJobFailed ×15 + issue-less velero OOMKilled), -1833 closable post-autoheal. Memories: [`memory/yt_backlog_close_list_20260814.md`](memory/yt_backlog_close_list_20260814.md), [`memory/mta_sts_enforce_flip_20260814.md`](memory/mta_sts_enforce_flip_20260814.md).
- **5th NL↔GR partition + same-day 3-layer auto-recovery (2026-08-14, gateway MRs !211/!212, infra !462/!463):** BOTH legs wedged with the frozen-decaps signature (encaps rising, decaps frozen, IKE READY **with** child SA — a 3rd variant): Freedom silently dead since 08-12 ~23:00 UTC — `IntersiteBGPLegDown` created IFRNLLEI01PRD-2338 **one minute later** and it sat Open, unconsumed, 34 h — Budget followed 08-14 06:56 → ~2h20m partition (corosync 4/6 NL / 2/6 GR-inquorate, LibreNMS storm both sites). Manual restore 09:15 UTC via the runbook re-key, all netmiko (operator directive: **netmiko/paramiko ONLY for Cisco, exec/config included**); 52/52 mesh + pvecm 6/6 + Galera 3/Synced verified; 9 YT symptom issues closed. **Smoking gun found while planning the fix: `vti-freedom-recovery.sh` detected the wedge minutes after the leg died and was MUTATIONS=OFF-shadow-suppressed the entire 34 h — the partition was preventable with existing tooling.** Same day the 3-layer auto-heal (runbook entry above) was built, live-drilled on both legs and ARMED. Transients: gr-pve02 false SNMP-down on GR NMS during the partition (poller starvation); Freedom WAN ISP-side BNG-reachability blips flipped default egress to Budget briefly (11:55 + ~12:44 local — distinct from the wedge class); `freedom-qos-toggle.sh` shadow-chatters would-actuate every ~2 min (trigger suspect, open). Memories: [`memory/intersite_partition_both_legs_20260814.md`](memory/intersite_partition_both_legs_20260814.md), [`memory/intersite_autoheal_build_20260814.md`](memory/intersite_autoheal_build_20260814.md).
- **nl-pve03 NVMe outage + estate-wide follow-through (2026-08-06, gateway MR !204, infra MR !454):** Kingston FURY controller hang 02:27Z → the agent's own sysfs PCIe-remove on the live host caused an 11h outage; what kept it down was PCI renumbering landing `driver_override=vfio-pci` on the rpool survivor + a bond NIC, plus the boot ESP existing only on the dead disk. Recovered 14:04Z via AMT KVM (MeshCommander headless), 29 guests, zero data loss. Post-mortem: [`docs/postmortem-nl-pve03-nvme-outage-2026-08-06.md`](docs/postmortem-nl-pve03-nvme-outage-2026-08-06.md) (MR !204); Kingston RMA `00706345` **approved 2026-08-10, failed drive shipped 2026-08-14** (PostNL `LW038611611NL`; replacement ≤10 wd after Dublin receipt, chase by ~2026-09-04); standing host rules + replacement checklist in [`.claude/rules/infrastructure.md`](.claude/rules/infrastructure.md) § Known Host nl-pve03 (IFRNLLEI01PRD-2273 stays open until the drive lands). Same-day follow-through: full 29-guest health check + every WARN remediated (FISHA arbiter, pacemaker-auto-online, keepalived check_haproxy ×4 nodes both sites, gpu01 snmpd D-state on dead `hard` NFS → fstab `soft`); **SeaweedFS capacity P0 cleared 95%→38% via operator-approved Velero wipe** — which taught the two-part rule now in the k8s repo (infra MR !454): **wiping a Velero bucket = object storage AND the `BackupRepository` CRs**. Verification backup `verify2-20260806` in flight at session end; **kernel-alignment reboots parked until a clean restore point exists**. Memories: [[pve03_nvme_hang_pcie_remove_crash_20260806]], [[pve03_post_recovery_health_check_20260806]], [[warn_remediation_execution_20260806]], [[pve_estate_boot_nic_firmware_audit_20260806]]; rules [[feedback_pci_renumbering_breaks_address_pinned_configs]], [[feedback_single_esp_on_one_mirror_member]].
- **Backlog reclamation + ghcr credential exposure (2026-07-31, infra MR !444):** NL open issues **389 → 26**, GR **7 → 2**, every close live-verified first (mass-closes were the "Correlated alert burst" storm + recovered flaps; the rest individually checked). **Security:** a GitLab **root admin PAT** had been anonymously readable in the public `ghcr.io/cubeos-app/api-builder` image history since 2026-01-29 — package deleted, rebuilt credential-free and **private**, least-privilege token + masked CI variable created, all other public packages anon-scanned clean; **the credential itself is deliberately NOT rotated (operator decision: GitLab is not internet-reachable)** — do not re-propose it. Root cause is now a standing rule in [`.claude/rules/ci-cd.md`](.claude/rules/ci-cd.md) § Build-time secrets: **`--build-arg` leaks into image history on the legacy builder AND on BuildKit v0.8**, so `DOCKER_BUILDKIT=1` is not a mitigation — remove the credential from the build instead (MR !444) and gate pushes with `docker history | grep`. **Fixes:** all 14 velero kopia repos green (blob repair), Wed vzdump `bwlimit` on nl-pve01, registry cleanup policies (offline GC is incompatible with a metadata-DB registry — see ci-cd.md), `kvm` runner tag, estate-wide nginx map (69 hosts; nothing internet-exposed vulnerable — 16/17 findings are the abandoned `yacht` image). **Two severity self-corrections worth remembering:** exposure was over-claimed twice by reasoning from `0.0.0.0` and a public-looking hostname instead of probing from off-net — the second cost 14 min of needless downtime on a healthy slurpit ([[feedback_verify_internet_reachability_before_severity]]). Memories: [[leaked_root_pat_is_gateway_own_credential_20260731]], [[nginx_estate_sweep_yacht_finding_20260731]], [[velero_kopia_recorruption_seaweedfs_regrowth_20260731]], [[slurpit_upgrade_attempt_rolled_back_20260731]], [[workspace_epic_blocked_by_shadow_gated_finalizer_20260731]], [[gr_queue_triage_20260731]], [[health_status_check_20260731]].
- **Multi-track infra + agentic-hardening batch (2026-07-09→10, gateway MRs !183-!191, infra !393/!394):** One long autonomous session across many fronts. **Infra:** ALL GR guest backups had been silently failing since 2026-04-30 (grpbs02 `ds01` `.chunks` empty → every backup ENOENT; = the weekly Thu GR-k8s-control-plane kill) — chunk store rebuilt, 44-guest reseed clean, control plane never blinked (fleecing-off-rpool+bwlimit proven), YT `IFRGRSKG01PRD-287`; ZFS-DIO made boot-durable on all 6 PVE (initramfs bake, [[feedback_modprobe_conf_needs_initramfs_bake]]); NL ctrl01 Sat-vzdump bwlimit (`IFRNLLEI01PRD-1741`); linkwarden 3-day 502 = abandoned manual upgrade, not Renovate (`-1742`). **RAG (`-1739`):** FAISS-read gap (incident/wiki arms were brute-force) + ollama keep_alive + synth right-size; **key correction: the gateway RAG uses FAISS, NOT milvus** — milvus-on-GPU was stealing VRAM without serving the gateway, stopped 2026-07-10 (churn reduced, gemma3:12b still needs more VRAM freed — see [`memory/gpu01_vram_map_milvus_unused_20260710.md`](memory/gpu01_vram_map_milvus_unused_20260710.md)). **Chaos plane:** drilling fine but follow-through broken — redteam CWD false-0/20, embedding backfill dead since 04-17, findings never verified, zero alerting → all fixed (per-scenario SLO, findings harvester, 5 alerts, holistic §45), `-1744`. **Guard model:** discovered `unified-guard.sh` is unwired ([[unified_guard_not_wired_finding_20260710]]); RETIRED it (a command-string blocklist is the wrong layer for autonomy) and **built the plan-adherence execution gate** (`-1746`, reasoning-vs-action: gates a dispatched session's mutating commands against its committed infragraph blast-radius PRE-execution, ARMED enforce) — see [`.claude/rules/platform-features.md`](.claude/rules/platform-features.md) § Guard model. Memories: [[gr_pbs_backup_outage_and_vzdump_kill_fixes_20260709]], [[rag_qa_holistic_fix_batch_20260710]], [[chaos_status_sweep_20260710]], [[linkwarden_502_manual_upgrade_20260710]], [[plan_adherence_gate_20260710]].
- **Renovate autonomy lane — go-live + full hardening + epic follow-ups (2026-07-07, gateway MRs !160-!168, infra !377/!382, gr-cilium !76):** Cleared the entire held Renovate backlog + everything Renovate opened (~20 MRs) one-by-one with per-MR discipline; then an 8-agent adversarial verification of the lane found + closed the real gaps. **Auto-merge now actually works** (routine docker bumps merge+deploy hands-off via a deterministic structural review + a `*/15` reconciler; k8s/helm/tf/openbao/stateful/**Dockerfile** all POLL for review — never auto-apply blind). Fixes: deterministic routine review, cap-excludes-synthetic-rows, `${9:-{}}` gates_json JSON-corruption, postmerge host/service-from-path + **not-found→escalate-not-revert** (3-way, no spurious auto-reverts), docker-tag MAJOR re-detection, dedup rate-cap retry, Dockerfile-never-auto gate, 2 Prometheus alerts, reconciler orchestrator-registered. Plus the epic follow-ups: **GR cilium → 1.19.5** (parity, canary-verified, health 13/13 + BGP), **nl-gpu01 driver 570→580 / CUDA 13.0 → milvus restored to GPU**, watchtower removed from milvus DB stacks (root cause of the dormancy). All e2e-verified + tested (6 renovate suites green). Runbook: [`docs/runbooks/renovate-mr-autonomy.md`](docs/runbooks/renovate-mr-autonomy.md) § Hardening 2026-07-07. Memories: [`memory/renovate_autonomous_mission_20260707.md`](memory/renovate_autonomous_mission_20260707.md), [`memory/renovate_epic_followups_batch_20260707.md`](memory/renovate_epic_followups_batch_20260707.md). Reusable: [`memory/feedback_inconclusive_never_auto_reverts.md`](memory/feedback_inconclusive_never_auto_reverts.md), [`memory/feedback_no_synthetic_rows_in_live_counters.md`](memory/feedback_no_synthetic_rows_in_live_counters.md).

- **Eval-stack + platform hygiene batch (2026-07-03, MRs gateway !134-!138 + !123/!126 rescued, !124/!125 closed; infra !355-!357):** holistic 88→98/0-fail (gpu01 daily-reboot cron REMOVED — it hit the 05:00Z health run; outcome-based Cronicle-aware checks; OpenClaw probes retired to SKIP); QA 819/0 (fixture date time-bomb, self-protected-restart test, Cronicle-stdin-JSON doc wipe); **local LLM judge was dead AGAIN 06-27→07-03** (MR !108's type=='text' filter vs untyped local envelopes — 2nd death caused by the fix for the 1st) — resurrected + strict approve-calibration rubric, judge-fooled 13→~3 genuine; **semantic judge-death detection now dual-layer** (`judge_scored_fraction` from judge-independent tables + crosscheck samples sessions-not-judgments); 14 doc-only alerts deployed in-cluster (ChatOpsCostBudgetHigh deliberately withheld — API-equivalent cost semantics need operator call); OpenObserve 846-restart watchtower-downgrade loop fixed + pinned v0.91.1; 80-file live-tree drift committed (stray repo-root gateway.db gitignored). Memories: [`memory/holistic_fix_batch_20260703.md`](memory/holistic_fix_batch_20260703.md), [`memory/qa_judge_fooled_fix_20260703.md`](memory/qa_judge_fooled_fix_20260703.md), [`memory/judge_calibration_local_judge_dead_again_20260703.md`](memory/judge_calibration_local_judge_dead_again_20260703.md), [`memory/judge_death_detection_gap_closed_20260703.md`](memory/judge_death_detection_gap_closed_20260703.md), [`memory/live_tree_drift_sweep_20260703.md`](memory/live_tree_drift_sweep_20260703.md). Open: ChatOpsCostBudgetHigh threshold; weekly Cronicle job for `scripts/commit-regen-artifacts.sh`.

- **Agentic benchmark (12/14 A) + orchestrator control-plane (2026-06-26, epics IFRNLLEI01PRD-1421/-1422/-1423, all merged to main):** scored the platform against Anthropic *Building Effective AI Agents* (-1422) + OpenAI *A Practical Guide to Building Agents* (-1423) as separate source-pure adversarially-verified scorecards, then improved to **12/14 dimensions at A** (the 2 at B — unified-guard off the dispatched path + HITL failure-threshold as passive Matrix warning — are deliberate operator decisions, not gaps). Key fixes: model-router counted markdown PIPEs not incident rows → all 818 sessions pinned to Opus (now routes simple low-risk → Sonnet with a never-downgrade-risky floor); OTLP export dead since ~March (stale `OTLP_AUTH`); `MemoryMax=12G` cap on dispatched `claude -p`; session tripwire (token/cost/tool-call kill, `~/gateway.tripwire_off`); central `scripts/lib/models.py` + drift guards; 11 sub-agents made discoverable via `~/.claude/agents` symlink. Plus the **orchestrator control-plane** (-1421): 3 self-monitoring bricks (Registry / Interaction Graph / Orchestration Benchmark) + 5 in-cluster PrometheusRules + 2 new self-hosted services (Healthchecks.io + Langfuse v2 on nlopenobserve01) — closes the dark-component failure class. Scorecards: [`docs/scorecard-anthropic-2026-06-26.md`](docs/scorecard-anthropic-2026-06-26.md), [`docs/scorecard-openai-2026-06-26.md`](docs/scorecard-openai-2026-06-26.md), [`docs/benchmark-synthesis-2026-06-26.md`](docs/benchmark-synthesis-2026-06-26.md). Memory: [`memory/orchestrator_control_plane_20260626.md`](memory/orchestrator_control_plane_20260626.md).
- **Autonomy-forward gate — human as circuit-breaker, LIVE + ENABLED 2026-06-16 (epic IFRNLLEI01PRD-1102, merged to main `778406b`):** the operator stopped voting on the Matrix approval polls (notifications off), so the old binary `auto=risk==low` gate stranded ~56% of sessions on the 30-min pause and paged no one. Replaced with a 3-band model in `classify-session-risk.py`: **AUTO** (low or reversible+prediction-eligible MIXED → `[AUTO-RESOLVE]`), **AUTO_NOTICE** (reversible MIXED on a P0 host / wide blast → auto + parallel SMS), **POLL_PAUSE** (HIGH / irreversible / deviation / no-prediction / jailbreak / P0-reboot → poll+pause+SMS). New session→SMS path via `/alert-session` on the Twilio bridge (dedup by issue_id, HIGH-only per operator). **[RETIRED 2026-07-09 — session SMS unwatched+paid → replaced by daily Matrix HITL digest `scripts/session-digest.py`; `~/gateway.autonomy_session_sms` removed, `handle_session_payload` now returns `suppressed`. Banding + platform tier-1 dead-man SMS unchanged. [`memory/session_sms_retired_matrix_digest_20260709.md`](memory/session_sms_retired_matrix_digest_20260709.md)]** Irreversible re-tagging closed real gaps (`terraform destroy` was MIXED; `mkfs`/`zpool destroy`/`dropdb` unmatched). **Enable/kill via sentinel files: `touch ~/gateway.autonomy_forward ~/gateway.autonomy_session_sms` = ON; `rm` = instant byte-identical-legacy revert** (no n8n edit). Safety floor (never auto): deviation/irreversible/no-prediction/partial/jailbreak — keyed on the fail-CLOSED -1044 prediction gate; band-aware weekly invariant in `audit-risk-decisions.sh`. Decisions vs plan: POLL_PROCEED folded into AUTO_NOTICE (no bridge surgery); verdict-gating is the pre-execution prediction gate (the match/partial/deviation verdict is post-execution). Runbook: [`docs/runbooks/risk-based-auto-approval.md`](docs/runbooks/risk-based-auto-approval.md) § Autonomy-forward gate. Full memory: [`memory/autonomy_forward_gate_20260616.md`](memory/autonomy_forward_gate_20260616.md). **Live-verified 2026-06-17 — first REAL Tier-2 auto-resolve: IFRNLLEI01PRD-1117 (nlnc01 Service up/down, critical) ran a genuine 26-turn claude-opus-4-8 session (conf 0.86), confirmed the host recovered, classified band=AUTO → reconcile → YT Done + `session_log.resolution_type=auto_resolved`. Gate discriminates correctly: 4 AUTO (low/read-only) vs 8 POLL_PAUSE since enablement — the high-risk seaweedfs OOM (-1113) correctly went POLL_PAUSE, not auto. Trace any auto-resolve via `grep <id> ~/logs/claude-gateway/pipeline-debug.log`.**
- **Self-verifying reliability + governance batch (2026-06-21, roadmap Stage-0/1, MRs !40/!41/!42/!43/!44/!45 + IaC !336):** 5 items shipped to main + live, E2E-verified (per-item QA 7/8/5/6/8 + adversarial synthesis GREEN). **I1 -1152** control-plane dead-man's-switch (`gateway-watchdog.sh` heartbeat → SMS); **I2 -1153** false-auto-resolve/repeat-incident governance metrics (migration 018, `write-governance-metrics.py` cron `*/17`) + **auto-demote DEFAULT-ON** (autonomy-forward: a ≥3×/30d genuine repeat-offender (host,rule) is auto-demoted to `analysis_only` → `tier1_suppression` **escalates** it instead of auto-resolving; excludes intentional known-transients; RAG-excluded via `project='chatops-governance'`; reversible 30d expiry; **no manual review** — circuit-breaker = metric+audit+expiry); **I3 -1154** synthetic-incident canary (isolated-DB spine probe); **I6 -1158** bi-temporal infragraph invalidation (migration 019, decay reporting-only, `invalidate_edge` ready but auto-trigger unwired); **I8 -1159** GEPA reflective prompt generator via `claude -p` (DORMANT, `PROMPT_GEPA_ENABLED=0`, Welch-t-test stays the gate). Also closed -1145 (infragraph action-lane learn + annotate-not-gate). Cancelled -1156/-1157; deferred -1155. Schema-drift found: `incident_knowledge.valid_until` is live-only (absent from schema.sql/migrations). Memories: [`memory/watchdog_deadman_20260621.md`](memory/watchdog_deadman_20260621.md), [`memory/governance_metrics_20260621.md`](memory/governance_metrics_20260621.md), [`memory/synthetic_canary_20260621.md`](memory/synthetic_canary_20260621.md), [`memory/bitemporal_invalidation_20260621.md`](memory/bitemporal_invalidation_20260621.md), [`memory/gepa_prompt_evolution_20260621.md`](memory/gepa_prompt_evolution_20260621.md).
- **Freedom XGS-PON PPPoE outage on nl-fw01 outside_freedom RESOLVED 2026-05-13 (IFRNLLEI01PRD-891, closed 2026-05-15):** 4d 15h 14m outage on line F0381391 — down 2026-05-08 07:46:36 UTC (ASA Track 1, BRAS `198.51.100.X` stopped returning PADO), restored between 2026-05-12 23:59 and 2026-05-13 00:00 UTC after Freedom NOC field-engineer dispatch + OLT/BRAS service-binding refresh (mechanism: ONT physical swap forces de-registration of old PON serial + new registration + BRAS re-bind in same window). First post-recovery NAT through assigned WAN IP `203.0.113.X` observed at 2026-05-13 00:56:09 UTC. Day-count NAT-events syslog evidence on nl-fw01: May-08=968k → May-09..12=0 each → May-13=863k → May-14=957k. Failover via `outside_budget → nlrtr01 Dialer1` carried all 6 affected VTIs (Tunnel4-9) the whole 4.6 days — **zero user-visible downtime**. Customer-side surface was fully exhausted (3 days of `shut/no shut` cycles + PoE cycles produced visible PADI egress but zero PADO return); phone NOC `088-0115666` (English supported) on 2026-05-11 was the action that triggered dispatch. Full post-mortem + reusable PPPoE diagnostic playbook + customer detail (postcode 2315 HP nr 17, customer F0381391, ASA `fake@freedom.nl` username is decorative, BRAS at `198.51.100.X`): [`memory/freedom_pppoe_outage_resolved_20260513.md`](memory/freedom_pppoe_outage_resolved_20260513.md).
- **GitHub mirror sync chain hardening (2026-05-01, IFRNLLEI01PRD-835):** 3 distinct `sync_to_github` failures (.gitignore-vs-cp-a, refs/pipelines transient, github push CAS race) closed across `5e7ff45`/`63be431`/`54ea10d`. Full memory: [`memory/github_sync_chain_hardening_20260501.md`](memory/github_sync_chain_hardening_20260501.md).
- **Agentic-platform sweep (2026-04-24):** 13-item triage; 10 closed in 4 commits (chaos dedup, cmd_recover lock, RAG cohort split, ChaosCollisionError exception state). Full memory: [`memory/agentic_batch_20260424.md`](memory/agentic_batch_20260424.md).
- **parsePoll hardening (IFRNLLEI01PRD-736, 2026-04-25):** 8 distinct parser bugs in Runner + Bridge closed across 5 commits. Live SHA-matched. Full memory: [`memory/parsepoll_fix_20260425.md`](memory/parsepoll_fix_20260425.md).
- **Agentic-platform sweep (2026-04-25):** Diagnosed + fixed `chaos-test.py` double-flock regression. 5/5 PASS. Full memory: [`memory/agentic_batch_20260425.md`](memory/agentic_batch_20260425.md).
- **Security scanner cron PATH fix + per-host email path (2026-05-04):** nuclei + testssl silently failing in cron for ~5 weeks because cron PATH excludes `/usr/local/bin`. Fixed via `export PATH` in `weekly-scan.sh` + testssl false-positive grep + per-host email path. Full memory: [`memory/scanner_nuclei_silently_broken_20260504.md`](memory/scanner_nuclei_silently_broken_20260504.md).
- **SeaweedFS cross-site replication recovery (2026-05-05):** Two independent stale-checkpoint failures (cross-site `filer.sync` 145d-stuck + GR intra-cluster `meta_aggregator`). Fixed via Atlantis MR `infrastructure/nl/production!290` + gRPC `KvPut` recovery script. Full runbook: [`docs/runbooks/seaweedfs-cross-site-replication.md`](docs/runbooks/seaweedfs-cross-site-replication.md). Full memory: [`memory/seaweedfs_filer_sync_stale_checkpoint_20260505.md`](memory/seaweedfs_filer_sync_stale_checkpoint_20260505.md).
- **Status-page chaos red-link bug + AS64512 noise removal (2026-05-06, kyriakos `f458a8e` + claude-gateway `ca4c2e9` MR !6):** 3 bugs (chaos.js `tunnelLabel` reversal for non-listed pairs, missing frontend chaosable allowlist, backend silent-pass on empty `tunnel_infos`) all required for the 600s no-op chaos run. Removed entire BGP info bar (RIPE `first_seen=2024-08-26` was a previous holder of AS64512). Full memory: [`memory/status_page_chaos_red_link_fix_20260506.md`](memory/status_page_chaos_red_link_fix_20260506.md).
- **kyriakos agentic-chatops Token Usage widget — 13-finding hardening (2026-05-11, `kyriakos:729f0bb`, pipeline 28833):** Same-day audit + fix of the live Token Usage widget on `kyriakos.papadopoulos.tech/projects/agentic-chatops/`. 13 bugs across 4 severities closed in a single commit: `esc()` helper (Bug 1, `<synthetic>` HTML-tag injection → blank model name), proportional `flex-grow` (Bug 2, log10 misrepresented 99.6% Opus segment as ~30%), per-segment tier preference (Bug 3, dual-tier model colouring), upstream synthetic-leak filtered at consumer (Bug 4), `<script type="application/json">` + `JSON.parse` (Bug 5), UTC-explicit date math (Bug 6), `num()` defensive coercion (Bug 7), aria-label dual-window disclosure (Bug 8), today (so far) annotation (Bug 9), zero-segment filter (Bug 10), consistent `.toFixed(1)` (Bug 11), unknown-tier grey fallback (Bug 12), data file refresh (Bug 13). Regression spec: `visual-audit/tests/token-usage-postfix-verify.spec.js` 17/17 PASS on live. Full memory: [`memory/agentic_stats_widget_audit_20260511.md`](memory/agentic_stats_widget_audit_20260511.md).
- **nl-gpu01 qcow2 io-error freeze RCA + monitoring wired (2026-05-12, k8s MRs !294 + !295):** VM VMID_REDACTED on nl-pve03 "frozen many times daily" was QEMU `paused (io-error)` with `I/O status: nospace` on scsi0, not a guest kernel freeze. Root cause: qcow2 created without `discard=on`, so guest deletes never propagated → 99.56 % cluster allocation despite guest `/` at 77 %. Fix in-flight: `qm set --scsiN ...,discard=on` on both disks + `qm set --memory 32768` (operator override) + restart + `docker prune` + `fstrim -av` (1.27 TiB UNMAP, allocation 99.56 % → 71.55 %, ZFS −210 GiB). `fstrim.timer` already weekly-enabled. Monitoring landed via two MRs on the K8s repo: `/etc/cron.weekly/gpu01-health-metrics` on nl-gpu01 + PrometheusRules `Gpu01DockerDanglingImagesHigh` (> 5 GiB / 6h) + `Gpu01FstrimTimerInactive` (== 0 / 1h). MR !294 first wired the alert on `gpu01_docker_reclaimable_bytes` but that metric is misleading (counts images held by stopped containers, drifts upward forever); MR !295 swapped to dangling-images. Full memory: [`memory/gpu01_freeze_qcow2_io_error_20260512.md`](memory/gpu01_freeze_qcow2_io_error_20260512.md), feedback: [`memory/feedback_docker_reclaimable_misleading.md`](memory/feedback_docker_reclaimable_misleading.md), [`memory/feedback_gpu01_target_ram_32g.md`](memory/feedback_gpu01_target_ram_32g.md).
- **Auto-resolve metric semantic switch + Phase 1b blast-radius fold + Prom-receiver TDZ fix (2026-05-12, commits `1434bb5` + `69bd6f7`):** Page-rendered Auto-resolve was reading 7% on the live agentic-chatops widget — real, but using event-based counting that inflated denominators 4.7× during repeat-alert storms. Flipped `scripts/agentic-stats.py` outcomes block to **per-incident, best-outcome semantics** (Google SRE Book Ch 6 / Datadog SRE Maturity 2024 / PagerDuty MTTR all measure per-incident); same 7d data now reads **28.57%**. Event-based counts preserved as `outcomes.auto_resolve.event_based.*` for transparency. Added `check_phase1b_blast_radius` in `scripts/lib/tier1_suppression.py` (direct analog of Alertmanager `inhibit_rules` / PagerDuty parent-child) — operator declares an outage's blast radius by inserting a row in `openclaw_memory` keyed to a YT control issue; Phase 1b activates only while that issue is open. Two control issues created (closed by default, rules inert): `IFRNLLEI01PRD-894` (NL `nl*` × {Devices/Service/Port up-down + Device Down}) + `IFRGRSKG01PRD-241` (GR `gr*`/`gr2*` same family). Seeded 4 transient-pattern rows in `incident_knowledge` for the noisiest Prometheus alerts (KubeClientErrors, HighPodRestartRate, ContainerOOMKilled, TargetDown on `nl-claude01`). Pushed TDZ fix to live Prometheus receivers — `Parse Alerts` Code node referenced `now` (L191) before its `const` declaration (L262), throwing `ReferenceError: Cannot access 'now' before initialization` on every flap-escalation (5+ pre-fix errors in the prior 24h, e.g. NL execution 329133). Runbook protocol followed: snapshot → edit → `node --check` → `validate-n8n-code-nodes.sh` → PUT → re-fetch → deactivate/activate toggle → commit. 45 unit tests + 21 e2e tests + Playwright `auto-resolve-live-audit.spec.js` 4/4 PASS. Prior 7d was 76% only because it included 51 cutover-canary synthetics; real baseline ≈ 46%. Current 28% is dragged by 43 cascade-symptom escalations from the 2026-05-11 n8n OOM + GR mass-flap, expected to age out of the 7d window by 2026-05-19 → steady-state ~55-65%. Deferred: SQLite migration for n8n staticData dedup state, LibreNMS alertKey normalization. Full memory: [`memory/auto_resolve_regression_diagnosis_20260512.md`](memory/auto_resolve_regression_diagnosis_20260512.md), playbook: [`memory/blast_radius_control_issues_20260512.md`](memory/blast_radius_control_issues_20260512.md), principles: [`memory/feedback_count_incidents_not_events.md`](memory/feedback_count_incidents_not_events.md), [`memory/feedback_no_dedup_writes_to_pad_numerator.md`](memory/feedback_no_dedup_writes_to_pad_numerator.md).
- **BREACH gzip-off across the deployed estate (2026-05-07, IaC commit `a165e06`):** Closed testssl BREACH (CVE-2013-3587) on `198.51.100.X` + `198.51.100.X` from `grsec01` 2026-05-07 scan. 3 layers fixed: 7 gitlab `/srv/*-nginx-base` images rebuilt + pushed; 4 source-repo nginx.conf overrides committed via API (cubeos website, meshsat-site/docs, papadopoulos.tech); portfolio runtime via AWX ad-hoc on DMZ + IaC commit. Plus `mas.example.net` (the actually-exploitable surface) closed at-source via `nl-matrix01:/srv/matrix/nginx-conf/nginx.conf:20` + matrix nginx container restart (single-file bind-mount + `sed -i` = stale inode trap, lesson saved separately). Mattermost direct + element-web template + mulecube-dashboard via project 11 also covered. Full memory: [`memory/breach_gzip_off_estate_20260507.md`](memory/breach_gzip_off_estate_20260507.md).
- **nlk8s-ctrl01 chronic apiserver crash-loop RESOLVED (2026-05-15 12:59 UTC, IFRNLLEI01PRD-863):** kube-apiserver had restartCount=**1665** (~27 days of crashing every ~24 min). Cause: VM VMID_REDACTED on nlpve04 had `balloon: 4096` active while peers nlk8s-ctrl02/nlk8s-ctrl03 had `balloon: 0`. nlpve04 host pressure → pvestatd auto-ballooned nlk8s-ctrl01 down to 3.7 GiB → etcd WAL/DB page cache evicted → fsync 288 ms (vs peer 70-80 ms) → apiserver KV/Range timeouts → `/healthz` HTTP 500 → kubelet liveness-kill → restart cycle. **Trap that delayed the fix:** `qm set --balloon 0` only writes a `[PENDING]` config change — doesn't live-remove the QEMU balloon device. `pvestatd` keeps reading the still-active old value and re-inflates within ~10 min. Force-deflate via `qm monitor balloon <mb>` is temporary. **Real fix is `qm set --balloon 0` + `qm reboot`** — reboot applies `[PENDING]` and the balloon device is gone entirely (no `balloon:` line in `qm status --verbose` post-fix). Verified durability: balloon device absent, VM 7.7/6.3 GiB free, etcd commit 75 ms, `/readyz` 46 ms, restartCount stable since 12:59 UTC. **Architectural rule** (no balloon on any etcd / apiserver / DB VM) in [`memory/feedback_no_balloon_on_k8s_control_plane.md`](memory/feedback_no_balloon_on_k8s_control_plane.md). **The `[PENDING]` gotcha** in [`memory/feedback_pve_balloon_zero_needs_reboot.md`](memory/feedback_pve_balloon_zero_needs_reboot.md). Full memory: [`memory/apiserver_ctrl01_balloon_chronic_restart_fixed_20260515.md`](memory/apiserver_ctrl01_balloon_chronic_restart_fixed_20260515.md). Plus same-day mass-close pass on IFR projects (NL 59→1, GR 38→2 = **94 issues closed** total — most stale per the auto-resolve gap doc; details in YT comments) + HAProxy BREACH ACL fix on 3 VPS (chzrh01 + notrf01 + **txhou01** — original IFRNLLEI01PRD-845 description named only 2; fix uses var-capture pattern matching existing `is_matrix`/`is_analytics` idiom; details in [`memory/librenms_extender_fleet_deployment_20260515.md`](memory/librenms_extender_fleet_deployment_20260515.md) Edge HAProxy section).
- **Fleet-wide LibreNMS extender hardening + apcupsd + smart.config sweep + nlpve04 PBS backup unstuck (2026-05-15):** Multi-track session covering all 6 PVE hosts (NL pve01-04 + GR pve01-02). **nlpve04 went from bare → 7 extenders + apcupsd + smart.config + functional pbc-host-backup.sh** (the backup had been silently failing 5× weekly since 2026-05-10 onboarding because the PBS fingerprint trust file was never copied across). **proxmox-extender now uses cache pattern fleet-wide** (`*/5 cron writes /var/cache/proxmox`, snmpd just `cat`s it) — bypasses the `/etc/pve/priv/authkey.key` root-only requirement that made the snmpd-as-Debian-snmp proxmox extender fail with `exit=13 cfs-lock 'authkey' error` on every host. Rejected sudo-prefix (Fabian Grünbichler: "sudo is not the right way to implement unprivileged services") and `Debian-snmp → www-data` (PVE locks `/etc/pve/priv/` at mode 700 — group bits zero, group membership doesn't help). **apcupsd installed on nl-pve03+nlpve04** using nl-pve01's existing SNMP-over-Ethernet config (shared `10.0.181.X` APC Smart-UPS 1500, no USB needed). **smart.config sweep** caught stale `# smart.config for gr-pve01` clone-artifact headers on nl-pve01+nl-pve03 (now fixed; nl-pve01 also had a phantom `nvme1` line for an empty M.2 slot — real disks are FireCuda 530 at nvme0+nvme2) AND an essentially-empty smart.config on gr-pve02 despite 5 real disks (rewrote with 3 SCSI + 2 MegaRAID, all 5 now reporting). End-to-end verified via NL LibreNMS API (device_id 23/27/58/155) + GR LibreNMS API (device_id 34/35) — all 7 apps OK on every host. Side-finds: `alertmanager-twilio-bridge.service` runs as systemd `--user` inside nl-claude01 LXC at `oom_score_adj=200` — preferred OOM victim by design; left in place per operator decision but flagged. Full memory: [`memory/librenms_extender_fleet_deployment_20260515.md`](memory/librenms_extender_fleet_deployment_20260515.md). Architectural patterns: [`memory/feedback_pve_root_extender_cache_pattern.md`](memory/feedback_pve_root_extender_cache_pattern.md) + [`memory/feedback_systemd_user_slice_oom_score.md`](memory/feedback_systemd_user_slice_oom_score.md) + updated [`memory/feedback_no_sudo_install_on_pve_hosts.md`](memory/feedback_no_sudo_install_on_pve_hosts.md). Pending follow-ups: vzdump job for nlpve04 in `/etc/pve/jobs.cfg` (workload still unbacked), stale node-pinned VMIDs in nl-pve01/nl-pve03 backup jobs.
- **Status-page BGP-layer fix + Prometheus alert + visibility indicators (2026-05-17):** kyriakos status diagram now renders RIPE/bgp.tools-aligned 7+2 transits (was 4+0). Shipped commits: claude-gateway `b7c8ca5` (`vpn-mesh-stats.py` suffix-pair aggregation + min-obs=4 + per-upstream cap=7) + kyriakos `221231c` (mesh-graph.js single-constant row-y bump + EXPECTED_UPSTREAMS baseline + visibility gauge + auto-refresh BGP-key fix + cache-buster v=43). Plus Prometheus alert chain via IaC MR `infrastructure/nl/production!304` (4 alerts: AS64512UpstreamMissing/CountLow/VisibilityLow/BGPMetricsExporterStale → Matrix + Twilio). Runbook: [`docs/runbooks/upstream-bgp-failure.md`](docs/runbooks/upstream-bgp-failure.md). **2 reverts along the way:** per-upstream radial fan layout (`37462f3`) rejected as "complete disaster"; v=42 deploy with `fullData.bgp` (camelCase) instead of `fullData.public_bgp` (snake_case from raw API) flipped upstreams red on 30s auto-refresh. Full memory: [`memory/status_diagram_upstream_render_gaps_20260516.md`](memory/status_diagram_upstream_render_gaps_20260516.md). Lessons: [`memory/feedback_mesh_graph_cache_buster.md`](memory/feedback_mesh_graph_cache_buster.md), [`memory/feedback_preserve_row_layout_on_status_diagram.md`](memory/feedback_preserve_row_layout_on_status_diagram.md), [`memory/feedback_mesh_graph_updatedata_key_shape.md`](memory/feedback_mesh_graph_updatedata_key_shape.md).
- **nl-gpu01 / VM VMID_REDACTED io-error freeze TRUE root cause = ZFS DIO race (2026-05-14, IFRNLLEI01PRD-900, fix on all 6 PVE hosts):** The 2026-05-12 `discard=on` + memory bump (IFRNLLEI01PRD-892, now closed-as-hardening) was a useful hardening layer but never RC. Real cause: **OpenZFS 2.3+ Direct-I/O verify-write race** with QEMU `cache=none` (`cache.direct=true`) + `aio=io_uring` against qcow2 on a ZFS dataset. Stack: PVE **9.1.9** kernel `6.17.2-2-pve` with ZFS **2.4.1-pve1** — exactly the configuration affected by the known PVE 9 / ZFS 2.3+ regression. Mechanism: QEMU does zero-copy DMA from its userspace buffer; ZFS 2.3+ post-write verify-CRC catches any guest mutation of that page mid-flight (ollama loading models maximises this); on mismatch ZFS returns EIO; qcow2's cluster-allocation path **maps EIO → ENOSPC** internally; QEMU `werror=enospc,stop` pauses the VM with bogus "nospace" status (pool was 52 % full). 121 cumulative `ereport.fs.zfs.dio_verify_wr` events over 2 months mapped 1:1 to freeze incidents. Fix is 3-layer on every PVE host: `zfs set direct=disabled <pool>` + `/etc/modprobe.d/zfs-dio-disable.conf` (`options zfs zfs_dio_enabled=0`) + runtime `echo 0 > /sys/module/zfs/parameters/zfs_dio_enabled`. Per `man zfsprops`, `disabled` is "the default behavior for OpenZFS 2.2 and prior releases" — zero risk, just reverts to pre-2.3 behavior, ARC absorbs writes via buffered path. Applied to all 6 PVE hosts (NL: nl-pve01/nl-pve03/nlpve04 with `rpool`, nl-pve02 module-only; GR: gr-pve01 with `rpool`, gr-pve02 with `ssd-pool`). Drift-check: `scripts/check-zfs-dio-disabled.sh` (PASS = all hosts safe). VM resumed cleanly, 110k subsequent writes, zero new DIO errors. **Important nuance:** the `direct=disabled` workaround is community-validated (multiple PVE 9 `[SOLVED]` forum threads converge on it) but **NOT in any pve.proxmox.com wiki page** — no Proxmox staff endorsement found. Don't ever claim "it's the official Proxmox recommendation" — say "OpenZFS-documented + community-validated PVE 9 workaround for the ZFS 2.3+ DIO regression." `cache=none` itself IS Proxmox-official for ZFS-backed VMs. Follow-ups: (1) re-run drift-check + DIO error count on 2026-05-21 to verify fix held under sustained ollama load; (2) optionally file a Proxmox forum thread to push for official documentation; (3) re-evaluate at next PVE major upgrade. Detail in [`.claude/rules/infrastructure.md`](.claude/rules/infrastructure.md) §"Known Host: nl-gpu01". Full memory: [`memory/gpu01_zfs_dio_race_root_cause_20260514.md`](memory/gpu01_zfs_dio_race_root_cause_20260514.md) (includes Research validation section). Feedback for all future PVE-onboarding work: [`memory/feedback_zfs_dio_must_be_disabled_on_pve.md`](memory/feedback_zfs_dio_must_be_disabled_on_pve.md). Diagnostic recipe: [`memory/feedback_zfs_dio_diagnostic_recipe.md`](memory/feedback_zfs_dio_diagnostic_recipe.md).

---

## Where to add new content (so CLAUDE.md doesn't regrow)

When adding documentation in future sessions, follow this allocation pattern:

| Content shape | Destination |
|---|---|
| Incident summary (post-mortem, multi-bug fix, dated-event narrative) | `memory/<descriptive_name>_<YYYYMMDD>.md` (also indexed in `memory/MEMORY.md`) |
| Systemic platform feature (dated `## <Feature> (implemented YYYY-MM-DD)`) | `.claude/rules/platform-features.md` |
| Host-specific operational note | `.claude/rules/infrastructure.md` |
| Workflow-editing convention or n8n gotcha | `.claude/rules/workflows.md` |
| OpenClaw / OOB / fallback-mode operational detail | `.claude/rules/openclaw.md` |
| CI/CD pipeline rule | `.claude/rules/ci-cd.md` |
| Stable architectural orientation (URLs, hostnames, top-level diagrams) | CLAUDE.md (this file) |
| Runbook / runnable recovery procedure | `docs/runbooks/<topic>.md` |

**Rule of thumb:** if it has a date in its headline, it almost certainly does not belong in CLAUDE.md.
- **Code-node edits require validator (post-14h-outage gate):** Before any `curl -X PUT /api/v1/workflows/<id>` that modifies a Code node's `jsCode`, run `scripts/validate-n8n-code-nodes.sh --file <patched-workflow.json>` (or `<workflow-id>` to fetch live). Must return **VALIDATION PASSED** — checks `node --check`, `new Function()` parse, exactly 1 top-level `return` (dead code is a `[FAIL]`), and flags duplicate top-level `var` declarations. The Build Prompt node was cleaned 2026-04-19 (90 KB → 36 KB, 3 returns → 1); the validator prevents the 14h-outage-class regression. Full runbook: `docs/runbooks/n8n-code-node-safety.md`.
- **Risk-based auto-approval (IFRNLLEI01PRD-632, 2026-04-19):** Runner has `Classify Risk` SSH node between Build Plan and Build Prompt. Classifier (`scripts/classify-session-risk.py`) emits `{risk_level, auto_approve_recommended, signals, plan_hash}`; Build Prompt injects `## SESSION RISK:` section instructing Claude to end with `[AUTO-RESOLVE]` (low-risk) or `[POLL]` (mixed/high). Matrix Bridge parses `[AUTO-RESOLVE]` and posts as `m.notice` (no ping). Every classification writes to `session_risk_audit` table; `scripts/audit-risk-decisions.sh` + holistic-health enforce the invariant. **Updated 2026-06-16 (IFRNLLEI01PRD-1102, autonomy-forward gate):** the classifier now also emits a `band` (AUTO / AUTO_NOTICE / POLL_PAUSE) + `sms_required`, and the audit invariant is band-aware — it FAILS if any auto-approved row is outside AUTO/AUTO_NOTICE or carries a floor signal (`irreversible:*` / `critical:p0-reboot` / `deviation`), superseding the old "no `auto_approved=1` with `risk_level != 'low'`" check (which would false-FAIL on legitimate reversible-MIXED auto-resolves). Gated behind the `~/gateway.autonomy_forward` sentinel (flag-off = byte-identical legacy). Integration replay in `scripts/test-risk-integration.sh` (10/10 deterministic cases). HIGH-risk categories: `maintenance`, `security-incident`, `deployment`. Fail-closed: `RISK_FAIL_CLOSED=1` forces `high` on parse errors.
- **RAG circuit breakers (IFRNLLEI01PRD-631, 2026-04-19):** 4 named breakers guard the RAG external-call path — `rag_rerank_crossencoder`, `rag_embed_ollama`, `rag_synth_haiku`, `rag_synth_ollama`. SQLite-backed state (`circuit_breakers` table); Prometheus metrics via `scripts/write-circuit-breaker-metrics.sh` cron `*/5`. `CircuitBreakerOpen` alert fires if any stays OPEN ≥10 min. Inspect: `cd scripts && python3 -m lib.circuit_breaker list`. Reset: `python3 -m lib.circuit_breaker reset <name>`. Lib at `scripts/lib/circuit_breaker.py` — imperative `allow()/record_success()/record_failure()` API or decorator.
- **Schema versioning (IFRNLLEI01PRD-635, 2026-04-20):** All session/audit tables (now 30, was 9 at landing) carry `schema_version INTEGER DEFAULT 1` stamped by every writer. Canonical registry: `scripts/lib/schema_version.py` (`CURRENT_SCHEMA_VERSION` dict + `SCHEMA_VERSION_SUMMARIES`, mirroring OpenAI Agents SDK `run_state.py:131`). Python writers `from schema_version import current as schema_current`; bash writers hardcode `1` with registry pointer. Readers call `check_row(table, row.schema_version)` which raises `SchemaVersionError` on future versions. Migration `scripts/migrations/006_schema_versioning.sql` (idempotent via apply.py). Holistic-health §33 asserts no null `schema_version`. **Operational rule: when you change the JSON shape of any payload column in these tables, bump `CURRENT_SCHEMA_VERSION[table]` AND add a new line to `SCHEMA_VERSION_SUMMARIES[table]` describing the change.** Full reference: `memory/openai_sdk_adoption_batch.md`.
- **OpenAI SDK adoption batch (IFRNLLEI01PRD-635..643, 2026-04-20):** 9 structural upgrades from `openai/openai-agents-python` v0.14.2: schema versioning on 9 tables (-635), immutable per-turn snapshots (-636), 13 typed events in `event_log` (-637), per-turn lifecycle hooks (-638), 3-behavior rejection taxonomy allow/reject_content/deny (-639), `HandoffInputData` zlib+b64 envelope 0.43% ratio (-640), gemma3:12b transcript compaction (-641), `agent_as_tool.py` for ambiguous-risk band 0.4-0.6 (-642), `handoff_depth` + cycle detection ≥5 forces `[POLL]` / ≥10 hard-halts (-643). 4 new tables → 35. **Not adopted:** OutputGuardrail (deferred), per-tool `needs_approval`, auto-trace to OpenAI, strict Pydantic sub-agent output, always-on `nest_handoff_history`. Full reference: `memory/openai_sdk_adoption_batch.md` + `README.extensive.md` §22.
- **QA suite (2026-04-20, expanded 2026-04-23):** `scripts/qa/run-qa-suite.sh` — pytest-style bash harness, **73 suite files** (was 44 at 2026-04-23, 30+ at landing), **~3-5 min runtime** under full-suite load, JSON scorecard in `scripts/qa/reports/`. **Per-suite timeout guard** (IFRNLLEI01PRD-724, default `QA_PER_SUITE_TIMEOUT=120s`, override via env) prevents any single slow/hung suite from wedging the orchestrator; synthetic FAIL record emitted to scorecard on timeout. Covers: writer coverage (schema_version=1 across 11 writers + 5 n8n INSERT sites), 85 rejection patterns (53 deny + 32 reject_content), 13 event-class payload shapes, 8-parallel concurrent fuzz, local HTTP mock for offline compaction, 6 e2e scenarios, 16 prompt-patcher tests, 7 benchmarks, plus 9 umbrella-added tests (test-656/-660/-718/-724/-726/-727). **Last hardened run (2026-04-23): 411/0/2 = 99.52%**, up from 368/4/2. **Run after any change to the adoption-batch surfaces or the patcher.**
- **Teacher-agent reliability pass (2026-04-23):** 5 post-ship bugs closed after operator DM audit: Command Router double-wiring (`501ff47`); `cmd_grade` UPDATE wrong PK left `completed_at=NULL` (`3d9c0da`); Mastery/SM-2/Bloom advanced on low grader_confidence — `low_conf` branch now holds schedule steady (`3d9c0da`); teacher-runner webhook `responseMode: responseNode→onReceived` + removed terminal Respond node (`33d64c8`+`99dc9fc`); `cmd_chat` SQLite-lock crash — fixed with `timeout=30` + `PRAGMA busy_timeout=30000` + post-to-Matrix-before-audit-UPDATE + try/except around DB block (`feb2bae`). Also `cec2c0c`: `docs/gulli-book-overview.md` + 30 chapter extracts (embeddings 1189→1309). Reusable: `memory/feedback_sqlite_busy_timeout.md` + `.claude/rules/workflows.md`. Full detail: `memory/teacher_agent_dm_audit_20260423.md`.
- **Teacher-agent — all 5 tiers (IFRNLLEI01PRD-651, IFRNLLEI01PRD-652, IFRNLLEI01PRD-653, IFRNLLEI01PRD-654, IFRNLLEI01PRD-655, 2026-04-20):** Five-tier introspective learning module teaching the operator agentic-systems theory using the system's own docs. **-651 foundation** (migration 013 `learning_progress` + `learning_sessions`; `scripts/lib/sm2.py` SuperMemo-2 EF clamped `[1.3, 2.5]`; `config/curriculum.json` + `scripts/rebuild-curriculum.py` 53 topics × 4 curricula auto-derived from wiki/docs/memory; **13/13 PASS**). **-652 intelligence** (`scripts/lib/bloom.py` 7-level progression `recall→teaching_back`; `scripts/lib/quiz_generator.py` + `quiz_grader.py` Ollama gemma3:12b `format=json` with hallucination gate — `verbatim_text` substring of sources — and Invariant #4 confidence-clarifier <0.6; breaker-aware via `rag_synth_ollama`; **17/17 PASS**). **-653 interface** (migration 014 `teacher_operator_dm` `public_sharing DEFAULT 0` privacy-first; `scripts/lib/matrix_teacher.py`; `scripts/teacher-agent.py` orchestrator with 12 subcommands; `.claude/agents/teacher-agent.md` read-only tool allowlist Read/Grep/Glob/Bash/ToolSearch — Edit/Write/MultiEdit excluded; `workflows/claude-gateway-teacher-runner.json` `/teacher-command` webhook; multi-user classroom `#learning` design; **14/14 PASS**). **-654 loop** (3 crons `30 8 * * *` morning-nudge / `0 16 * * 0` class-digest / `*/5` metrics-exporter; 10 `learning_*` metrics; 3 alerts in `prometheus/alert-rules/teacher-agent.yml` — `TeacherAgentMetricsAbsent`/`TeacherAgentMorningNudgeStale`/`TeacherAgentClassDigestStale`; 12-panel `grafana/teacher-agent.json` dashboard; **9/9 PASS**). **-655 gate** (`scripts/audit-teacher-invariants.sh` enforces 6 invariants + privacy default; `scripts/teacher-calibration-baseline.py` 12-fixture / 5-band harness with `--offline` deterministic stub; `docs/runbooks/teacher-agent.md` ops runbook with `!learn is silent` debug ladder + 5-stage rollback; **9/9 PASS**). **Combined: 62/62 QA tests.** Real-data calibration (≥20 graded answers) deferred to natural accumulation. Plan: `docs/plans/teacher-agent-implementation-plan.md`. Full memory: `memory/teacher_agent_foundation.md` (named "all 5 tiers done").
- **CLI-session RAG capture (IFRNLLEI01PRD-646/-647/-648, 2026-04-20, ALL WIRED 2026-04-24):** Interactive `claude` CLI sessions flow into RAG tables (closes ~2,300 JSONL gap). Tier 1 `backfill-cli-transcripts.sh` (archive + parse-tool-calls + extract-cli-knowledge, tagged `issue_id='cli-<uuid>'`, watermark file). Tier 2 gemma3:12b over summary rows → structured incident_knowledge (`project='chatops-cli'`, breaker-aware). Tier 3 `parse-tool-calls.py::extract_issue_id_from_path` resolves JSONL → `cli-<uuid>`. `kb-semantic-search.py` `CLI_INCIDENT_WEIGHT=0.75` discounts cli rows at retrieval. QA `test-646-cli-session-rag-capture.sh` 12/12 PASS. **Cron installed** (`30 4 * * *`) + firing nightly — 04-24 run processed 50 files / 255 transcript chunks / 2831 tool-call rows / 25 knowledge extractions. Details: `memory/cli_session_rag_capture.md`.
- **Preference-iterating prompt patcher (IFRNLLEI01PRD-645, 2026-04-20, ALL WIRED):** Policy iteration at the prompt level. Low-scoring (surface, dimension) → `scripts/prompt-patch-trial.py --start` generates 3 candidate variants (concise/detailed/examples) + control. Build Prompt deterministically buckets via `hash(issue_id|trial_id) % (N+1)` (lines 393-417). `scripts/finalize-prompt-trials.py` (cron `17 3 * * *`) runs one-sided Welch t-test at ≥15 samples per arm; promotes if lift ≥0.05 & p<0.1, else aborts (or 14d timeout). Library: `scripts/lib/prompt_patch_trial.py` (race-safe SQLite). Enable: `PROMPT_TRIAL_ENABLED=1`. Prometheus via `scripts/write-trial-metrics.sh` (`*/10`). 2 new tables → 37 total. QA `test-645-prompt-trials.sh` 16/16 PASS. **5 active trials** since 2026-04-20 (investigation_quality, evidence_based, actionability, safety_compliance, completeness). Runbook: `docs/runbooks/prompt-patch-trials.md`. Details: `memory/preference_iterating_prompt_patcher.md`.
- **Agentic-platform sweep (2026-04-24):** 13-item triage; 10 closed in 4 commits — `ee65ec7` (chaos dedup + TRIAGE_JSON booleans + CHAOS_STATE_PATH), `b9c0661` (cmd_recover lock + 3-line end format + ETA + collision ABORT + MTBF cron + teacher hint renderer + SEARCH_BUDGET_S), `65b1e23` (RAG cohort split — `RAGLatencyP95High` now scoped to `category="real"`), `f4f2cd4` (`ChaosCollisionError` carries marker data on the exception, not re-read from except block — observed live 20:05/20:15 UTC as `scenario=unknown` posts). Infra MR `nl/production!270` `12cd22b6` via Atlantis. New metrics: `chaos_mtbf_seconds`/`chaos_last_failure_ago_seconds`/`chaos_success_streak`/`chaos_failure_count`/`chaos_availability_ratio` per chaos_type × rolling window (cron `*/5`). YT closed: `-695`, `-707`, `-703`. Three items deferred **as operator decisions, not code gaps**: real-data teacher calibration (needs ≥20 graded answers), OpenClaw v4.22 upgrade path selection, chaos-drill cadence policy. Reusable lesson: `memory/feedback_capture_state_on_exception_raise.md` (on `with lock:` raise the lock releases before `except` runs — capture state at raise time, never re-read in `except`). Full summary: `memory/agentic_batch_20260424.md`.
- **parsePoll hardening (IFRNLLEI01PRD-736, 2026-04-25):** Operator reported "the poll has bugs" in `#infra-nl-prod`. Three deep-sweep rounds across Runner `Prepare Result` and Bridge `Prepare Bridge Response` Code nodes closed **8 distinct parser bugs** in 5 commits (`9f680fc` → `eec74a9` → `fdb0971` → `c4eae6c`): early-`[POLL]` hijack via unanchored regex, trailing-prose absorption (17/70 historical polls), latent-fallback wrong field name, code-fence delimiter as phantom option, parser duplicated in Bridge (caught via repo-wide grep), single-blank-line break too aggressive (broke MESHSAT-664 loose-list), nested sub-bullets absorbed (MESHSAT-623), Markdown horizontal rule absorbed. **Verification corpus:** 250 historical executions / 34 PASS · 21 adversarial / 21 PASS · 25 real JSONL / 25 PASS · 70 round-trip / 53 PASS + 17 GOOD-FAIL · 8/8 bridge · 12/12+10/10 QA test-635/-727. Live state: Runner `qadF2WcaBsIR7SWG` versionId `1b9d71df`, Bridge `QGKnHGkw4casiWIU` versionId `21a9b2c3`, both `active=True`, live ↔ repo ↔ HEAD SHA-matched. Reusable lessons: `memory/feedback_anchor_llm_output_markers.md`, `memory/feedback_explicit_stop_conditions_when_sweeping_llm_lists.md`, `memory/feedback_grep_for_parser_duplication.md`, `memory/feedback_use_real_execution_data_for_regression.md`. Full incident: `memory/parsepoll_fix_20260425.md`.
- **Agentic-platform sweep (2026-04-25):** Diagnosed + fixed regression introduced by 04-24 `b9c0661`: `chaos-test.py:cmd_start` outer `fcntl.flock` + `b9c0661`'s inner `marker_lock()` re-flocked the same file on a separate fd → Linux per-fd EAGAIN against the same process → every `chaos-test.py start` ABORTed since 2026-04-23 evening (counter stuck at 122/107 = 6 lost intensives / 18 baseline experiments). Fixed in `8075721` (remove outer flock; inner `marker_lock()` preserves cross-drill protection via `chaos_marker.py:install_marker`) + `b0647df` (refresh stale references in docstring + `save_state` comment). Validated: `test-709-chaos-marker-lock` 5/5 PASS + scratch-isolated 7-step e2e (marker_lock + check + write + own-drill identity + cross-drill raise + exception-attrs + cross-process contention). Side-fix: `node_exporter` on `nl-openclaw01` `Exited(143)` since 2026-04-22 (deliberate `docker stop`) — restored. YT closed: `-728`, `-731`, `-732`. New: `-733` filed for `gemma3:12b` `num_gpu 49` Modelfile pinning. Reusable lesson: `memory/feedback_no_double_flock_same_path.md`. Full summary: `memory/agentic_batch_20260425.md`.
- **NVIDIA DLI cross-audit + P0+P1 implementation (IFRNLLEI01PRD-747..-751, 2026-04-29):** 19-transcript NVIDIA DLI Agentic-AI cross-audit graded the system **A (4.4/5.0)** on the 12-dim rubric — lowest of the 9 sources audited so far. Same-day implementation of all 7 P0+P1 items in 4 commits lifted to **A+ (4.83)**; 9-source aggregate **A+ (4.79)**. Commits: G1 `8aabf27` (long-horizon replay + 39-fixture jailbreak corpus incl. **8 Greek**), G2 `cac272a` (intermediate semantic rail DARK-FIRST + grammar-constrained decoding), G3 `4af78cf` (team-formation skill + ITS budget), G4 `2e3fb9f` (server-side session-replay endpoint `lJEGboDYLmx25kBo`). Operator gates closed cert-pass-2 `cac226a`: 5 cron entries, `Check Intermediate Rail` Code node in Runner (now 50 nodes), session-replay ACTIVE, Greek fixtures, YT 747-751 → Done via direct REST workaround. Schema: `event_log` v=1→4 (+4 event_types: `team_charter`, `its_budget_consumed`, `intermediate_rail_check`, `session_replay_invoked`); +1 versioned table `long_horizon_replay_results` (migration 015); 18→19 versioned, 26→27 workflows, 6→7 skills, 44→51 QA suite files (+57 tests, 411→468 PASS = 99.57%). Reusable lessons: `memory/feedback_youtrack_mcp_state_bug.md`, `memory/feedback_n8n_sandbox_no_child_process.md`, `memory/feedback_dataclass_importlib_quirk.md`. Single source-of-record: `docs/agentic-platform-state-2026-04-29.md`. Full memory: `memory/nvidia_dli_cross_audit_20260429.md`.
- **Security scanner cron PATH fix + per-host email path (2026-05-04):** Daily scanners `nlsec01`+`grsec01` had **nuclei + testssl silently failing in cron for ~5 weeks** because cron's default `PATH=/usr/bin:/bin` excludes `/usr/local/bin` where 13 of 21 scanner tools live; `2>/dev/null` was hiding bash exit-127. Daily emails reported "Nuclei findings: 0 / TLS issues: 0" interpretable as clean scans, actually meaning "tool didn't run." Fix: `export PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin` near top of `weekly-scan.sh`. Also closed: 5 instances of `0\n0` bash bug (`grep -c X || echo 0` doubled when grep returns 0 → `[: 0\n0: integer expression expected`); testssl false-positive substring grep (`grep -iE "VULNERABLE|NOT ok"` matched "not vulnerable" via case-insensitive substring → 44→2 real TLS findings); pre-create `nuclei.txt` with `:` defensive; capture nuclei stderr to `/opt/scans/reports/nuclei-stderr-<DATE>.log` (kept 14 days, future silent failures leave breadcrumbs). Added `118.91.186.185` (defra01agri01 = mail.meshsat.org) to both scanners' `targets.txt` — scanner coverage 5 → 6 public IPs. **New per-host filtered email path** via `format-report-host.py` (filters scan artifacts to single target IP, calls `format-report.py` with subject label `[scanner -> target_ip]`) + `EXTRA_HOST_EMAILS=("ip:email" ...)` array in `weekly-scan.sh`; currently `118.91.186.185 → defra01agri01-sec-rpt@mail.example.net`. **New env-overridable knobs** for ad-hoc runs: `TARGETS_FILE`, `SMTP_TO`, `EXTRA_HOST_EMAILS_DISABLE=1`. Real findings now visible (previously hidden): **self-signed cert on `203.0.113.X:443` OpenVPN-AS portal** (weak 4-byte serial + self-signed CA in chain + missing CRL/OCSP — priority: replace with Let's Encrypt) + 2× BREACH gzip on CH+NO VPS (low risk). Backups on both scanners: `/opt/scans/{weekly-scan.sh,format-report.py,targets.txt}.pre-fix-20260504`. Reusable lessons: `memory/feedback_cron_path_excludes_usr_local_bin.md`, `memory/feedback_substring_grep_negation_filter.md`. Full memory: `memory/scanner_nuclei_silently_broken_20260504.md`. Updated `.claude/rules/infrastructure.md` Security Scanners section.
- **SeaweedFS cross-site replication recovery (2026-05-05):** Two independent stale-checkpoint failures, same shape (persisted offset → GC'd change-log volume → permanent retry-loop), discovered + fixed end-to-end in one session. (1) **Cross-site `filer.sync`** stuck since **2025-12-11 19:07:15 UTC** (145 days) — GR change-log volumes 1+2 GC'd, NL pod in tight retry on `failed to get next log entry for HH-MM.<chunk-id>: volume <N> not found`. Fixed via Atlantis [MR `infrastructure/nl/production` !290](https://gitlab.example.net/infrastructure/nl/production/-/merge_requests/290) adding `filer_sync_{a,b}_from_ts_ms` override variables. **Upstream flag-naming gotcha**: `-a.fromTsMs` controls direction `b→a` (REMOTE→LOCAL, where filer A is the SINK), `-b.fromTsMs` controls `a→b` — verified vs `weed/command/filer_sync.go` @ tag 4.01 (the `a→b` goroutine consumes `syncOptions.bFromTsMs`). First commit `59ac870` had the assignment swapped; corrective `2f5897b` swapped it back. (2) **GR intra-cluster `meta_aggregator`** between `seaweedfs-filer-0` ↔ `seaweedfs-filer-1` had been broken since 2025-12-28 / 2026-03-24 (separate failure, same pattern). Cross-site writes were landing on whichever GR filer Cilium load-balanced to, but the OTHER filer never learned about them via `meta_aggregator`, causing intermittent 404s on cluster-mesh reads. Fixed via gRPC `KvPut` on each filer's `Meta`+peer-signature leveldb key — recovery tool committed at `scripts/seaweedfs/fix_meta_offset.py` (Python + grpcio + pinned filer.proto v4.01, no pod restart needed because `meta_aggregator` retries every ~2s and re-reads the offset on each retry). **Methodology slip caught by operator**: first diagnosis chased filer-sync logs and assumed the cross-site path was the only break. Per-pod state on GR (filer-0 had 4 entries in test bucket, filer-1 had 2) revealed the real second issue in one query. Reusable lesson: `memory/feedback_per_pod_state_for_multi_replica_diagnosis.md` ("when a multi-replica service shows intermittent 200/404, check each pod's state independently — the load-balanced svc hides per-pod divergence"). Verification 2026-05-05 15:53 UTC: zero `volume not found` errors on all 4 filer pods, both directions of cross-site sync progressing, **10/10 stress reads** via GR cluster-mesh service after PUT to NL. Companion docs MR !291 (merged) added a Known Issues entry to `infrastructure/nl/production/k8s/CLAUDE.md`. Full runbook: `docs/runbooks/seaweedfs-cross-site-replication.md`. Full memory: `memory/seaweedfs_filer_sync_stale_checkpoint_20260505.md`. Pre-existing data divergence on GR filer-1 (files written 2026-03-24 → 2026-05-05 only landed on filer-0) is an out-of-scope follow-up — listed in the runbook with three remediation options.
- **Status-page chaos red-link bug + AS64512 noise removal (2026-05-06, kyriakos `f458a8e` + claude-gateway `ca4c2e9` MR !6):** Operator's 19:13 UTC chaos run lasted 600s with zero red links painted. Root caused live from `/tmp/chaos-payload.json` to **3 distinct bugs that all had to fire together**: (1) `chaos.js` `tunnelLabel(src,tgt)` returned the *reversed* canonical form (`"NO-DMZ01 ↔ NL"`) for any pair not listed in `TUNNEL_WAN` — newly-onboarded sites (TX 2026-05-06, NO-DMZ01/02 2026-05-05) weren't in the legacy 4-site map, so the frontend submitted non-canonical labels that the backend's `CHAOS_TUNNELS` dict (NL-first canonical) silently missed; (2) frontend let users click inter-VPS swanctl tunnels (`CH ↔ NO-DMZ02` etc.) that aren't in `CHAOS_TUNNELS` by design; (3) `chaos-test.py:_cmd_start_locked` silently dropped tunnels not in `CHAOS_TUNNELS` (`tunnel_infos = []` after the loop) and progressed to a `status=active` response with empty `tunnels_killed`, while the kill subprocess later `sys.exit(1)`d in the background — net effect was a 600s no-op active run with a misleading green dashboard. Fixes: `SITE_ORDER = ['NL','GR','NO','CH','TX','NO-DMZ01','NO-DMZ02']` deterministic-priority fallback in `tunnelLabel()`; `CHAOSABLE_TUNNELS` allowlist (frontend mirror of `CHAOS_TUNNELS`) + submit-time gate + click-time gate via `window.__chaosableTunnels`; backend hard-error on empty `tunnel_infos` returning submitted+available keys for diagnostic clarity. Same session removed the **entire `mh-graph-bar`** ("AS64512 · prefix · visibility · since Aug 2024 · …") + the SVG `mg-origin` center label — the RIPE `first_seen=2024-08-26` was a previous holder of AS64512; operator's RIPE registration is `2025-12-05 13:13`, AS reassigned afterwards, so the date was misleading from day one. Cache versions: `chaos.js?v=7→9`, `mesh-graph.js?v=34→36`. Tests: 22/22 unit (`visual-audit/tests/chaos-tunnel-label.unit.js`) + 4/4 Playwright (`status-page-chaos-redness.spec.js`, route()-mocks `/api/chaos-*` so it never touches firewalls). **Cross-repo contract:** when adding a new mesh entity to `chaos-test.py:CHAOS_TUNNELS`, also add to `chaos.js:CHAOSABLE_TUNNELS` + `chaos.js:SITE_ORDER` — see `kyriakos:CLAUDE.md` "Chaos UI canonical-label contract" section. Reusable lessons: `memory/feedback_canonical_label_helpers_need_site_priority_fallback.md`, `memory/feedback_no_silent_pass_when_allowlist_lookup_misses.md`, `memory/feedback_remove_completely_means_the_whole_widget.md`, `memory/feedback_cf_edge_serves_stale_during_deploy_verification.md`. Full memory: `memory/status_page_chaos_red_link_fix_20260506.md`.
- **Infragraph epic build-out — model-based control LIVE (2026-06-09, IFRNLLEI01PRD-1029, MRs !20-!28 + IaC !327):** causal infra dependency graph (356 nodes / 414 edges; truth layers: pve-live 0.95 → librenms 0.90 → netbox+cables 0.85-0.90 → declared → incident-miner capped 0.75 below the 0.8 suppression cutoff) wired into triage end-to-end in ONE DAY: advisory dependency context in every triage + Tier-2 prompt; mandatory plan_hash-keyed prediction gate in the Runner (no approval poll without a committed machine prediction — remediation lane fails CLOSED, enforcement ×3: control flow + bypass QA against the live workflow export + always-on weekly audit); mechanical match/partial/deviation verdicts (LLM never adjudicates its own outcome); Phase B shadow recording + weekly gate scorecard; Phase C per-rule proposal/approval lane. 2026-05-11 backtest passed the falsifiable shuffled-control criterion (0.367 ≤ 0.5×) after 4 honest iteration rounds. First machine-proposed rule approved by operator: IFRNLLEI01PRD-1046 (nlpve04 cascade fold), production-exact verified. Frozen baseline for the closeout verdict: 0.4156 per-incident auto-resolve (30d). QA 65/65 across 8 suites. Full memory: [`memory/infragraph_epic_buildout_20260609.md`](memory/infragraph_epic_buildout_20260609.md).