lessons-learned · git:20260718.c182f80 · 2026-07-18 · sha256 7f605bbe3168a2ce
lessons-learned git:20260718.c182f80A
Immutable. This exact content never changes and is served at /api/v1/blob/7f605bbe3168a2ce.
No tool to install first
claude plugin marketplace add https://markdownregistry.com/mp/a/art_e43rvj7xqkrc7yyk/7f605bbe3168a2ced88ffa7503dcb0e22658f6fe7d3155be2f8ef5427db6b448.json
Then /plugin install lessons-learned@markdownregistry-e43rvj7xqkrc7yyk in Claude Code. It installs this exact version, the one graded A here, pinned to the commit it was graded at, and leaves it there.
--- name: lessons-learned description: >- Accumulated engineering lessons from real production incidents across the owner's projects. Load PROACTIVELY during development, code review, and planning of any feature that sends notifications, handles errors, or runs on a schedule - and whenever a new incident is analyzed, append its lesson here. Trigger on "проверь на повторяющиеся события", "lessons", or when implementing notification/alert/email logic. --- # Lessons learned (production incidents) Rules distilled from real incidents. When developing or reviewing, walk this list against the change. When a new incident is analyzed, append a dated entry: symptom, root cause, rule. ## Notifications and recurring events **Rule: every automated notification path MUST have all three of:** 1. a diff against the last NOTIFIED state (not the last observed state), 2. a per-entity cooldown window, 3. persistence of that state OUTSIDE process memory (DB/Redis) so restarts don't reset it. - **2026-07 RaceModel "16 identical weather emails/day".** A 15-minute scan diffed the forecast against the previous observation and mailed every follower per flip; an oscillating forecast (Spa weekend) mailed on every crossing of the threshold. Fix: diff vs last-notified snapshot + 6h per-race cooldown. Cache/prediction invalidation stays keyed to raw observations - only the outbound notification is debounced. - **2026-06 RaceModel "5 model-fallback admin alerts in one evening".** The 6-hour email cooldown lived in process memory; every deploy reset it, and fallbacks cluster exactly around deploys. Fix: Redis SET NX EX gate keyed by alert type; in-memory throttle is only the Redis-down fallback. - Checklist for any watcher/scanner: What happens if the source flaps A→B→A→B? What happens after a deploy mid-incident? Does suppressed notification still update internal state so a real trend fires later? ## Email deliverability - **2026-07 RaceModel/Brevo: 30% of monthly sends "blocked".** Tests and dev environments sent real emails through the production provider key to example.com addresses; the provider blocklisted them and sender reputation paid for it. Rule: test/dev environments MUST use a noop email adapter enforced in code (not just .env convention); assert in CI that the provider key is absent/ignored under pytest. - Bulk/notification email needs List-Unsubscribe + List-Unsubscribe-Post headers (Gmail/Yahoo bulk-sender requirements) and a per-category unsubscribe, with transactional (verification/reset) never suppressible. - Per-category sender addresses (notifications@, account@, news@) keep one category's reputation damage from sinking the others. ## Error handling **Rule: no failure may be invisible.** Every except/catch must log with context at the right level; every "nothing happened" path needs a log line an operator can find. A swallowed rejection in a UI component needs a visible error state, not a silent null render. - **2026-07 RaceModel: model fell back to stub silently for hours.** Boot-time R2 load failure had no retry and the hot-reload watcher (a) watched a legacy loader with a wrong manifest key and (b) was never even scheduled in lifespan. Users saw seed data with no banner. Rules: background watchers must be provably scheduled (test that lifespan starts them); self-heal loops must retry the REAL load path; degraded mode must be visible to users (badge) and operators (health endpoint field). - **2026-07 RaceModel auth middleware `except Exception: pass`** kept stale JWT claims when the DB blipped and let deleted/blocked users keep access. Rule: auth fallbacks must fail closed on "user not found" and log the degradation path. - Magic links in email must never mutate state on GET - corporate mail scanners prefetch links. GET renders a confirm page; the POST mutates. State transitions behind them need an atomic claim (UPDATE ... WHERE status='pending', check rowcount) or concurrent clicks double-fire mass emails. ## Process - Pyright "Import could not be resolved" across a whole repo usually means the session has no venv wired - verify with the project's own test suite before "fixing" imports. - A test that passes alone but fails in the suite: shared mutable state (same DB rows/keys across tests). Use unique per-test identifiers.