consumer-regression-suite · git:20260922.71ae16a · 2026-09-22 · sha256 64d1391512fb50b4
consumer-regression-suite git:20260922.71ae16aB
Immutable. This exact content is served forever at /api/v1/blob/64d1391512fb50b4.
--- name: consumer-regression-suite description: Build every downstream consumer of VibeTags against a chosen VibeTags version and report which ones actually pass. Use when the user says "regression suite", "consumer sweep", "test the consumers", "check the downstream repos", "does the new version break anything", or before cutting a VibeTags release. --- # Consumer regression suite VibeTags' own 1537 tests say the processor works. They say nothing about whether a real consumer still builds. This skill answers that second question, for every Java repo under `../` that depends on VibeTags. The executable core is `tools/consumer-sweep.sh` in this repo. This document is the judgement around it: what to run, what a result means, and what not to believe. ## Step 1 — Decide which VibeTags the sweep is testing Two different questions, and they need different setups. Ask which one is wanted if it is not obvious from the request. **"Does the released version still work?"** Use the published version directly. Nothing to install; consumers resolve it from Maven Central. **"Does what we are about to release still work?"** `main` is usually ahead of the newest tag, so the published artifact is not the code under test. Check first: ```bash git -C <vibetags> log "$(git -C <vibetags> tag --sort=-v:refname | head -1)"..main --oneline ``` If that is non-empty, build and install `main` locally: ```bash cd vibetags-annotations && mvn install -DskipTests cd ../vibetags && mvn install -DskipTests cd ../vibetags-bom && mvn install ``` Installing `main` under a version number that is also on Central makes the local copy differ from the published one for every later build on this machine. Say so, and offer the cleanup in Step 5. Installing under a fresh `-SNAPSHOT` avoids it but means no consumer PR can pin the result. ## Step 2 — Run the sweep ```bash bash tools/consumer-sweep.sh <version> # every consumer bash tools/consumer-sweep.sh <version> blindbean # one ``` Per repo it fetches, branches off `origin/main`, rewrites every place that repo declares the VibeTags version, builds with the repo's own wrapper, and prints the build's real exit code. It commits nothing, pushes nothing and opens nothing. Consumers, and how each declares the version: | repo | build | JDK | declares the version in | |---|---|---|---| | `blindbean` | Maven (`mvnw`) | default | `pom.xml` | | `codekarta` | Maven + Gradle | 21 to 25 (`JDK21_HOME`) | `pom.xml` **and** `build.gradle.kts` — both, kept in sync | | `common-license-lib` | Maven + Gradle | default | `pom.xml` **and** `build.gradle.kts` | | `skill3` | Gradle | default | `build.gradle` | | `async-test-lib` | Maven + Gradle | default | `pom.xml` only; Gradle reads it from the POM | Add a repo by adding a row to `CONSUMERS` in the script, not by running it by hand. ### The exit status, and why it has three values | status | meaning | |---:|---| | `0` | every consumer was built and passed | | `1` | at least one consumer failed or errored | | `2` | nothing failed, but at least one consumer was never built | The footer names the same thing in words: `Built 3 of 5 consumer(s): 0 failed, 2 skipped.` `2` exists because a skip is not a milder failure, it is the absence of a measurement, and the two call for opposite actions: `1` means go and look at a broken consumer, `2` means you still know nothing about the ones that did not run. Do not report a `2` as a pass. Until #806 the script ended on a `printf` and always exited `0`, so both of those read as a clean sweep to anything checking the status, including this skill's own reader. ## Step 3 — Read the results honestly A red result is a claim about VibeTags, and most red results are not. Before reporting any failure as a regression: **Rerun it on the base.** Check out `origin/main` with its existing pinned version and run the same command. A test that fails both ways is the consumer's problem, and saying otherwise sends someone hunting a bug that is not there. `blindbean`'s `FheAsyncConcurrencyTest.concurrentBfvOperationsAreThreadSafe` fails roughly one run in three on both the old and new version — it is a 60-second timeout in a thread-safety test and it flakes on a loaded machine. **Compare like with like.** A repo that builds with both Maven and Gradle must be compared tool-for-tool. Running Maven on the new version and Gradle on the base produced a convincing three-file "guardrail drift" here that was entirely an artifact of the comparison. **Distinguish content drift from line-ending churn.** On Windows, `git status` lists a file whose line endings moved even when its text did not. `git diff --numstat` compares after git's normalisation, so an empty numstat with a dirty status is EOL churn, not drift. Only a non-empty numstat means the new VibeTags renders something different — which is a real finding, because the consumer's committed guardrail files are then stale and its check-mode gate will fail. **A first build in a fresh worktree is not evidence.** It can touch files that every subsequent build leaves alone. Reproduce anything surprising before reporting it. ## Step 4 — Fix in VibeTags, not in the consumer If the sweep finds a genuine regression, the fix belongs in VibeTags with a regression test in the VibeTags suite. Working around it in the consumer hides it from the next consumer. Adding a validation check is a line in `ValidationRules.PAIRS`; the other invariants are in `CLAUDE.md` and `docs/LOAD-BEARING.md`. ## Step 5 — Hand it over, do not finish it Report a table: repo, pass or fail, and for every failure whether it reproduces on the base. Report skipped and not-run separately from passed. Then stop. Opening the consumer PRs is a separate, deliberate act: - A consumer cannot pin a version that is not published. If the sweep tested an unreleased `main`, the consumer bumps have to wait for the release, and saying this out loud is part of the report. - Each consumer PR is one bump in one repo. Do not fold in whatever else that repo's `main` is missing. - **Every consumer is swept in a `git worktree`, never in its checkout.** Switching the branch of a checkout somebody is working in is destructive, and on a developer machine a consumer checkout is dirty far more often than not. Keep it that way: `IN_PLACE_REPOS` in the script is empty and meant to stay empty. Clean up with `git -C ../<repo> worktree prune`. If `main` was installed locally over a published version, offer to purge it so a later build resolves the real artifact: ```bash rm -rf ~/.m2/repository/se/deversity/vibetags/*/<version> ``` ## Traps this suite has already paid for - The Maven on PATH here is 3.8.6 and `blindbean`'s enforcer requires 3.9.0+. Always prefer the repo's `./mvnw`; only `blindbean` ships one. - `verify` is a Maven phase, not a Gradle task. A repo built both ways needs two commands. - `cmd | tail` reports `tail`'s exit code. Every build in the script writes to a log and reads `$?` directly. - `sed -i` rewrites a file even when the pattern matches nothing, converting CRLF to LF on Windows and inventing drift in files the bump never needed to touch. - Sweeping a repo **in its checkout** with uncommitted work either fails or drags that work into the branch, and the script skips such a repo on purpose. Never stash or commit somebody else's work to get past it. Since every consumer is worktree-swept, this guard should now never fire; if it does, a repo has been put back into `IN_PLACE_REPOS`. - **A partial sweep reads exactly like a complete one.** The footer prints the same either way, so read the counts: "Built N of 5". This has produced a silently partial result twice, in #617 and again on 2026-09-22 (#790), when four of five repos were skipped for dirty trees and only the footer said so. - **Run this repo's gates after `git add`, not before.** `ReleaseScriptCoverageTest` reads `git ls-files`, so a brand-new file is invisible to it while untracked. A local `mvn verify -Pe2e` went green on these very files and CI then failed on all 17 jobs, because both of them quoted a real release version in an example. Never put a literal release version in a new file: say `<version>`. - Six PRs opening at once made Maven Central answer 429 and failed an unrelated consumer job. A resolution failure naming artifacts you did not touch (`junit-bom` here) is infrastructure; rerun it rather than investigating it. - **Count the rows against the `CONSUMERS` list.** The footer prints after the loop whatever the loop did, so a sweep that covered two repos of five looked exactly like a complete one. The cause was Gradle reading the heredoc that feeds `while read`; every build now runs with `</dev/null`. Keep it that way, and keep counting rows — the footer is not evidence. - **A Gradle consumer can only resolve what its `repositories` block names.** `common-license-lib` and `skill3` declare `mavenCentral()` alone, so an unpublished version fails at resolution before compiling anything. The script detects an unpublished version and injects `mavenLocal()` for the duration of the build, restoring the file afterwards and saying so in the notes. A row carrying that note is a real build result; do not silently promote it to "1.0.x works for this consumer as shipped" — as shipped, that consumer cannot see the artifact at all. - **A consumer that pins a JDK fails on a newer default JDK before VibeTags runs.** `codekarta` requires JDK 21 through 25, so sweeping on a newer default JDK reports false failures for JaCoCo instrumentation. Provide `JDK21_HOME` so the script switches `JAVA_HOME` for that repo's build. Without it the script builds on a default JDK inside the row's range (`21-25`) and skips the repo, naming the range, outside it; enforcer `RequireJavaVersion` failures are reported as toolchain errors rather than regressions.