Immutable. This exact content is served forever at /api/v1/blob/a877fca74e135419.
--- name: corpus-maintenance description: Use for authored-source correspondence corpus maintenance: verify the vendored CIVIL/EVIL snapshot for drift against upstream, re-harvest or re-pin it, run the offline benchmark, and decide when to advance the pin. --- # Corpus maintenance Use this skill when the user asks to maintain, refresh, drift-check, re-pin, or benchmark the **authored-source correspondence corpus** — the vendored source oracle behind the decompiler's offline benchmark. Produce a short status report: whether the pinned snapshot still corresponds to upstream, whether it needs a re-harvest or a pin bump, and what the benchmark says about the current decompiler. Run from the repository root. The corpus tooling is the `DecompilerHarness` (`tools/DecompilerHarness`), a dev tool, not the shipped `dotnet-inspect` product; run it with `dotnet run --project tools/DecompilerHarness -c Release --`. ## What the corpus is The corpus is a vendored JSONL where each row is a real method identity plus a checksum-verified authored member body captured through SourceLink at harvest time. It lets the benchmark grade source correspondence fully offline, so `SourceUnavailable` becomes a drift signal rather than an expected network outcome. It has two halves: - **CIVIL** (Curated Index of Varied IL) — `civil/corpus.jsonl`, harvested from the 14 pinned assemblies in `eng/prepare-decompiler-corpus.sh`. This is the stable regression target that grows in lock step with the fixed real-world corpus. - **EVIL** (Edge-case Verification of IL Legibility) — `evil/corpus.jsonl`, an adversarial stress set of the most difficult real methods drawn from a much broader pool (`eng/prepare-evil-corpus.sh`), difficulty-ranked off the shared IL substrate. Both live on the `vendor/authored-source-corpus` orphan branch so the harvested third-party source snapshots never enter `main`'s history. Restore them into a git worktree first: ```bash bash eng/restore-authored-source-corpus.sh ``` That materializes `external/authored-source-corpus`. Because it is a worktree on the orphan branch, edits made there (a re-harvest) commit directly to `vendor/authored-source-corpus`. ## The pin, and why it is stable Every corpus assembly is a **pinned** published package version, so the only thing that varies across a decompiler change is the tool, never the input. The dotnet-inspect self-assemblies come from the published `dotnet-inspect.any` package (`SELF_VERSION` in `eng/prepare-decompiler-corpus.sh`), not the local `artifacts/bin` build — that removes corpus drift (#1404) and breaks the circularity where a decompiler change would rebuild both the tool and its own corpus at once. Two pin axes matter: - **Package version** — `SELF_VERSION` (and the third-party versions in `prepare-decompiler-corpus.sh`). - **Source commit** — each row's SourceLink commit, which is where its authored body was captured. Pinning buys attribution and reproducibility; "update to latest" is a deliberate, reviewed re-pin (a re-harvest), because pins rot as upstream moves. Drift verification is the bridge that turns "should we re-pin?" into an evidence-driven decision. ## Verify drift (start here) Drift verification re-acquires each vendored row's authored source *today* and compares it byte-for-byte (newline-normalized) against the stored snapshot. It does **not** run the decompiler; it audits the corpus, not the tool. Prepare the pinned assemblies, then verify CIVIL, resolving dotnet-inspect's own rows from local git via `--repo`: ```bash bash eng/prepare-decompiler-corpus.sh /tmp/corpus-assemblies.txt dotnet run --project tools/DecompilerHarness -c Release -- \ --verify-authored-corpus external/authored-source-corpus/civil/corpus.jsonl \ --repo "$(git rev-parse --show-toplevel)" \ $(cat /tmp/corpus-assemblies.txt) ``` Each row lands in one of three states: - **Verified** — the re-acquired body matches the vendored snapshot. - **Drifted** — the body differs; the upstream source, the harvest slice, or the stored row itself has changed. The report names the row with a short line-count/first-diff summary. - **Unavailable** — the source could not be re-acquired or sliced (offline with no `--repo`, commit gone, checksum mismatch, or an extraction regression). The row is surfaced, never silently dropped. It is report-only by default (exit 0). The run's own integrity still governs the exit code with a named blocker: any corpus row whose assembly was not supplied (`unmatchedRows > 0`) or a run that evaluated no rows fails, so an empty or partially-unmatched run is never a success. Add `--fail-on-drift` to make it a fail-closed gate — the run then exits nonzero unless *every* evaluated row is Verified (any Drifted or Unavailable row fails): ```bash dotnet run --project tools/DecompilerHarness -c Release -- \ --verify-authored-corpus external/authored-source-corpus/civil/corpus.jsonl \ --repo "$(git rev-parse --show-toplevel)" --fail-on-drift \ $(cat /tmp/corpus-assemblies.txt) ``` Always supply the same pinned assemblies the corpus was harvested from; a missing assembly is a blocker, not a pass. ## Interpret and decide - **All Verified** — the pinned snapshot still corresponds to upstream. No action; the corpus is a faithful oracle. - **Drifted rows** — upstream (or the slice) moved off the pinned commit. This is expected as repositories evolve and is **not a bug** — the fix is a deliberate re-harvest / re-pin (below), not a code change. Confirm the drift is upstream churn and not a harvester regression before advancing. - **Unavailable rows** — usually an outage, a missing PDB, or a bad/incomplete `--repo` checkout, not corpus rot. Re-run with a complete `--repo` and network before concluding the source is gone. Persistent Unavailable at a known-good commit points at an extraction regression worth an issue. Distinguish "the corpus drifted from upstream" (re-pin) from "the harness can no longer resolve known-good source" (investigate the harness). ## Re-harvest and re-pin Advancing the snapshot is a deliberate, reviewed act. To re-harvest CIVIL from the current pins straight into the vendored worktree: ```bash bash eng/prepare-decompiler-corpus.sh /tmp/corpus-assemblies.txt dotnet run --project tools/DecompilerHarness -c Release -- \ --harvest-authored-corpus external/authored-source-corpus/civil/corpus.jsonl \ --harvest-target 12000 \ --repo "$(git rev-parse --show-toplevel)" \ $(cat /tmp/corpus-assemblies.txt) ``` `--repo` (repeatable) reads each target's authored source from a local git clone, arbitrated by the PDB checksum, falling back to the network on any mismatch or miss. Pointing it at this checkout resolves the dotnet-inspect self-corpus rows with no GitHub round-trip; it also unlocks private, large, or offline source repositories. Third-party rows are checksum-arbitrated and fall back to the network. To **update to latest** (advance the pin, not just re-capture the same commit), bump `SELF_VERSION` in `eng/prepare-decompiler-corpus.sh` (and any third-party version), re-emit the real-world baseline, then re-harvest as above. Re-harvest EVIL from its broad pool the same way: ```bash bash eng/prepare-evil-corpus.sh /tmp/evil-pool dotnet run --project tools/DecompilerHarness -c Release -- \ --harvest-evil-corpus external/authored-source-corpus/evil/corpus.jsonl \ --harvest-target 12000 \ $(cat /tmp/evil-pool/assemblies.txt) ``` Commit the regenerated `corpus.jsonl` in the `external/authored-source-corpus` worktree (it commits to the orphan branch), and land the pin bump on `main` through a normal reviewed PR. Keep the two in step so the benchmark and the fixed real-world corpus stay aligned. ## Benchmark The benchmark feeds the vendored authored bodies into the same source-correspondence oracle the on-demand census uses and emits a taste card. It validates the corpus and the current decompiler together; run it after a re-harvest or when asked for a decompiler correspondence read: ```bash dotnet run --project tools/DecompilerHarness -c Release -- \ --benchmark-authored-corpus external/authored-source-corpus/civil/corpus.jsonl \ $(cat /tmp/corpus-assemblies.txt) ``` Add `--json` for structured output. Each target is `Correct`, one of four valid-but-different taste buckets, `Invalid`, or a diagnostic bucket. The run exits nonzero on any `Invalid`, `Drift`, or `Unsupported`, and on a dishonest run (unmatched rows or zero targets). A `Drift` bucket here means the corpus identity no longer resolves in the pinned assembly — treat it like a Drifted verify row. ## Guardrails - Keep the benchmark hermetic: it must stay fully offline over the vendored bodies. Do not add live network resolution to the benchmark path. - Do not delete the `vendor/authored-source-corpus` orphan branch or migrate its bodies onto `main`. The vendored snapshot is the reproducible oracle. - Build the harness with the solution graph: `dotnet build dotnet-inspect.slnx -c Release`. If a user-global NuGet feed trips NU1507, add `--configfile ./nuget.config`. - Verify and benchmark over the pinned assemblies, never a fresh local build of the tool (that reintroduces #1404 circularity). ## Report shape Keep the report short: ```md Corpus maintenance: <verified | drifted | re-pinned | benchmarked> - Corpus: CIVIL <verified/drifted/unavailable counts>; EVIL <if checked>. - Drift cause: <none | upstream churn (re-pin) | outage/--repo (retry) | harness regression (issue)>. - Action: <none | re-harvest | bump SELF_VERSION + re-harvest | file issue>. - Benchmark (if run): <correct / taste-bucket / invalid / drift counts>, exit <code>. ```