1 added, 2 removed. Audit A to A.
---
name: corpus-maintenance
description: Use for authored-source correspondence corpus maintenance: verify the vendored CIVIL/EVIL snapshot for drift against upstream, re-harvest or re-pin it, run the offline benchmark, and decide when to advance the pin.
---
# Corpus maintenance
Use this skill when the user asks to maintain, refresh, drift-check, re-pin, or
benchmark the **authored-source correspondence corpus** — the vendored source
oracle behind the decompiler's offline benchmark. Produce a short status report:
whether the pinned snapshot still corresponds to upstream, whether it needs a
re-harvest or a pin bump, and what the benchmark says about the current
decompiler.
Run from the repository root. The corpus tooling is the `DecompilerHarness`
(`tools/DecompilerHarness`), a dev tool, not the shipped `dotnet-inspect`
product; run it with `dotnet run --project tools/DecompilerHarness -c Release --`.
## What the corpus is
The corpus is a vendored JSONL where each row is a real method identity plus a
checksum-verified authored member body captured through SourceLink at harvest
time. It lets the benchmark grade source correspondence fully offline, so
`SourceUnavailable` becomes a drift signal rather than an expected network
outcome. It has two halves:
- **CIVIL** (Curated Index of Varied IL) — `civil/corpus.jsonl`, harvested from
the 14 pinned assemblies in `eng/prepare-decompiler-corpus.sh`. This is the
stable regression target that grows in lock step with the fixed real-world
corpus.
- **EVIL** (Edge-case Verification of IL Legibility) — `evil/corpus.jsonl`, an
adversarial stress set of the most difficult real methods drawn from a much
broader pool (`eng/prepare-evil-corpus.sh`), difficulty-ranked off the shared
IL substrate.
Both live on the `vendor/authored-source-corpus` orphan branch so the harvested
third-party source snapshots never enter `main`'s history. Restore them into a
git worktree first:
```bash
bash eng/restore-authored-source-corpus.sh
```
That materializes `external/authored-source-corpus`. Because it is a worktree on
the orphan branch, edits made there (a re-harvest) commit directly to
`vendor/authored-source-corpus`.
## The pin, and why it is stable
Every corpus assembly is a **pinned** published package version, so the only
thing that varies across a decompiler change is the tool, never the input. The
dotnet-inspect self-assemblies come from the published `dotnet-inspect.any`
package (`SELF_VERSION` in `eng/prepare-decompiler-corpus.sh`), not the local
`artifacts/bin` build — that removes corpus drift (#1404) and breaks the
circularity where a decompiler change would rebuild both the tool and its own
corpus at once.
Two pin axes matter:
- **Package version** — `SELF_VERSION` (and the third-party versions in
`prepare-decompiler-corpus.sh`).
- **Source commit** — each row's SourceLink commit, which is where its authored
body was captured.
Pinning buys attribution and reproducibility; "update to latest" is a
deliberate, reviewed re-pin (a re-harvest), because pins rot as upstream moves.
Drift verification is the bridge that turns "should we re-pin?" into an
evidence-driven decision.
## Verify drift (start here)
Drift verification re-acquires each vendored row's authored source *today* and
compares it byte-for-byte (newline-normalized) against the stored snapshot. It
does **not** run the decompiler; it audits the corpus, not the tool. Prepare the
pinned assemblies, then verify CIVIL, resolving dotnet-inspect's own rows from
local git via `--repo`:
```bash
bash eng/prepare-decompiler-corpus.sh /tmp/corpus-assemblies.txt
dotnet run --project tools/DecompilerHarness -c Release -- \
--verify-authored-corpus external/authored-source-corpus/civil/corpus.jsonl \
--repo "$(git rev-parse --show-toplevel)" \
$(cat /tmp/corpus-assemblies.txt)
```
Each row lands in one of three states:
- **Verified** — the re-acquired body matches the vendored snapshot.
- **Drifted** — the body differs; the upstream source, the harvest slice, or the
stored row itself has changed. The report names the row with a short
line-count/first-diff summary.
- **Unavailable** — the source could not be re-acquired or sliced (offline with
no `--repo`, commit gone, checksum mismatch, or an extraction regression). The
row is surfaced, never silently dropped.
It is report-only by default (exit 0). The run's own integrity still governs the
exit code with a named blocker: any corpus row whose assembly was not supplied
(`unmatchedRows > 0`) or a run that evaluated no rows fails, so an empty or
partially-unmatched run is never a success. Add `--fail-on-drift` to make it a
fail-closed gate — the run then exits nonzero unless *every* evaluated row is
Verified (any Drifted or Unavailable row fails):
```bash
dotnet run --project tools/DecompilerHarness -c Release -- \
--verify-authored-corpus external/authored-source-corpus/civil/corpus.jsonl \
--repo "$(git rev-parse --show-toplevel)" --fail-on-drift \
$(cat /tmp/corpus-assemblies.txt)
```
Always supply the same pinned assemblies the corpus was harvested from; a
missing assembly is a blocker, not a pass.
## Interpret and decide
- **All Verified** — the pinned snapshot still corresponds to upstream. No
action; the corpus is a faithful oracle.
- **Drifted rows** — upstream (or the slice) moved off the pinned commit. This is
expected as repositories evolve and is **not a bug** — the fix is a deliberate
re-harvest / re-pin (below), not a code change. Confirm the drift is upstream
churn and not a harvester regression before advancing.
- **Unavailable rows** — usually an outage, a missing PDB, or a bad/incomplete
`--repo` checkout, not corpus rot. Re-run with a complete `--repo` and network
before concluding the source is gone. Persistent Unavailable at a known-good
commit points at an extraction regression worth an issue.
Distinguish "the corpus drifted from upstream" (re-pin) from "the harness can no
longer resolve known-good source" (investigate the harness).
## Re-harvest and re-pin
Advancing the snapshot is a deliberate, reviewed act. To re-harvest CIVIL from
the current pins straight into the vendored worktree:
```bash
bash eng/prepare-decompiler-corpus.sh /tmp/corpus-assemblies.txt
dotnet run --project tools/DecompilerHarness -c Release -- \
--harvest-authored-corpus external/authored-source-corpus/civil/corpus.jsonl \
--harvest-target 12000 \
--repo "$(git rev-parse --show-toplevel)" \
$(cat /tmp/corpus-assemblies.txt)
```
`--repo` (repeatable) reads each target's authored source from a local git clone,
arbitrated by the PDB checksum, falling back to the network on any mismatch or
miss. Pointing it at this checkout resolves the dotnet-inspect self-corpus rows
with no GitHub round-trip; it also unlocks private, large, or offline source
repositories. Third-party rows are checksum-arbitrated and fall back to the
network.
To **update to latest** (advance the pin, not just re-capture the same commit),
bump `SELF_VERSION` in `eng/prepare-decompiler-corpus.sh` (and any third-party
version), re-emit the real-world baseline, then re-harvest as above. Re-harvest
EVIL from its broad pool the same way:
```bash
bash eng/prepare-evil-corpus.sh /tmp/evil-pool
dotnet run --project tools/DecompilerHarness -c Release -- \
--harvest-evil-corpus external/authored-source-corpus/evil/corpus.jsonl \
--harvest-target 12000 \
$(cat /tmp/evil-pool/assemblies.txt)
```
Commit the regenerated `corpus.jsonl` in the `external/authored-source-corpus`
worktree (it commits to the orphan branch), and land the pin bump on `main`
through a normal reviewed PR. Keep the two in step so the benchmark and the
fixed real-world corpus stay aligned.
## Benchmark
The benchmark feeds the vendored authored bodies into the same
source-correspondence oracle the on-demand census uses and emits a taste card. It
validates the corpus and the current decompiler together; run it after a
re-harvest or when asked for a decompiler correspondence read:
```bash
dotnet run --project tools/DecompilerHarness -c Release -- \
--benchmark-authored-corpus external/authored-source-corpus/civil/corpus.jsonl \
$(cat /tmp/corpus-assemblies.txt)
```
Add `--json` for structured output. Each target is `Correct`, one of four
valid-but-different taste buckets, `Invalid`, or a diagnostic bucket. The run
exits nonzero on any `Invalid`, `Drift`, or `Unsupported`, and on a dishonest run
(unmatched rows or zero targets). A `Drift` bucket here means the corpus identity
no longer resolves in the pinned assembly — treat it like a Drifted verify row.
## Guardrails
- Keep the benchmark hermetic: it must stay fully offline over the vendored
bodies. Do not add live network resolution to the benchmark path.
- Do not delete the `vendor/authored-source-corpus` orphan branch or migrate its
bodies onto `main`. The vendored snapshot is the reproducible oracle.
- Build the harness with the solution graph: `dotnet build dotnet-inspect.slnx
- -c Release`. If a user-global NuGet feed trips NU1507, add `--configfile
- ./nuget.config`.
+ -c Release`.
- Verify and benchmark over the pinned assemblies, never a fresh local build of
the tool (that reintroduces #1404 circularity).
## Report shape
Keep the report short:
```md
Corpus maintenance: <verified | drifted | re-pinned | benchmarked>
- Corpus: CIVIL <verified/drifted/unavailable counts>; EVIL <if checked>.
- Drift cause: <none | upstream churn (re-pin) | outage/--repo (retry) | harness regression (issue)>.
- Action: <none | re-harvest | bump SELF_VERSION + re-harvest | file issue>.
- Benchmark (if run): <correct / taste-bucket / invalid / drift counts>, exit <code>.
```