git:20261013.533a84e to git:20261013.421fd31

---

name: ds-change-delta

description: "Render a before/after telemetry delta bracketing a methodology-change"

user-invocable: true

---

# /ds-change-delta

Render a before/after telemetry delta bracketing a methodology-change

"cut point" - a git commit SHA or an ISO8601 date. Answers "did this

specific commit help or hurt" by re-deriving sessions, subagent spawns

per session, hook denies per session, tokens per session, wall time per

session, and (for a git-checkout `--repo` with a resolvable `gh` remote)

commits per merged PR, for two equal-length windows either side of the

cut point, from the same live telemetry stores `ds-cost`, `ds-evaluate`,

`ds-hook-fire-report`, and `ds-branch-prune` already read. The `eval`

block adds the same comparison over `ds-eval`'s figures (cost-state

dollars per session), split by entrypoint and stated with its

confounders.

Implementation: `bin/ds-change-delta` (Python 3 stdlib).

## Usage

```

ds-change-delta --cut <SHA|ISO8601> [--cut-repo PATH] [--window-days N] [--pr-lookahead-days N] --repo PATH [--repo PATH ...] [--json] [--no-eval]

```

- `--cut` (required): a git SHA (resolved to its **committer** date on

`--cut-repo`, deliberate for a squash-merge repo) or an ISO8601

date/datetime. A 7-40 hex-char argument is tried as a SHA FIRST

(real all-digit abbreviated SHAs exist and are also valid compact ISO

dates, so SHA resolution is never skipped based on digit/letter

composition); if that fails, it falls through to ISO date parsing.

An argument that resolves neither way exits 1 with an error message.

- `--cut-repo` (optional, default: the first `--repo` that resolves to

a real directory with `.agentic/` or `.git`).

- `--window-days N` (optional, default 14). Before window

`[cut - N days, cut)`, after window `[cut, cut + N days)` - always

genuinely equal-length, never truncated to "now".

- `--repo PATH` (repeatable, at least one required). Session, spawn,

hook-deny, token, and wall-time metrics are computed over the MERGED

row set across every `--repo` - not independently per repo and then

combined. PR history is reported separately, per repo. Session-log

rows count once per `session_uuid` (its latest-`ts` row, in whichever window

that row falls), so pre-2026-08-08 per-turn cumulative rows do not

inflate session counts or sums.

- `--pr-lookahead-days N` (optional, default 7): merged PRs up to N days

past a window's end still count for that window's tickets.

- `--json` for the full machine-readable report; default is a fixed-width

table.

- `--no-eval` skips the `eval` side (see below) and omits its key.

## Spawn telemetry: `agent_model` and `per_ticket` (DS-246)

Both read only hook spawn rows with `data.telemetry_v: 2` from each

`--repo`'s `.agentic/events.jsonl`; older rows never contribute. Their

status comes from the same precedence as every other metric, over the v2

rows' own extent (intersected across repos, reported as `v2_coverage`):

a window the v2 rows do not fully bracket is `INSUFFICIENT_COVERAGE`, and

a store with no v2 rows is `ABSENT`, both with `null` values.

- `agent_model`: per agent x model, the runs ending in the window with

model id goes to `dollars_exact`; an id priced by family (for example

`claude-opus-5` at the Opus 5.5 rate) goes to `dollars_family_estimated`

and is named in `family_priced_models`. A run with no tokens is counted

in `tokens_null_runs`, never priced as zero.

- `per_ticket`: one row per ticket whose first v2 spawn (else its ledger

`opened_ts`) falls in the window, keyed by the hooks' `task_id`. Each

row reports Skeptic reviews (every Skeptic run, with

`skeptic_reviews_with_findings` counting those that carry a findings

count), max iteration, Critical/Major/Minor totals,

`re_raised` (`null` when no `findings_log` entry records it), QA runs and

fail-backs (FAIL or PARTIAL), merged and follow-up PRs, re-entries more

than 1h after the first merge, Critical+Major findings after it (`null`

when the review count fell back to the ledger's `skeptic_rounds`, or when

any Skeptic run after the first merge carries no findings count),

`ledger_skeptic_rounds` (shown beside a telemetry count so an undercount

is visible), `unit_count` (from the ledger, default 1), rework loops

(`max(skeptic_reviews - unit_count, 0) + qa_fail_backs +

max(prs_merged - unit_count, 0)`, so a clean ticket with one review and

one PR per unit reports 0), end-to-end wall (final merge minus first

spawn) and dollars. A ticket whose ledger `opened_ts` predates the v2

extent is listed in `excluded_pre_v2` instead; a ticket with no ledger

row that began before the v2 extent cannot be detected, so it is

anchored at its first v2 spawn with a short wall and fewer reviews.

- PR attribution: a PR number in a repo's `ticket-ledger.jsonl` counts

for every ticket recording it. Any other PR counts for T only when its

title starts with `T:`, `[T]`, `[T]:` or `type(T):`, or else when the

last `/`-separated segment of its branch starts with T; a PR that only

mentions T does not count, and one matching several tickets goes to

`ambiguous_prs`. When `gh` is unavailable, only the PR-derived fields

(and `loops`) are `null`.

## Evaluation side: `eval`

`bin/_eval.py`'s `evaluate()` (the `ds-eval` engine) runs once per window

over every Claude Code profile's transcript store. Every figure uses its

envelope `{status: OK|PARTIAL|ABSENT, value, coverage, missing}`: a

missing input is `ABSENT` with `value: null` and a reason, never 0, and an

after window still running is `PARTIAL` with "after window not yet

elapsed".

- `before` / `after`: `ds-eval`'s own window figures - `sessions`,

`cost_usd` (cost reconciled to the harness cost-state),

`usd_per_reconciled_session`, `straddling_sessions`. `evaluate()` lists a

session in every window it overlaps.

- The figures below count each session once, on the side its first

timestamp falls in (`meta.sessions_assigned`). The figure is cost-state

dollars per session, over the sessions whose cost-state covers them.

Empty sessions (no usage record and no cost-state spend, i.e. sessions

that never reached the model) are left out of every rate, mix and prefix

figure and counted in `confounders.<side>.empty_sessions`.

- Small samples: each rate carries its session count as `sessions`, and

each delta and `mix_shifted` carry `n` (`{before, after}`). Fewer than

10 sessions on either side makes the figure `PARTIAL` with reason

"n < 10", so a K=5 prefix figure is always `PARTIAL`.

- `confounders.before` / `confounders.after`: session shares by

entrypoint, model and Claude Code version (the usage key with the most

conductor messages), `base_commits` (the first-parent commits on each

repo's base ref in the window) and `empty_sessions`.

- `pooled` and `per_entrypoint` (`cli`, `sdk-cli`, ...): before, after and

delta. A pooled delta can carry the opposite sign of every entrypoint's

own delta when the mix moves, so read `per_entrypoint` first.

- `mix_shifted`: `true` when any entrypoint's session share moved more

than 10 points (exactly 10 is not a shift); `points` gives each

entrypoint's movement.

- `prefix_test` at K = 5, 25 and 100: the mean cost per session of the

last K sessions before the cut against the first K after it. A step

change (for example a one-time prompt-prefix removal) shows the same

delta at every K; a trend grows with K. It cannot tell a fixed

per-session saving from a per-turn rate change: both lower a session's

cost, and the test sees only whole-session cost. Fewer than K sessions

on a side is `PARTIAL`. The pooled

`prefix_test` mixes entrypoints, so when `mix_shifted` is true it

compares mostly one entrypoint before the cut against another after

it; `prefix_test_per_entrypoint` runs the same test within each

entrypoint and is the one to read then.

A window still running is `PARTIAL` on its own side: the after window of

a recent cut, and the before window too when `--cut` is in the future.

The default table prints `n=<before>/<after>` and the `PARTIAL` reasons

on each `eval` row.

The `eval` side never calls `gh` or `ds-feedback`, and any failure inside

it becomes an `ABSENT` `eval` block naming the error, never a non-zero

exit. It costs two transcript passes: tens of seconds on a repo with

hundreds of sessions in the two windows. `--no-eval` skips it and omits

the `eval` key.

## Coverage and status

Every metric's `--json` output names the file(s) or query it read and

the row/record count it consumed - the default table is a compact

summary and omits both; use `--json` to see the provenance. When a

store is absent, empty, or does not reach the

requested window on one side, that side reports a status word

(`ABSENT`, `NOT_YET_ELAPSED`, `INSUFFICIENT_COVERAGE`, `ZERO_SESSIONS`,

or `OK`) and a `null` delta rather than a misleading numeric zero - the

`value` field is also `null` on every non-`OK` status, including

`NOT_YET_ELAPSED`; a not-yet-elapsed after window never reports a

partial informational figure. Token

and wall-time metrics additionally carry a `data_quality` flag

(`"zero-filled"`, `"zero-filled-before"`, or `"zero-filled-after"`) that

forces `delta` to `null` even when both sides otherwise report `OK` -

this checkout's own session-log is majority zero-filled for

tokens/wall_seconds, and without this check a real instrumentation gap

would print as a large numeric methodology-change effect. The

`data_quality` check only runs when both sides report `OK`: if one side

is a genuine zero and the other side is any non-`OK` status, the zero

side's real value is shown as-is (never flagged `zero-filled`) and

`delta` stays `null` because a non-`OK` opposite side already forces it.

With multiple `--repo` arguments, a metric's before/after VALUE always

sums across every repo's rows, but coverage is judged per store as the

INTERSECTION of each contributing repo's own recorded extent, not the

union of the merged rows - a repo whose telemetry only starts after the

cut (e.g. newly onboarded) cannot be masked by a sibling repo that alone

brackets the whole window. When one repo's extent is the limiting one,

the `note` field names it. That `note` (including which repo limited

coverage) is reachable only via `--json` - the default table does not

surface it.

## Known limitation

Passing a repo root AND one of that repo's own `.claude/worktrees/*`

checkouts as separate `--repo` arguments double-counts fires rows: the

parent repo's stranded-copy glob already reads the child worktree's own

`.agentic/.enforcement-fires.jsonl`, and passing the child again as its

own `--repo` reads it a second time. Avoid passing a repo and its own

nested worktree checkouts together.

## What this tool does NOT measure

- **Skeptic-loop yield per round** - this repo's telemetry cannot support

that measurement at all (see `MEMORY.md`'s KNW-20260818-002 through

-006); it is out of scope here and is never synthesized from a proxy

signal. `per_ticket` reports review counts and findings totals, not

per-round yield.

- **Consumer-repo PR history beyond a resolvable `gh` remote** - a

`--repo` with no `origin` remote `gh` can resolve reports

`NOT_A_GIT_REPO` for that one signal while still contributing to the

merged session/hook/token/wall metrics.

This tool reports correlational deltas across a cut point, never a

verdict on whether a methodology change was good or bad - read the

status fields before trusting any number, and treat a real delta as one

input to a human judgment, not the judgment itself.

## Read-only guarantee

The tool never writes to any telemetry file and never mutates git or gh

state. Every subprocess call is one of `git rev-parse --verify`,

`git show`, `git remote get-url`, `gh repo view`, or `gh api graphql`

(all read-only). See `bin/tests/test_ds_change_delta.py`'s R5 test for

the enforced argv allowlist and a fixture-repo HEAD/status

byte-identity assertion. The `eval` side runs through `bin/_eval.py`'s

own reads: `git log`, `git rev-list --count`, `git remote`, the base-ref

lookups (`git show-ref`, `git symbolic-ref`, `git rev-parse`), all

read-only, with no `gh` and no `ds-feedback` call. The tool writes

nothing itself, under the home directory or anywhere else; `gh`, when

present, is run for PR history and may write its own state there.

## Known limitation: PR-history window granularity

PR-history windows are built as GitHub search `merged:` datetime ranges

(second precision), with the end instant one second before the window's

own end - a PR merged exactly on the shared boundary between the before

and after windows lands in exactly one of them, never both. The residual

granularity is one second: a PR merged in the same second as a window

boundary cannot be resolved further, since `mergedAt` carries no

sub-second precision. Not a practical concern at any realistic

`--window-days`.

## Known limitation: GitHub search's 1,000-result ceiling

The GraphQL `search` connection this tool uses for PR history has a

documented hard cap of 1,000 total results per query, regardless of

pagination depth. A `--window-days` wide enough to push a repo's merged-PR

`issueCount` past that cap will never let the completeness check

reconcile, and the tool correctly reports `INCOMPLETE_WINDOW` rather than

silently truncating - a self-detecting condition, not a silent one. Not a

practical concern at the default 14-day window.

## Retirement condition (Pillar 8)

A one-shot diagnostic, matching `bin/ds-skill-load-rate`'s retirement

shape: it retires once an operator has run it across a representative

set of methodology-change PRs (e.g. the first 5-10 real uses), recorded

in a decision entry whether the reported deltas ever changed a

merge/rollback/rollout decision, and made an explicit call on whether

the tool is worth keeping - not an invocation-count trigger, since the

tool keeps no record of its own invocations (read-only guarantee above).