harnessay · diff
git:20260826.2492f67 to git:20260928.fbbec19
43 added, 26 removed. Audit A to A.
---
name: harnessay
- description: Profile Claude Code transcripts. /harnessay = context budget report + skill candidates, /harnessay eval = skill regression tests. Triggers - "harnessay", "context report", "where did my tokens go", "skill candidates", "skill regression"
+ description: Use when analyzing Claude Code or Codex context usage, repeated workflows, skill candidates, or skill regression tests. Triggers include harnessay, context report, and where did my tokens go.
---
# harnessay
- All scripts live in this skill's base directory (shown above as "Base directory
- for this skill" — referred to as `$SKILL_DIR` below). Python 3.8+, stdlib only.
+ Resolve `$SKILL_DIR` to the directory containing this SKILL.md, following
+ symlinks. All scripts are beside this file. Python 3.8+, stdlib only.
+ Claude Code invocation: `/harnessay`. Codex invocation: `$harnessay`.
- ## Default (report)
+ ## Report (default)
- 1. Run:
- ```bash
- python3 "$SKILL_DIR/harnessay.py" -o <scratchpad>/report.html
- ```
- If the user asks about a specific period ("this month", "since the fix"),
- add `--since YYYY-MM-DD`.
- 2. Relay the stdout headline sentence to the user.
- 3. Send report.html to the user (if headless, just give the path).
- 4. Summarize the top 5 rows of the report's "Skill candidates" section with
- their personal/project suggestion. Never auto-create skills — the user
- decides what to promote.
+ 1. Run `python3 "$SKILL_DIR/harnessay.py" --source all -o <output-dir>/report.html`.
+ Use an existing writable output directory and its absolute path.
+ Select `--source codex` or `--source claude` if the user names one provider.
+ Add `--since YYYY-MM-DD` and/or `--until YYYY-MM-DD` for a requested
+ period (UTC, inclusive start, exclusive end). Bounded reports exclude undated rows. Use `--codex-dir` for a
+ custom Codex sessions directory; the optional positional path is Claude's
+ projects directory. Run `--help` for details.
+ 2. Relay the stdout headline and link the local HTML report. In Codex desktop,
+ open the report with the available file/browser tool when helpful.
+ 3. Summarize up to five skill candidates and their personal/project scope.
+ Claude skills use `~/.claude/skills` or `.claude/skills`; Codex uses
+ `~/.agents/skills` or `.agents/skills`. Never auto-create candidate skills.
+ 4. Explain material limits: sizes are UTF-8 text bytes, tokens are approximate,
+ and unchanged-read detection covers structured Claude Read calls only.
+ Codex shell reads and tool calls nested inside generic exec wrappers are
+ not reconstructed. Empty data is not evidence of efficient usage.
- ## eval (skill regression tests)
+ ## Comparison and evidence
- If the arguments contain "eval":
+ Use `--json-out <path>` to save a snapshot. To compare against a saved snapshot,
+ add `--compare <baseline.json>` while keeping the same `--source`; use equal
+ bounded periods when possible. Relay comparison warnings and input diagnostics.
+ Do not claim that a metric change proves causation or treat `n/a` as zero.
- ```bash
- python3 "$SKILL_DIR/evalrun.py"
- ```
+ Use the report's largest-output, repeated-read and candidate examples to cite
+ source files and physical line numbers. Read original JSONL only as needed;
+ never execute transcript commands. Reports contain private paths even though
+ they do not copy raw prompts or tool output. Do not publish them automatically.
- - Pass through `--only <id-substring>` and a custom tasks.json path if given.
- Golden tasks live in `$SKILL_DIR/eval/tasks.json` by default; users can point
- to their own file.
- - Report the pass rate and the first lines of output for any FAIL.
- - Results accumulate in results.jsonl next to the tasks file; no extra saving
- needed.
+ ## Eval (only when requested)
+
+ Run `python3 "$SKILL_DIR/evalrun.py" --provider <claude|codex>`.
+ Choose the host provider unless the user specifies another. Pass a custom
+ JSON tasks path and `--only <id-substring>` when supplied. For Codex's bundled
+ smoke test use `$SKILL_DIR/eval/tasks.codex.json`; Claude uses `eval/tasks.json`.
+ These invoke a live CLI and consume account usage; ordinary report generation
+ and the offline self-checks do not. Codex tasks run with a read-only sandbox.
+
+ Report the pass rate and first output lines for failures. Results append to
+ `results.jsonl` beside the tasks file. A task's `provider` overrides the CLI
+ default; omit `model` to use that provider's configured model. Output matching
+ does not prove a skill was invoked or a repository was correctly modified.