Immutable. This exact content is served forever at /api/v1/blob/bc4e15812de2eba2.
--- name: test-harness-auditor description: "This skill should be used when auditing a repo's test, lint, type-check, static analysis, build, and debug infrastructure for AI coding agents. Use when entering a new repo, when asked to 'audit tests', 'audit harness', 'check test infrastructure', 'lint audit', 'what testing tools are configured', or when a repo has no .claude/lint-rules.json. Generates optimized configs for the lint-on-write hook." argument-hint: "[path]" user-invocable: true allowed-tools: - Bash - Read - Write - Edit - Glob - Grep --- # Test Harness Auditor Audit any repo's feedback infrastructure across six layers and generate optimized configs for AI coding agents. ## When to Run - Entering a new repo with no `.claude/lint-rules.json` - User asks to audit tests, lint setup, or agent infrastructure - After cloning a repo to check what feedback loops exist - Periodically to catch configuration drift ## Two-Phase Workflow ### Phase 1: Audit (read-only) Run the audit script to scan the current repo: ```bash uv run ~/.claude/skills/test-harness-auditor/scripts/audit.py ``` Or target a specific directory: ```bash uv run ~/.claude/skills/test-harness-auditor/scripts/audit.py /path/to/repo ``` For machine-readable output (consumed by Phase 2): ```bash uv run ~/.claude/skills/test-harness-auditor/scripts/audit.py --json > /tmp/audit.json ``` To save a snapshot for drift detection (tracks score changes over time): ```bash uv run ~/.claude/skills/test-harness-auditor/scripts/audit.py --save ``` Combine flags: `--json --save` saves the snapshot AND outputs JSON. On subsequent `--save` runs, the report includes a drift section showing score regressions, config changes, and residue file changes. The script produces a structured Markdown report (or JSON with `--json`) with: - **Stack summary**: detected language, frameworks, package manager, actual scripts from package.json - **Scorecard**: 0-3 score for each of the six layers (test, lint, type-check, SA, build, debug) - **Findings**: per-layer details on what was detected - **Debugging residue**: files matching `*_v2.*`, `*_backup.*`, `*_fixed.*` patterns - **Recommendations**: prioritized by impact on agent feedback quality (P0-P3) Present the report to the user. Ask which recommendations to implement before proceeding to Phase 2. ### Phase 1.5: Convention Extraction (optional) Extract "never X"/"always Y" constraints from CLAUDE.md into candidate lint rules: ```bash uv run ~/.claude/skills/test-harness-auditor/scripts/extract_conventions.py ``` Outputs JSON with candidate lint-rules.json entries derived from project constraints. Present candidates to the user for approval before merging. ### Phase 2: Config Generation (after user approval) Run the generation script (optionally with audit JSON for accurate commands): ```bash uv run ~/.claude/skills/test-harness-auditor/scripts/generate.py --audit /tmp/audit.json ``` Or without audit data (re-detects stack): ```bash uv run ~/.claude/skills/test-harness-auditor/scripts/generate.py ``` When `--audit` is used, generate.py uses actual commands from package.json (vitest, playwright, biome, etc.) instead of generic templates, and detects separate E2E vs unit test runners. This produces three outputs: 1. **`.claude/lint-rules.json`** — custom grep-based rules for the lint-on-write hook - Stack-specific rules (security, debugging residue, error boundaries, observability) - Auto-includes matching rule packs from `rule-library/` (react, rust-workspace, python-cli) - Merges with existing config if present (preserves user customizations) - Tagged rules (`_tag` field) enable idempotent re-runs 2. **CLAUDE.md testing section** — test/lint/typecheck/build/SA commands - Follows claude-md-manager conventions (command-first, concise) - **Section-aware merge**: when existing CLAUDE.md is found, surgically replaces only `## Commands` and `## Testing` sections, preserving all other content - Present as a proposal — do not overwrite existing CLAUDE.md content 3. **Hook recommendations** — which PostToolUse hooks to enable - lint-on-write (primary), test-on-fix, type-check-on-write For each generated config, present it to the user and ask for approval before writing. ## Scoring System | Score | Meaning | |-------|---------| | 0 | Absent — agent is flying blind on this layer | | 1 | Minimal — basic tool present but not configured for agents | | 2 | Adequate — tool configured and runnable | | 3 | Excellent — strict mode, mutation testing, or advanced config | ## Six Assessment Layers 1. **Test suite**: framework, runner command, coverage config, mutation testing 2. **Linting**: standard linter, custom rules, agent-specific rules 3. **Type checking**: type checker, strict mode, CI integration 4. **Static analysis**: security scanners, complexity checkers, dependency audit 5. **Build/compilation**: build command, incremental build, CI validation 6. **Debugger/REPL**: debugger availability, REPL access ## Integration - **lint-on-write hook**: generated `lint-rules.json` is consumed by `~/.claude/hooks/lint-on-write.py` (violations are severity-tiered: BLOCKING > HIGH > MEDIUM) - **claude-md-manager**: generated CLAUDE.md sections follow its conventions (WHAT/WHY/HOW, command-first) - **agents-md-manager**: for cross-agent compatibility, consider also generating AGENTS.md - **agnix**: complementary tool — validates the *agent config files* themselves (385 rules for CLAUDE.md/AGENTS.md/SKILL.md stale paths, dead commands, context rot). Our skill validates the *codebase infrastructure*. ## Rule Library Domain-specific rule packs in `rule-library/` are auto-loaded by generate.py based on detected frameworks and stack: | Pack | Matches | Rules | |------|---------|-------| | `react.json` | react, next frameworks | fetch-in-effect, excessive-useState, inline-style, direct-DOM, indexOf | | `functional-ts.json` | react, next + typescript | 21 rules: immutability (array-mutation, sort-reverse, delete-operator), const-by-default (let-binding, var), declarative (for-loop, forEach, while, switch), React (class-component, class-state-props, effect-mutation), type safety (any-type), async (raw-promise), testability (nondeterministic-call, console-residue) | | `rust-workspace.json` | rust stack | chained-clone, box-dyn, allow-dead-code, todo-macro, relative-path-dep | | `python-cli.json` | python stack | os.path-vs-pathlib, bare-print, subprocess-call, Optional-vs-union | To add a custom rule pack, create a JSON file in `rule-library/` with `_frameworks` (list) and/or `_stack` (string) matching fields, plus a `rules` array. Pack rules use `pack:` prefix in `_tag` for dedup. ## References Load these on-demand when deeper context is needed: - `references/stack-profiles.md` — per-stack detection rules and tool recommendations - `references/factory-lint-categories.md` — 7 Factory.ai agent lint categories with grep patterns - `references/anti-patterns.md` — 10 AI-specific anti-patterns with detection heuristics ## Scope - First-class stacks: JavaScript/TypeScript, Rust, Python, Go, Ruby - Other stacks get basic detection with generic recommendations - Does not write or modify test files - Does not install tools (recommends what to install) - Does not modify CI/CD pipelines