git:20260424.9b749cd to git:20260425.203da12

71 added, 248 removed. Audit A to A.

What this update changes in the checks

No check changed: every fixed check gives the same answer before and after this update.

A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.

---
name: scoring-checks
- description: Add a new deterministic scoring check in src/scoring/checks/ that evaluates config quality. Follows the Check[] return pattern, uses point constants from src/scoring/constants.ts, and integrates via filterChecksForTarget() in src/scoring/index.ts. Use when user says 'add scoring check', 'new check', 'modify scoring criteria', or works in src/scoring/checks/. Do NOT use for display changes or refactoring scoring logic.
- paths:
- - src/scoring/checks/**/*.ts
- - src/scoring/constants.ts
- - src/scoring/index.ts
+ description: Add a new deterministic scoring check in src/scoring/checks/. Returns Check[] array, uses constants from src/scoring/constants.ts, integrates in src/scoring/index.ts. Use when adding scoring logic, creating checks, modifying check weights. Do NOT use for display/UI changes or check result formatting.
---
- # Adding a Scoring Check
-
- Add a new deterministic check that evaluates a single aspect of AI agent config quality. All checks must be filesystem-based with no network calls or LLM inference.
+ # Scoring Checks
## Critical
- - **Check must be deterministic**: Same filesystem state → same result every time. No randomness, no external APIs.
- - **Point values come from constants.ts**: Every `earnedPoints` and `maxPoints` must reference `POINTS_*` from `src/scoring/constants.ts`. Do NOT hardcode numbers.
- - **Always return `Check[]` array**: Export a function `check<Category>(dir: string): Check[]` where category is one of: `existence`, `quality`, `grounding`, `accuracy`, `freshness`, `bonus`.
- - **Every check must have**: `id` (kebab-case, unique), `name`, `category`, `maxPoints`, `earnedPoints`, `passed`, `detail`, and optional `suggestion`/`fix`.
- - **Fix object fields**: `action` (string describing what to do), `data` (context for the fix), `instruction` (user-facing guidance).
- - **Register in src/scoring/index.ts**: Add the import and spread the result into the `allChecks` array in `computeLocalScore()`.
- - **Target filtering**: If the check is platform-specific (Claude-only, Cursor-only, etc.), add its ID to the appropriate `*_ONLY_CHECKS` set in `constants.ts`.
+ 1. **Check must return `Check[]`** — Never return a single check or a Promise. Return an empty array `[]` only if there are genuinely zero checks; always push at least one check object.
+ 2. **Use constants from `src/scoring/constants.ts`** — All `maxPoints` and `earnedPoints` values must reference `POINTS_*` constants; never hardcode numbers.
+ 3. **Deterministic only** — No LLM calls, no network. Scoring checks use `fs`, `path`, and `execSync` for git commands only.
+ 4. **File structure**: Add checks to existing category files in `src/scoring/checks/`. Export a **named function** with signature `check{Category}(dir: string): Check[]`.
+ 5. **Integration point**: Spread the result into `allChecks` in `computeLocalScore()` inside `src/scoring/index.ts`.
## Instructions
- ### Step 1: Define point constants in src/scoring/constants.ts
-
- Verify before proceeding: Is your check measurable with a numeric point value?
-
- Add constants below the appropriate category section (existence, quality, grounding, accuracy, freshness, bonus):
-
- ```typescript
- // In the appropriate CATEGORY section, e.g., Quality checks (25 pts):
- export const POINTS_YOUR_CHECK_NAME = 4; // 1-12 pts typical
+ 1. **Choose or create a check file** in `src/scoring/checks/` matching the category (`existence.ts`, `quality.ts`, `grounding.ts`, `accuracy.ts`, `freshness.ts`, `bonus.ts`, `sources.ts`).
+ - Import `type { Check }` from `../index.js`.
+ - Import constants from `../constants.js`.
+ - Add a new `export function check{Category}(dir: string): Check[]` function (or add to the existing one).
+ - Verify file compiles: `npx tsc --noEmit`.
- // If threshold-based, add a companion array:
- export const YOUR_THRESHOLD_ARRAY = [
- { minValue: 10, points: 4 },
- { minValue: 5, points: 2 },
- ] as const;
- ```
+ 2. **Inspect the filesystem** using the `dir` parameter.
+ - Use `fs.existsSync(join(dir, 'path'))`, `fs.readFileSync(join(dir, 'file'), 'utf-8')`, etc.
+ - For git-based checks: `execSync('git log ...', { cwd: dir })`.
+ - No external APIs, no LLM calls.
- Check existing patterns: Token budgets use `TOKEN_BUDGET_THRESHOLDS`, code blocks use `CODE_BLOCK_THRESHOLDS`, concreteness uses `CONCRETENESS_THRESHOLDS`.
+ 3. **Build the Check object**.
+ - `id`: Unique kebab-case string (e.g., `'claude_md_exists'`, `'skills_count'`).
+ - `name`: Human-readable display name.
+ - `category`: One of `'existence' | 'quality' | 'grounding' | 'accuracy' | 'freshness' | 'bonus'`.
+ - `maxPoints` / `earnedPoints`: Must reference `POINTS_*` constants.
+ - `passed`: `earnedPoints >= Math.ceil(maxPoints * 0.6)` (or custom logic).
+ - `detail`: Human-friendly message explaining the result.
+ - Optional: `suggestion` (if not passed) and `fix` object with `action`, `data`, `instruction`.
- Verify: Review `CATEGORY_MAX` object to ensure your check fits within its category's point budget.
+ 4. **Reference constants for all numeric values**.
+ - Add new constants to `src/scoring/constants.ts` if needed.
+ - Example: `export const POINTS_YOUR_CHECK = 4;` then use `maxPoints: POINTS_YOUR_CHECK`.
- ### Step 2: Create or edit check function in src/scoring/checks/
+ 5. **Register in src/scoring/index.ts**.
+ - The existing category functions (`checkExistence`, `checkQuality`, etc.) are already called in `computeLocalScore()`.
+ - If adding a new function: import it and spread into `allChecks`.
+ - Verify tests pass: `npm test -- src/scoring/__tests__/`.
- Choose the file based on category. Each file exports a `check<Name>(dir: string): Check[]` function:
+ 6. **Write a test** in `src/scoring/checks/__tests__/{category}.test.ts`.
+ - Use `mkdtempSync` to create a temp directory; write files to simulate conditions.
+ - Test both passing and failing cases.
+ - Verify: `npx vitest run src/scoring/checks/__tests__/{category}.test.ts`.
- - `existence.ts` — files/directories exist (CLAUDE.md, .cursorrules, skills, MCP servers)
- - `quality.ts` — config structure, size, clarity (code blocks, token budget, concreteness, duplicates)
- - `grounding.ts` — references to actual project files/directory structure
- - `accuracy.ts` — validity of references, git-based config drift
- - `freshness.ts` — git commit-based staleness, secrets, permissions
- - `bonus.ts` — hooks, learned content, OpenSkills format
- - `sources.ts` — source configuration and usage
+ ## Examples
- Create the function following this structure:
+ **User says**: "Add a scoring check that verifies CLAUDE.md exists and is not empty."
+ **Actions**:
+ 1. In `src/scoring/checks/existence.ts`, add to the `checkExistence(dir: string): Check[]` function:
```typescript
import type { Check } from '../index.js';
- import {
- POINTS_YOUR_CHECK,
- YOUR_THRESHOLD_ARRAY,
- } from '../constants.js';
- import { readFileOrNull } from '../utils.js'; // or other helpers
+ import { POINTS_CLAUDE_MD_EXISTS } from '../constants.js';
+ import { existsSync, readFileSync } from 'fs';
+ import { join } from 'path';
- export function checkYourCategory(dir: string): Check[] {
+ export function checkExistence(dir: string): Check[] {
const checks: Check[] = [];
-
- // 1. Measure something concrete
- const yourMetric = /* e.g., countFiles(), validatePaths(), etc. */;
- const threshold = YOUR_THRESHOLD_ARRAY.find(t => yourMetric >= t.minValue);
- const earnedPts = threshold?.points ?? 0;
-
+ const claudeMdPath = join(dir, 'CLAUDE.md');
+ const exists = existsSync(claudeMdPath);
+ const nonEmpty = exists && readFileSync(claudeMdPath, 'utf-8').trim().length > 0;
+ const earned = nonEmpty ? POINTS_CLAUDE_MD_EXISTS : 0;
checks.push({
- id: 'your_unique_check_id',
- name: 'Human-readable check name',
- category: 'quality', // matches function context
- maxPoints: POINTS_YOUR_CHECK,
- earnedPoints: earnedPts,
- passed: earnedPts >= Math.ceil(POINTS_YOUR_CHECK * 0.6), // or custom logic
- detail: `${earnedPts}/${POINTS_YOUR_CHECK} points — ${yourMetric} items found`,
- suggestion: earnedPts >= POINTS_YOUR_CHECK ? undefined : 'Action to improve',
- fix: earnedPts >= POINTS_YOUR_CHECK ? undefined : {
- action: 'verb_noun', // e.g., 'add_code_blocks', 'fix_references'
- data: { currentValue: yourMetric, targetValue: 10 },
- instruction: 'Specific, actionable guidance for the user.',
- },
+ id: 'claude_md_exists',
+ name: 'CLAUDE.md exists and is not empty',
+ category: 'existence',
+ maxPoints: POINTS_CLAUDE_MD_EXISTS,
+ earnedPoints: earned,
+ passed: earned > 0,
+ detail: exists ? (nonEmpty ? 'CLAUDE.md present and non-empty' : 'CLAUDE.md is empty') : 'CLAUDE.md missing',
});
-
return checks;
}
```
- Verify ID uniqueness: Run `grep -r "'your_unique_check_id'" src/scoring/checks/` — should return only your new check.
-
- ### Step 3: Handle platform-specific filtering (if applicable)
-
- If your check only applies to certain agents (Claude, Cursor, Codex, GitHub Copilot), register it in `src/scoring/constants.ts`:
-
- ```typescript
- // Add to the appropriate set:
- export const CLAUDE_ONLY_CHECKS = new Set([
- 'claude_md_exists',
- 'your_new_check_id', // ← add here
- 'claude_rules_exist',
- ]);
- ```
-
- Available sets (update exactly one if applicable):
- - `CLAUDE_ONLY_CHECKS` — Claude Code targets
- - `CURSOR_ONLY_CHECKS` — Cursor targets
- - `CODEX_ONLY_CHECKS` — Codex/OpenCode targets
- - `COPILOT_ONLY_CHECKS` — GitHub Copilot targets
- - `BOTH_ONLY_CHECKS` — Both Claude AND Cursor (cross-platform parity)
- - `NON_CODEX_CHECKS` — Everything except Codex/OpenCode
- - `CLAUDE_OR_CODEX_CHECKS` — Claude OR Codex
-
- Verify filtering: Examine `filterChecksForTarget()` in `src/scoring/index.ts` to ensure your category will work correctly for your target agents.
-
- ### Step 4: Register in src/scoring/index.ts
-
- Import your function at the top:
-
- ```typescript
- import { checkYourCategory } from './checks/your-file.js';
- ```
-
- Add to `computeLocalScore()` inside the `allChecks` array initialization:
-
- ```typescript
- export function computeLocalScore(dir: string, targetAgent?: TargetAgent): ScoreResult {
- const target = targetAgent ?? detectTargetAgent(dir);
-
- const allChecks: Check[] = [
- ...checkExistence(dir),
- ...checkQuality(dir),
- ...checkGrounding(dir),
- ...checkAccuracy(dir),
- ...checkYourCategory(dir), // ← ADD HERE IN ORDER
- ...checkFreshness(dir),
- ...checkBonus(dir),
- ...checkSources(dir),
- ];
- // ... rest of function
- }
- ```
-
- Verify registration: Run `npm test src/scoring/__tests__/accuracy.test.ts` (or similar) — all existing tests should still pass.
-
- ### Step 5: Write deterministic unit tests
-
- Create or edit `src/scoring/checks/__tests__/your-file.test.ts`:
-
- ```typescript
- import { describe, it, expect } from 'vitest';
- import { mkdtempSync, writeFileSync, rmSync } from 'fs';
- import { join } from 'path';
- import { checkYourCategory } from '../your-file.js';
- import { POINTS_YOUR_CHECK } from '../../constants.js';
-
- describe('checkYourCategory', () => {
- it('awards full points when condition passes', () => {
- const dir = mkdtempSync('test-scoring-');
- try {
- // Set up the passing condition
- writeFileSync(join(dir, 'SOME_FILE.md'), 'content that satisfies check');
-
- const checks = checkYourCategory(dir);
- const check = checks.find(c => c.id === 'your_unique_check_id');
-
- expect(check).toBeDefined();
- expect(check?.passed).toBe(true);
- expect(check?.earnedPoints).toBe(POINTS_YOUR_CHECK);
- } finally {
- rmSync(dir, { recursive: true });
- }
- });
-
- it('awards zero points when condition fails', () => {
- const dir = mkdtempSync('test-scoring-');
- try {
- // Don't create the required condition
- const checks = checkYourCategory(dir);
- const check = checks.find(c => c.id === 'your_unique_check_id');
-
- expect(check?.passed).toBe(false);
- expect(check?.earnedPoints).toBe(0);
- } finally {
- rmSync(dir, { recursive: true });
- }
- });
-
- it('returns correct detail message', () => {
- const dir = mkdtempSync('test-scoring-');
- try {
- const checks = checkYourCategory(dir);
- const check = checks.find(c => c.id === 'your_unique_check_id');
- expect(check?.detail).toBeTruthy();
- } finally {
- rmSync(dir, { recursive: true });
- }
- });
- });
- ```
-
- Run tests: `npm test src/scoring/checks/__tests__/your-file.test.ts`. All must pass before shipping.
-
- ## Examples
-
- ### Example 1: Existence Check
-
- **Trigger**: User says "Add a check to verify .claude/rules/ directory exists."
-
- **Actions**:
- 1. Add `export const POINTS_CLAUDE_RULES = 3;` to constants.ts
- 2. In existence.ts: `existsSync(join(dir, '.claude', 'rules'))` → true/false
- 3. Import in index.ts and add `...checkExistence(dir)` (already done)
- 4. Test with mkdtempSync; verify earnedPoints matches POINTS_CLAUDE_RULES
-
- **Result**: Check `id: 'claude_rules_exist'` returns `earnedPoints: 3, passed: true` when dir exists.
-
- ### Example 2: Quality Check with Thresholds
-
- **Trigger**: User says "Verify config has at least 3 code blocks with executable commands."
-
- **Actions**:
- 1. Add to constants.ts:
- ```typescript
- export const CODE_BLOCK_THRESHOLDS = [
- { minBlocks: 3, points: 8 },
- { minBlocks: 2, points: 6 },
- { minBlocks: 1, points: 3 },
- ] as const;
- ```
- 2. In quality.ts:
- - Parse CLAUDE.md with regex to count \`\`\` blocks
- - Match against CODE_BLOCK_THRESHOLDS
- - Return points based on threshold match
- 3. In fix: Suggest which commands to add
-
- **Result**: 3+ blocks = 8 pts, 2 blocks = 6 pts, 1 block = 3 pts, 0 blocks = 0 pts.
-
- ### Example 3: Accuracy Check (Reference Validation)
+ 2. Add to `src/scoring/constants.ts`: `export const POINTS_CLAUDE_MD_EXISTS = 10;`
- **Trigger**: User says "Check that all file paths mentioned in config actually exist."
+ 3. `checkExistence` is already called in `computeLocalScore()` in `src/scoring/index.ts` — no changes needed there.
- **Actions**:
- 1. Add `export const POINTS_REFERENCES_VALID = 8;` to constants.ts
- 2. In accuracy.ts:
- - Extract backtick-quoted paths and dir patterns from config
- - Check existence with `existsSync(join(dir, path))`
- - Calculate ratio: `valid / total`
- - Award partial points: `Math.round(ratio * POINTS_REFERENCES_VALID)`
- 3. In fix: List invalid paths the user should fix
+ 4. Test with `npm test -- src/scoring/`.
- **Result**: 80% valid refs = ~6 pts; 100% valid = 8 pts; 0% valid = 0 pts.
+ **Result**: Check appears in score output under the existence category.
## Common Issues
- **Issue**: "My check doesn't appear in the score report."
- **Fix**: 1) Verify ID in `*_ONLY_CHECKS` if platform-specific. 2) Verify import and spread in `index.ts` `allChecks` array. 3) Run `npm test` to ensure no tsc errors. 4) Check `detectTargetAgent()` returns your target platform.
-
- **Issue**: "Points are hardcoded but should use constants."
- **Fix**: Replace all literal numbers like `earnedPoints: 5` with `earnedPoints: POINTS_YOUR_CHECK`. Constants are in `src/scoring/constants.ts` — use them consistently.
-
- **Issue**: "Check makes an API call or network request."
- **Fix**: Scoring MUST be deterministic and offline. Use only: `fs` module (readFileSync, existsSync, readdirSync), `path`, `execSync` for git commands. No HTTP, no LLM calls, no external services.
-
- **Issue**: "Platform-specific check appears for the wrong agent."
- **Fix**: 1) Verify check ID is in correct `*_ONLY_CHECKS` set. 2) Double-check `filterChecksForTarget()` handles your platform set. 3) Test with `detectTargetAgent()` on a real project.
-
- **Issue**: "Test fails with 'Module not found' error."
- **Fix**: Ensure file is in `src/scoring/checks/` (not nested). Use `.js` extension in imports (TypeScript transpiles to ES modules). Run `npm run build` to check for tsc errors.
+ **"Check is not appearing in score output"**
+ - Verify the check ID is not in a `*_ONLY_CHECKS` set that excludes your target agent.
+ - Verify the function is called (or its result is spread) in `computeLocalScore()` in `src/scoring/index.ts`.
+ - Run `npm run build && npm test` to ensure no import errors.
- **Issue**: "Detail message is confusing or too technical."
- **Fix**: Use friendly language: `3 code blocks found (need 3 for full points)` instead of `codeBlockCount=3`. Make it clear WHY they got/lost points.
+ **"Points are hardcoded but should use constants"**
+ - Replace all literal numbers like `earnedPoints: 5` with `earnedPoints: POINTS_YOUR_CHECK`.
+ - Constants are in `src/scoring/constants.ts` — add new ones if needed.
- **Issue**: "Threshold-based check gives wrong points for edge cases."
- **Fix**: Test all boundaries: value=0, value=threshold, value>>threshold. Use `.find()` to match highest-to-lowest: `find(t => value >= t.minValue)`.
+ **"Platform-specific check appears for wrong agent"**
+ - Add check ID to the appropriate `*_ONLY_CHECKS` set in `src/scoring/constants.ts`.
+ - Verify `filterChecksForTarget()` in `src/scoring/index.ts` handles your platform set.
- **Issue**: "Two checks have the same ID."
- **Fix**: Run `grep -r "'my_id'" src/scoring/checks/` to find duplicates. IDs must be globally unique across all check files. Use descriptive names like `claude_md_exists`, not `check_1`.
+ **"Test fails with file not found"**
+ - Use `mkdtempSync` to create a real temp directory; write files with `writeFileSync`.
+ - Always clean up in a `finally` block: `rmSync(dir, { recursive: true })`.