scoring-checks · diff
git:20260330.4e5d8f6 to git:20260402.460f56e
204 added, 151 removed. Audit A to A.
What this update changes in the checks
No check changed: every fixed check gives the same answer before and after this update.
A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.
---
name: scoring-checks
- description: Add a new deterministic scoring check in src/scoring/checks/ that evaluates CLAUDE.md quality. Follows the Check[] return pattern, uses point constants from src/scoring/constants.ts, and integrates via filterChecksForTarget() in src/scoring/index.ts. Use when user says 'add scoring check', 'new check', 'modify scoring criteria', or works in src/scoring/checks/. Do NOT use for display changes, refactoring scoring logic, or changing how checks are executed.
+ description: Add a new deterministic scoring check in src/scoring/checks/ that evaluates config quality. Follows the Check[] return pattern, uses point constants from src/scoring/constants.ts, and integrates via filterChecksForTarget() in src/scoring/index.ts. Use when user says 'add scoring check', 'new check', 'modify scoring criteria', or works in src/scoring/checks/. Do NOT use for display changes or refactoring scoring logic.
+ paths:
+ - src/scoring/checks/**/*.ts
+ - src/scoring/constants.ts
+ - src/scoring/index.ts
---
- # scoring-checks
+ # Adding a Scoring Check
+ Add a new deterministic check that evaluates a single aspect of AI agent config quality. All checks must be filesystem-based with no network calls or LLM inference.
+
## Critical
- - **All checks are deterministic**: No LLM calls, no network I/O, no randomness. Evaluation must be 100% reproducible given the same input.
- - **Return type is `Check[]`**: Each check MUST return an array of objects with `{ id: string; points: number; detail?: string }`. Points are deducted from 100, not added.
- - **Point values come from `src/scoring/constants.ts`**: Never hardcode point values. Import `POINTS_*` constants and use them directly. Example: `POINTS_MISSING_TITLE` is the deduction for a missing title.
- - **Integration happens automatically**: Once your check exports a default function matching the signature, it is auto-discovered and integrated via `filterChecksForTarget()` in `src/scoring/index.ts`. No manual registration needed.
- - **Check signature**: `async (fingerprint: Fingerprint, target: 'claude' | 'cursor' | 'codex', config: Config) => Promise<Check[]>`
- - **Use `=== 100` for validation**: When checking if the score is perfect, use strict equality `=== 100`, not `>= 100`. Score cannot exceed 100.
+ - **Check must be deterministic**: Same filesystem state → same result every time. No randomness, no external APIs.
+ - **Point values come from constants.ts**: Every `earnedPoints` and `maxPoints` must reference `POINTS_*` from `src/scoring/constants.ts`. Do NOT hardcode numbers.
+ - **Always return `Check[]` array**: Export a function `check<Category>(dir: string): Check[]` where category is one of: `existence`, `quality`, `grounding`, `accuracy`, `freshness`, `bonus`.
+ - **Every check must have**: `id` (kebab-case, unique), `name`, `category`, `maxPoints`, `earnedPoints`, `passed`, `detail`, and optional `suggestion`/`fix`.
+ - **Fix object fields**: `action` (string describing what to do), `data` (context for the fix), `instruction` (user-facing guidance).
+ - **Register in src/scoring/index.ts**: Add the import and spread the result into the `allChecks` array in `computeLocalScore()`.
+ - **Target filtering**: If the check is platform-specific (Claude-only, Cursor-only, etc.), add its ID to the appropriate `*_ONLY_CHECKS` set in `constants.ts`.
## Instructions
- **Step 1: Create the check file**
-
- Create `src/scoring/checks/{name}.ts`. Name must be kebab-case and match the check's purpose.
+ ### Step 1: Define point constants in src/scoring/constants.ts
- ```typescript
- import { Check, Fingerprint } from '../types.js';
- import { POINTS_MISSING_CONTENT } from '../constants.js';
- import type { Config } from '../../llm/types.js';
+ Verify before proceeding: Is your check measurable with a numeric point value?
- export default async function checkMyFeature(
- fingerprint: Fingerprint,
- target: 'claude' | 'cursor' | 'codex',
- config: Config
- ): Promise<Check[]> {
- const checks: Check[] = [];
+ Add constants below the appropriate category section (existence, quality, grounding, accuracy, freshness, bonus):
- // evaluation logic here
- // if condition fails:
- checks.push({
- id: 'my-feature-missing',
- points: POINTS_MISSING_CONTENT,
- detail: 'User message explaining why this was deducted',
- });
+ ```typescript
+ // In the appropriate CATEGORY section, e.g., Quality checks (25 pts):
+ export const POINTS_YOUR_CHECK_NAME = 4; // 1-12 pts typical
- return checks;
- }
+ // If threshold-based, add a companion array:
+ export const YOUR_THRESHOLD_ARRAY = [
+ { minValue: 10, points: 4 },
+ { minValue: 5, points: 2 },
+ ] as const;
```
- Verify: File exists at `src/scoring/checks/{name}.ts` with correct function signature.
+ Check existing patterns: Token budgets use `TOKEN_BUDGET_THRESHOLDS`, code blocks use `CODE_BLOCK_THRESHOLDS`, concreteness uses `CONCRETENESS_THRESHOLDS`.
- **Step 2: Determine what to evaluate**
+ Verify: Review `CATEGORY_MAX` object to ensure your check fits within its category's point budget.
- Inspect the fingerprint object (passed as first argument). Common fields:
- - `fingerprint.claude_md.content` — raw CLAUDE.md file content
- - `fingerprint.claude_md.parsed.sections` — parsed sections (title, instructions, etc.)
- - `fingerprint.cursor_rules` — array of Cursor rule files
- - `fingerprint.codex_instructions` — Codex instruction content
- - `fingerprint.codebase_files` — array of codebase files
+ ### Step 2: Create or edit check function in src/scoring/checks/
- Verify: Check `src/fingerprint/types.ts` to see all available fields on `Fingerprint`.
+ Choose the file based on category. Each file exports a `check<Name>(dir: string): Check[]` function:
- **Step 3: Evaluate the target type**
+ - `existence.ts` — files/directories exist (CLAUDE.md, .cursorrules, skills, MCP servers)
+ - `quality.ts` — config structure, size, clarity (code blocks, token budget, concreteness, duplicates)
+ - `grounding.ts` — references to actual project files/directory structure
+ - `accuracy.ts` — validity of references, git-based config drift
+ - `freshness.ts` — git commit-based staleness, secrets, permissions
+ - `bonus.ts` — hooks, learned content, OpenSkills format
+ - `sources.ts` — source configuration and usage
- Use the `target` parameter to apply checks only to the relevant agent config. Example:
+ Create the function following this structure:
```typescript
- if (target !== 'claude') return []; // Only check CLAUDE.md
+ import type { Check } from '../index.js';
+ import {
+ POINTS_YOUR_CHECK,
+ YOUR_THRESHOLD_ARRAY,
+ } from '../constants.js';
+ import { readFileOrNull } from '../utils.js'; // or other helpers
- if (!fingerprint.claude_md.parsed?.sections?.instructions) {
+ export function checkYourCategory(dir: string): Check[] {
+ const checks: Check[] = [];
+
+ // 1. Measure something concrete
+ const yourMetric = /* e.g., countFiles(), validatePaths(), etc. */;
+ const threshold = YOUR_THRESHOLD_ARRAY.find(t => yourMetric >= t.minValue);
+ const earnedPts = threshold?.points ?? 0;
+
checks.push({
- id: 'missing-instructions',
- points: POINTS_MISSING_INSTRUCTIONS,
- detail: 'CLAUDE.md must include an Instructions section.',
+ id: 'your_unique_check_id',
+ name: 'Human-readable check name',
+ category: 'quality', // matches function context
+ maxPoints: POINTS_YOUR_CHECK,
+ earnedPoints: earnedPts,
+ passed: earnedPts >= Math.ceil(POINTS_YOUR_CHECK * 0.6), // or custom logic
+ detail: `${earnedPts}/${POINTS_YOUR_CHECK} points — ${yourMetric} items found`,
+ suggestion: earnedPts >= POINTS_YOUR_CHECK ? undefined : 'Action to improve',
+ fix: earnedPts >= POINTS_YOUR_CHECK ? undefined : {
+ action: 'verb_noun', // e.g., 'add_code_blocks', 'fix_references'
+ data: { currentValue: yourMetric, targetValue: 10 },
+ instruction: 'Specific, actionable guidance for the user.',
+ },
});
+
+ return checks;
}
```
- Verify: Check only evaluates for the correct target. If target is not relevant, return an empty array.
+ Verify ID uniqueness: Run `grep -r "'your_unique_check_id'" src/scoring/checks/` — should return only your new check.
- **Step 4: Define point deductions**
+ ### Step 3: Handle platform-specific filtering (if applicable)
- Open `src/scoring/constants.ts`. If your deduction type doesn't exist, add it:
+ If your check only applies to certain agents (Claude, Cursor, Codex, GitHub Copilot), register it in `src/scoring/constants.ts`:
```typescript
- export const POINTS_MISSING_TITLE = 5;
- export const POINTS_VAGUE_INSTRUCTIONS = 10;
- // etc.
+ // Add to the appropriate set:
+ export const CLAUDE_ONLY_CHECKS = new Set([
+ 'claude_md_exists',
+ 'your_new_check_id', // ← add here
+ 'claude_rules_exist',
+ ]);
```
- Then import and use in your check:
+ Available sets (update exactly one if applicable):
+ - `CLAUDE_ONLY_CHECKS` — Claude Code targets
+ - `CURSOR_ONLY_CHECKS` — Cursor targets
+ - `CODEX_ONLY_CHECKS` — Codex/OpenCode targets
+ - `COPILOT_ONLY_CHECKS` — GitHub Copilot targets
+ - `BOTH_ONLY_CHECKS` — Both Claude AND Cursor (cross-platform parity)
+ - `NON_CODEX_CHECKS` — Everything except Codex/OpenCode
+ - `CLAUDE_OR_CODEX_CHECKS` — Claude OR Codex
- ```typescript
- import { POINTS_MISSING_TITLE, POINTS_VAGUE_INSTRUCTIONS } from '../constants.js';
+ Verify filtering: Examine `filterChecksForTarget()` in `src/scoring/index.ts` to ensure your category will work correctly for your target agents.
- checks.push({ id: 'no-title', points: POINTS_MISSING_TITLE, detail: '...' });
- ```
+ ### Step 4: Register in src/scoring/index.ts
- Verify: All point values are imported from constants, not hardcoded.
+ Import your function at the top:
- **Step 5: Handle edge cases**
+ ```typescript
+ import { checkYourCategory } from './checks/your-file.js';
+ ```
- For each check, handle missing or null data gracefully:
+ Add to `computeLocalScore()` inside the `allChecks` array initialization:
```typescript
- const content = fingerprint.claude_md.content || '';
- if (!content.trim()) {
- checks.push({
- id: 'empty-claude-md',
- points: POINTS_MISSING_CONTENT,
- detail: 'CLAUDE.md is empty or missing.',
- });
+ export function computeLocalScore(dir: string, targetAgent?: TargetAgent): ScoreResult {
+ const target = targetAgent ?? detectTargetAgent(dir);
+
+ const allChecks: Check[] = [
+ ...checkExistence(dir),
+ ...checkQuality(dir),
+ ...checkGrounding(dir),
+ ...checkAccuracy(dir),
+ ...checkYourCategory(dir), // ← ADD HERE IN ORDER
+ ...checkFreshness(dir),
+ ...checkBonus(dir),
+ ...checkSources(dir),
+ ];
+ // ... rest of function
}
```
- Do NOT throw errors. Always return a `Check[]` array, even if it's empty.
-
- Verify: All code paths return `Promise<Check[]>`. No exceptions are thrown.
+ Verify registration: Run `npm test src/scoring/__tests__/accuracy.test.ts` (or similar) — all existing tests should still pass.
- **Step 6: Write tests**
+ ### Step 5: Write deterministic unit tests
- Create `src/scoring/__tests__/{name}.test.ts`:
+ Create or edit `src/scoring/checks/__tests__/your-file.test.ts`:
```typescript
import { describe, it, expect } from 'vitest';
- import check from '../checks/{name}.js';
- import type { Fingerprint } from '../types.js';
+ import { mkdtempSync, writeFileSync, rmSync } from 'fs';
+ import { join } from 'path';
+ import { checkYourCategory } from '../your-file.js';
+ import { POINTS_YOUR_CHECK } from '../../constants.js';
- describe('check{Name}', () => {
- it('returns no checks when content is valid', async () => {
- const fingerprint: Fingerprint = {
- claude_md: {
- content: 'valid content',
- parsed: { sections: { instructions: 'do X' } },
- },
- // ... other required fields
- };
- const result = await check(fingerprint, 'claude', {} as any);
- expect(result).toEqual([]);
+ describe('checkYourCategory', () => {
+ it('awards full points when condition passes', () => {
+ const dir = mkdtempSync('test-scoring-');
+ try {
+ // Set up the passing condition
+ writeFileSync(join(dir, 'SOME_FILE.md'), 'content that satisfies check');
+
+ const checks = checkYourCategory(dir);
+ const check = checks.find(c => c.id === 'your_unique_check_id');
+
+ expect(check).toBeDefined();
+ expect(check?.passed).toBe(true);
+ expect(check?.earnedPoints).toBe(POINTS_YOUR_CHECK);
+ } finally {
+ rmSync(dir, { recursive: true });
+ }
});
- it('deducts points when content is missing', async () => {
- const fingerprint: Fingerprint = {
- claude_md: { content: '', parsed: { sections: {} } },
- // ...
- };
- const result = await check(fingerprint, 'claude', {} as any);
- expect(result).toContainEqual(
- expect.objectContaining({ id: 'my-check-id', points: expect.any(Number) })
- );
+ it('awards zero points when condition fails', () => {
+ const dir = mkdtempSync('test-scoring-');
+ try {
+ // Don't create the required condition
+ const checks = checkYourCategory(dir);
+ const check = checks.find(c => c.id === 'your_unique_check_id');
+
+ expect(check?.passed).toBe(false);
+ expect(check?.earnedPoints).toBe(0);
+ } finally {
+ rmSync(dir, { recursive: true });
+ }
});
+
+ it('returns correct detail message', () => {
+ const dir = mkdtempSync('test-scoring-');
+ try {
+ const checks = checkYourCategory(dir);
+ const check = checks.find(c => c.id === 'your_unique_check_id');
+ expect(check?.detail).toBeTruthy();
+ } finally {
+ rmSync(dir, { recursive: true });
+ }
+ });
});
```
- Run: `npm run test -- src/scoring/__tests__/{name}.test.ts`
-
- Verify: All tests pass. Test covers happy path and failure cases.
+ Run tests: `npm test src/scoring/checks/__tests__/your-file.test.ts`. All must pass before shipping.
- **Step 7: Verify integration**
+ ## Examples
- Run `npm run build && npm run test`. The check is auto-discovered. No changes to `src/scoring/index.ts` are needed.
+ ### Example 1: Existence Check
- Run a manual test:
+ **Trigger**: User says "Add a check to verify .claude/rules/ directory exists."
- ```bash
- npm run build
- node dist/bin.js score --path /tmp/test-repo
- ```
+ **Actions**:
+ 1. Add `export const POINTS_CLAUDE_RULES = 3;` to constants.ts
+ 2. In existence.ts: `existsSync(join(dir, '.claude', 'rules'))` → true/false
+ 3. Import in index.ts and add `...checkExistence(dir)` (already done)
+ 4. Test with mkdtempSync; verify earnedPoints matches POINTS_CLAUDE_RULES
- Look for your check ID in the output. Verify: Score calculation includes your check's deductions.
+ **Result**: Check `id: 'claude_rules_exist'` returns `earnedPoints: 3, passed: true` when dir exists.
- ## Examples
+ ### Example 2: Quality Check with Thresholds
- **User**: "Add a check to verify CLAUDE.md has a title section."
+ **Trigger**: User says "Verify config has at least 3 code blocks with executable commands."
**Actions**:
- 1. Create `src/scoring/checks/title.ts`
- 2. Add `POINTS_MISSING_TITLE = 5` to `src/scoring/constants.ts`
- 3. Write check logic:
+ 1. Add to constants.ts:
```typescript
- if (target !== 'claude') return [];
- if (!fingerprint.claude_md.parsed?.sections?.title) {
- checks.push({
- id: 'missing-title',
- points: POINTS_MISSING_TITLE,
- detail: 'CLAUDE.md must have a title section at the top.',
- });
- }
- return checks;
+ export const CODE_BLOCK_THRESHOLDS = [
+ { minBlocks: 3, points: 8 },
+ { minBlocks: 2, points: 6 },
+ { minBlocks: 1, points: 3 },
+ ] as const;
```
- 4. Write tests in `src/scoring/__tests__/title.test.ts`
- 5. Run `npm run test && npm run build`
+ 2. In quality.ts:
+ - Parse CLAUDE.md with regex to count \`\`\` blocks
+ - Match against CODE_BLOCK_THRESHOLDS
+ - Return points based on threshold match
+ 3. In fix: Suggest which commands to add
- **Result**: Score now deducts 5 points if CLAUDE.md lacks a title. Integration is automatic.
+ **Result**: 3+ blocks = 8 pts, 2 blocks = 6 pts, 1 block = 3 pts, 0 blocks = 0 pts.
- ---
+ ### Example 3: Accuracy Check (Reference Validation)
- **User**: "Modify the grounding check to be stricter about vague language."
+ **Trigger**: User says "Check that all file paths mentioned in config actually exist."
**Actions**:
- 1. Open `src/scoring/checks/grounding.ts`
- 2. Add new detection for vague words: `['roughly', 'approximately', 'maybe']`
- 3. Create `POINTS_VAGUE_LANGUAGE = 3` in constants
- 4. Push new check: `{ id: 'vague-language', points: POINTS_VAGUE_LANGUAGE, detail: '...' }`
- 5. Update tests to cover the new words
- 6. Run `npm run test -- src/scoring/__tests__/grounding.test.ts`
+ 1. Add `export const POINTS_REFERENCES_VALID = 8;` to constants.ts
+ 2. In accuracy.ts:
+ - Extract backtick-quoted paths and dir patterns from config
+ - Check existence with `existsSync(join(dir, path))`
+ - Calculate ratio: `valid / total`
+ - Award partial points: `Math.round(ratio * POINTS_REFERENCES_VALID)`
+ 3. In fix: List invalid paths the user should fix
- **Result**: Grounding check now penalizes vague language more strictly.
+ **Result**: 80% valid refs = ~6 pts; 100% valid = 8 pts; 0% valid = 0 pts.
## Common Issues
- **Issue**: "Check not appearing in the score output"
- - **Cause**: File is not being imported or function signature is wrong.
- - **Fix**: Verify the file is in `src/scoring/checks/`, is named correctly, and exports a default async function with signature `(fingerprint, target, config) => Promise<Check[]>`.
- - **Verify**: Run `npm run build` and check for any TypeScript errors. Run `node dist/bin.js score --path /tmp/test` and look for your check ID in output.
+ **Issue**: "My check doesn't appear in the score report."
+ **Fix**: 1) Verify ID in `*_ONLY_CHECKS` if platform-specific. 2) Verify import and spread in `index.ts` `allChecks` array. 3) Run `npm test` to ensure no tsc errors. 4) Check `detectTargetAgent()` returns your target platform.
- **Issue**: "TypeError: Cannot read property 'parsed' of undefined"
- - **Cause**: Accessing fingerprint fields without null checks.
- - **Fix**: Always guard with optional chaining or explicit checks:
- ```typescript
- const sections = fingerprint.claude_md?.parsed?.sections;
- if (!sections) return []; // Handle gracefully
- ```
+ **Issue**: "Points are hardcoded but should use constants."
+ **Fix**: Replace all literal numbers like `earnedPoints: 5` with `earnedPoints: POINTS_YOUR_CHECK`. Constants are in `src/scoring/constants.ts` — use them consistently.
- **Issue**: "Score calculation includes my check but points don't match"
- - **Cause**: Using hardcoded point values instead of constants, or using `>= 100` instead of `=== 100`.
- - **Fix**: Import all point values from `src/scoring/constants.ts`. When checking if score is perfect, use `score === 100`, not `score >= 100`.
+ **Issue**: "Check makes an API call or network request."
+ **Fix**: Scoring MUST be deterministic and offline. Use only: `fs` module (readFileSync, existsSync, readdirSync), `path`, `execSync` for git commands. No HTTP, no LLM calls, no external services.
- **Issue**: "Test passes locally but fails in CI"
- - **Cause**: Test relies on file system state or environment variables.
- - **Fix**: Use `memfs` or mock the fingerprint object directly. See existing tests in `src/scoring/__tests__/` for patterns.
+ **Issue**: "Platform-specific check appears for the wrong agent."
+ **Fix**: 1) Verify check ID is in correct `*_ONLY_CHECKS` set. 2) Double-check `filterChecksForTarget()` handles your platform set. 3) Test with `detectTargetAgent()` on a real project.
- **Issue**: "My check runs but returns incorrect points"
- - **Cause**: Logic error in condition or wrong constant value.
- - **Fix**: Add console logs or debug statements. Run `npm run test -- --inspect-brk` to step through. Verify the constant value in `constants.ts` matches expected deduction.
+ **Issue**: "Test fails with 'Module not found' error."
+ **Fix**: Ensure file is in `src/scoring/checks/` (not nested). Use `.js` extension in imports (TypeScript transpiles to ES modules). Run `npm run build` to check for tsc errors.
+
+ **Issue**: "Detail message is confusing or too technical."
+ **Fix**: Use friendly language: `3 code blocks found (need 3 for full points)` instead of `codeBlockCount=3`. Make it clear WHY they got/lost points.
+
+ **Issue**: "Threshold-based check gives wrong points for edge cases."
+ **Fix**: Test all boundaries: value=0, value=threshold, value>>threshold. Use `.find()` to match highest-to-lowest: `find(t => value >= t.minValue)`.
+
+ **Issue**: "Two checks have the same ID."
+ **Fix**: Run `grep -r "'my_id'" src/scoring/checks/` to find duplicates. IDs must be globally unique across all check files. Use descriptive names like `claude_md_exists`, not `check_1`.