scoring-checks · diff
git:20260319.c97e02c to git:20260323.a091579
63 added, 122 removed. Audit A to A.
What this update changes in the checks
No check changed: every fixed check gives the same answer before and after this update.
A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.
---
name: scoring-checks
- description: Adds a new deterministic scoring check to src/scoring/checks/ implementing the Check interface from src/scoring/index.ts. Point values from src/scoring/constants.ts. Checks receive dir:string and return Check[]. Use when user says 'add scoring check', 'new check', 'score X', or modifies src/scoring/. Do NOT use for LLM-based scoring, real-time validation, or checks requiring external API calls.
+ description: Adds a new deterministic scoring check to src/scoring/checks/ implementing the Check interface. Point values from src/scoring/constants.ts. Checks are synchronous functions receiving dir:string and returning Check[]. Use when user says 'add scoring check', 'new check', 'score X', or modifies src/scoring/. Do NOT use for LLM-based scoring or checks requiring external API calls.
---
# Scoring Checks
## Critical
- 1. **Checks are deterministic and synchronous.** No LLM calls, no async operations, no external APIs. Use only filesystem reads and sync analysis.
- 2. **Every check must implement the `Check` interface** from `src/scoring/index.ts`:
- ```typescript
- export interface Check {
- id: string; // kebab-case, e.g., "claude-md-exists"
- label: string; // Human-readable, e.g., "CLAUDE.md exists"
- points: number; // From src/scoring/constants.ts
- passed: boolean; // true/false
- details?: string; // Optional explanation
- }
- ```
- 3. **Points must be defined in `src/scoring/constants.ts` BEFORE the check.** Do not hardcode point values in the check file.
- 4. **Each check file must export a function with signature:** `export function checkName(dir: string): Check[]` that returns an array of Check objects.
- 5. **Import and register the check in `src/scoring/index.ts`** in the `runChecks()` function.
+ - **Synchronous only**: No async/await, no Promise, no LLM calls. Checks must complete instantly on local filesystem inspection.
+ - **Check interface**: Every check returns `Check[]` where `Check = { name: string; points: number; reason: string }`. Points MUST match a constant in `src/scoring/constants.ts`.
+ - **No side effects**: Do not write files, modify state, or call external APIs. Only inspect the filesystem.
+ - **Export as named function**: Export your check function from `src/scoring/checks/your-check.ts` and import it in `src/scoring/checks/index.ts` in the `CHECKS` array.
## Instructions
- 1. **Define point value in `src/scoring/constants.ts`**
- - Add constant: `export const CHECK_NAME_POINTS = <number>;`
- - Example: `export const CLAUDE_MD_EXISTS_POINTS = 15;`
- - Verify the constant is exported and value aligns with project scoring scale (typically 5–50 points per check)
-
- 2. **Create the check file at `src/scoring/checks/<check-name>.ts`**
- - File name must be kebab-case (e.g., `claude-md-exists.ts`)
- - Verify the file path follows existing pattern: see `src/scoring/checks/existence.ts`, `quality.ts`, etc.
-
- 3. **Implement the check function with the Check interface**
- ```typescript
- import { Check } from '../index';
- import { CHECK_NAME_POINTS } from '../constants';
- import { existsSync } from 'fs';
+ 1. **Review the Check interface and constants**
+ - Open `src/scoring/index.ts` and locate the `Check` type: `{ name: string; points: number; reason: string }`
+ - Open `src/scoring/constants.ts` and identify point values: `POINTS.EXISTENCE`, `POINTS.QUALITY`, etc.
+ - Verify your check's points value exists in constants before proceeding.
- export function checkName(dir: string): Check[] {
- const filePath = `${dir}/path/to/check`;
- const passed = existsSync(filePath);
- return [{
- id: 'check-id',
- label: 'Human label',
- points: CHECK_NAME_POINTS,
- passed,
- details: passed ? 'Found.' : 'Not found.'
- }];
- }
- ```
- - Use `existsSync`, `readFileSync`, `statSync` from `fs` module (sync only)
- - Use `glob.sync()` for multi-file patterns
- - Return an array (even single checks must be wrapped in `[]`)
- - Verify the check handles missing files gracefully (returns `passed: false`, not throws)
+ 2. **Create the check file in src/scoring/checks/**
+ - File path: `src/scoring/checks/your-check-name.ts`
+ - Import from `src/scoring/constants.ts`: `import { POINTS } from '../constants'`
+ - Signature: `export function checkYourName(dir: string): Check[]`
+ - Verify the function name is camelCase and matches the check purpose.
- 4. **Import and call the check in `src/scoring/index.ts`**
- - Add import: `import { checkName } from './checks/<check-name>';`
- - Add call in `runChecks(dir: string)` function:
- ```typescript
- const checks: Check[] = [];
- checks.push(...checkName(dir));
- ```
- - Verify the function is called before the final return statement
+ 3. **Implement the check logic**
+ - Use `fs.existsSync()` or `fs.statSync()` to inspect files (no fs/promises)
+ - Use `glob.sync()` from the `glob` package for file pattern matching (matches existing checks)
+ - Return an empty array `[]` if the check passes or does not apply
+ - Return `[{ name: 'Check Name', points: POINTS.CONSTANT, reason: 'Explanation of failure' }]` on failure
+ - Example: `fs.existsSync(path.join(dir, '.cursor/rules.md'))` to check file existence
+ - Verify your fs calls complete synchronously before proceeding.
- 5. **Write tests in `src/scoring/__tests__/<check-name>.test.ts`**
- - Use Vitest and `memfs` for isolated filesystem
- - Example:
- ```typescript
- import { describe, it, expect, beforeEach } from 'vitest';
- import { vol } from 'memfs';
- import { checkName } from '../checks/<check-name>';
+ 4. **Register the check in src/scoring/checks/index.ts**
+ - Import your function: `import { checkYourName } from './your-check-name'`
+ - Add it to the `CHECKS` array: `export const CHECKS = [checkExistence, checkQuality, ..., checkYourName]`
+ - Verify the CHECKS array is exported as a named export.
- describe('checkName', () => {
- beforeEach(() => vol.reset());
- it('passes when file exists', () => {
- vol.fromJSON({ '/project/file.md': 'content' });
- const result = checkName('/project');
- expect(result[0].passed).toBe(true);
- });
- it('fails when file missing', () => {
- const result = checkName('/project');
- expect(result[0].passed).toBe(false);
- });
- });
- ```
- - Verify both pass and fail cases are tested
+ 5. **Add unit tests in src/scoring/checks/__tests__/**
+ - File path: `src/scoring/checks/__tests__/your-check-name.test.ts`
+ - Use Vitest: `import { describe, it, expect, beforeEach, afterEach } from 'vitest'`
+ - Use `memfs` to mock the filesystem (see `src/scoring/checks/__tests__/existence.test.ts` for pattern)
+ - Test both pass (empty array) and fail (returns Check with correct points) cases
+ - Run `npm run test -- src/scoring/checks/__tests__/your-check-name.test.ts` to verify.
- 6. **Run scoring tests to validate**
- - Execute: `npm run test -- src/scoring`
- - Verify all new tests pass and the check integrates without errors
- - Verify the check appears in `caliber score` output
+ 6. **Update src/scoring/constants.ts if needed**
+ - If your points value does not exist, add it to `POINTS` object
+ - Example: `POINTS.MY_CHECK = 15` (use integers)
+ - Verify the constant is exported before proceeding.
## Examples
- **User says:** "Add a check that verifies CLAUDE.md exists and has content"
+ **User says:** "Add a check that scores points if AGENTS.md exists"
**Actions:**
- 1. Add to `src/scoring/constants.ts`: `export const CLAUDE_MD_EXISTS_POINTS = 15;`
- 2. Create `src/scoring/checks/claude-md-exists.ts`:
+ 1. Create `src/scoring/checks/agents-file.ts`:
```typescript
- import { Check } from '../index';
- import { CLAUDE_MD_EXISTS_POINTS } from '../constants';
- import { existsSync, readFileSync } from 'fs';
import path from 'path';
+ import fs from 'fs';
+ import { POINTS } from '../constants';
+ import type { Check } from '../index';
- export function checkClaudeMdExists(dir: string): Check[] {
- const filePath = path.join(dir, 'CLAUDE.md');
- const passed = existsSync(filePath) && readFileSync(filePath, 'utf-8').trim().length > 0;
- return [{
- id: 'claude-md-exists',
- label: 'CLAUDE.md exists and has content',
- points: CLAUDE_MD_EXISTS_POINTS,
- passed,
- details: passed ? 'CLAUDE.md found with content.' : 'CLAUDE.md missing or empty.'
- }];
+ export function checkAgentsFile(dir: string): Check[] {
+ const agentsPath = path.join(dir, 'AGENTS.md');
+ if (!fs.existsSync(agentsPath)) {
+ return [{
+ name: 'AGENTS.md exists',
+ points: POINTS.EXISTENCE,
+ reason: 'AGENTS.md not found'
+ }];
+ }
+ return [];
}
```
- 3. Update `src/scoring/index.ts`:
- - Add: `import { checkClaudeMdExists } from './checks/claude-md-exists';`
- - In `runChecks()`: `checks.push(...checkClaudeMdExists(dir));`
- 4. Create test in `src/scoring/__tests__/claude-md-exists.test.ts`
- 5. Run: `npm run test -- src/scoring/checks`
+ 2. Add to `src/scoring/checks/index.ts`: `import { checkAgentsFile } from './agents-file'` and add to `CHECKS` array
+ 3. Create `src/scoring/checks/__tests__/agents-file.test.ts` with memfs mocks
+ 4. Run `npm run test -- src/scoring/checks`
- **Result:** New check appears in `caliber score` output with 15 points on pass.
+ **Result:** Check runs synchronously, returns points only if file missing.
- ## Common Issues
+ ## Anti-patterns
- **"Cannot find module 'src/scoring/checks/check-name'"**
- - Verify file exists at exact path: `src/scoring/checks/<check-name>.ts`
- - Verify import in `src/scoring/index.ts` uses correct filename
- - Run: `ls -la src/scoring/checks/` to list files
+ 1. **Do NOT use async/await or Promise.** Checks must be fully synchronous. Use `fs.readFileSync()`, not `fs.promises.readFile()`. Use `glob.sync()`, not `glob.glob()`. Correct approach: `const files = glob.sync('**/*.ts', { cwd: dir })`
- **"Check returns undefined or null"**
- - Check function must return an array `Check[]`, never `undefined`
- - Always wrap return: `return [{ ... }];` even for single checks
- - Verify `return` statement exists before the closing brace
+ 2. **Do NOT call LLM functions or external APIs.** No `llmCall()`, no `fetch()`, no `axios.get()`. Checks run offline and must complete in <100ms. Correct approach: Parse JSON from a local file with `JSON.parse(fs.readFileSync(...))` if needed.
- **"Check undefined in scoring output"**
- - Verify `checkName()` is called in `runChecks()` in `src/scoring/index.ts`
- - Verify `checks.push(...checkName(dir));` is present
- - Run: `npm run test -- src/scoring` to catch registration errors
+ 3. **Do NOT return positive points on pass.** Return `[]` (empty array) for passing checks. Only return `Check[]` with points on failure. Correct approach: `if (condition) return []; return [{ name: 'Check', points: POINTS.X, reason: '...' }];`
- **"Test fails: 'No such file or directory'"**
- - Use `memfs` and `vol.fromJSON()` to mock filesystem; do NOT use real `fs`
- - Example: `vol.fromJSON({ '/project/CLAUDE.md': 'content' });`
- - Verify `beforeEach(() => vol.reset());` clears state between tests
+ ## Common Issues
- **"Points value is wrong or missing"**
- - Verify constant is defined in `src/scoring/constants.ts`: `export const CHECK_NAME_POINTS = <number>;`
- - Verify import in check file: `import { CHECK_NAME_POINTS } from '../constants';`
- - Verify check uses imported constant: `points: CHECK_NAME_POINTS,` not hardcoded number
- - Run: `grep -n 'CHECK_NAME_POINTS' src/scoring/constants.ts src/scoring/checks/<check-name>.ts` to verify both exist
+ - **Error: `fs.readFile is not a function`** → You used `fs/promises`. Use `import fs from 'fs'` (not `fs.promises`) and call `fs.readFileSync(path)` instead.
+ - **Error: `glob.glob is not a function`** → You used async glob. Use `import { glob } from 'glob'` and call `glob.sync(pattern, { cwd: dir })` instead.
+ - **Test fails: `ENOENT: no such file or directory`** → Mock the filesystem with memfs before calling your check. See `existence.test.ts`: `const vol = new Volume(); vol.fromJSON({...}); const fs = vol.promises` (note: for sync tests, use `vol` directly).
+ - **Check not running in `npm run test:coverage`** → Verify your check is added to the `CHECKS` array in `src/scoring/checks/index.ts` and that the test file is named `your-check-name.test.ts` in `src/scoring/checks/__tests__/`.