git:20260319.c97e02c to git:20260424.9b749cd

342 added, 90 removed. Audit A to A.

What this update changes in the checks

No check changed: every fixed check gives the same answer before and after this update.

A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.

---
name: caliber-testing
- description: Writes Vitest tests for caliber modules following project patterns. LLM calls globally mocked via src/test/setup.ts; override per-test with vi.spyOn. Use temp dirs (os.tmpdir()) for learner/storage tests, memfs for fingerprint tests. Use when user says 'write test', 'add test', 'test coverage', 'unit test'. Do NOT re-mock llmCall globally or use localStorage-like APIs directly.
+ description: Writes Vitest tests following project patterns: __tests__/ directories, vi.mock() for module mocking with vi.hoisted() for test-time factories, global LLM mock from src/test/setup.ts, environment variable save/restore in beforeEach/afterEach, vi.clearAllMocks() lifecycle, and test file organization. Use when user says 'write tests', 'add test coverage', 'test this', creates *.test.ts files, or when test failures appear in CI. Do NOT use for non-test code or for debugging without writing tests.
+ paths:
+ - src/**/__tests__/*.test.ts
---
# Caliber Testing
## Critical
- - **Global LLM mock already active**: All tests inherit `vi.mock()` from `src/test/setup.ts`. Do NOT re-mock `src/llm/index.ts` or `llmCall`. Override specific test behavior with `vi.spyOn(llmModule, 'llmCall').mockResolvedValueOnce({...})`.
- - **Imports matter**: Always import test utilities from `src/test/setup.ts` or `vitest`. Import the module under test with correct path (e.g., `import { scanLocalState } from '../scanner'`).
- - **Temp directories for I/O**: Use `os.tmpdir()` + `fs.mkdtempSync()` for learner/storage tests. Clean up with `fs.rmSync(dir, { recursive: true })`.
- - **memfs for file-tree**: Fingerprint file-tree tests use `memfs` virtual filesystem (`import { vol } from 'memfs'`). Initialize with `vol.fromJSON({...})` before each test.
- - **Type safety**: All test code must be TypeScript. Use exact return types from `src/**/*.ts` (e.g., `ConfigState`, `ScanResult`, `ErrorResponse`).
+ - All test files MUST be placed in `__tests__/` directories parallel to source files: `src/[module]/__tests__/[module].test.ts`
+ - Register any new `__tests__/` directories in `vitest.config.ts`'s `include` glob (already configured: `src/**/*.test.ts`)
+ - NEVER mock `src/test/setup.ts` — it is the global LLM provider mock already applied to all tests
+ - Environment variable tests MUST save `process.env` in `beforeEach`, restore in `afterEach`, and explicitly delete env vars to test absence
+ - Temporary file/directory cleanup MUST happen in `afterEach`, not in individual test cleanup. Use `fs.rmSync(dir, { recursive: true, force: true })`
+ - When a test requires unmocking modules mocked in global setup, call `vi.unmock('../module.js')` BEFORE the import statement
+ - Run `pnpm test` locally and `pnpm test:coverage` before committing to verify coverage thresholds (lines: 50, functions: 50, branches: 50, statements: 50)
## Instructions
- 1. **Create test file in same directory structure as source**
- - Source: `src/fingerprint/git.ts` → Test: `src/fingerprint/__tests__/git.test.ts`
- - Source: `src/commands/init.ts` → Test: `src/commands/__tests__/init.test.ts`
- - Verify no test file already exists before creating.
+ ### Step 1: Create the test file in the correct directory
+ Create `src/[module]/__tests__/[module].test.ts`. The parent source file is `src/[module]/[module].ts`.
- 2. **Import test setup and dependencies**
- ```typescript
- import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
- import { llmCall, llmJsonCall } from '../../llm';
- // Global mock from src/test/setup.ts handles llmCall already
- ```
- - For learner tests: `import * as os from 'os'; import * as fs from 'fs'`.
- - For fingerprint file-tree tests: `import { vol } from 'memfs'`.
- - Verify imports resolve without circular dependencies.
+ **Verify**: The `__tests__` directory exists at the same level as the source file being tested.
- 3. **Set up beforeEach/afterEach lifecycle**
- - **For temp dir tests**: `beforeEach(() => { testDir = fs.mkdtempSync(path.join(os.tmpdir(), 'caliber-test-')); })` and `afterEach(() => { fs.rmSync(testDir, { recursive: true }); })`.
- - **For memfs tests**: `beforeEach(() => { vol.reset(); vol.fromJSON({...}); })` and `afterEach(() => { vol.reset(); })`.
- - **For CLI/command tests**: Call `vi.clearAllMocks()` in `afterEach`.
- - Verify cleanup executes even if test fails.
+ ### Step 2: Import test framework
+ At the top of every test file, import from `vitest`:
+ ```typescript
+ import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
+ ```
- 4. **Write test case matching existing test patterns**
- - Reference `src/**/__tests__/*.test.ts` for style.
- - Test structure: `it('should [behavior] when [condition]', async () => { ... })`.
- - Assert on return values and side effects (file writes, LLM calls).
- - If testing LLM integration: `expect(llmCall).toHaveBeenCalledWith(expect.objectContaining({...}))`.
- - Verify test is deterministic (no random delays, use `vi.useFakeTimers()` if needed).
+ Add additional imports based on what you're testing:
+ - File system: `import fs from 'fs'; import path from 'path'; import os from 'os';`
+ - Temporary files: Use `fs.mkdtempSync(path.join(os.tmpdir(), 'caliber-prefix-'))`
+ - Exec: `import { execSync } from 'child_process';`
- 5. **Mock LLM responses per-test only**
- ```typescript
- vi.spyOn(llmModule, 'llmCall').mockResolvedValueOnce({
- content: [{type: 'text', text: JSON.stringify({...})}],
- usage: {input_tokens: 100, output_tokens: 50}
- });
- ```
- - Return object matching `LLMResponse` type from `src/llm/types.ts`.
- - Restore with `vi.restoreAllMocks()` in `afterEach` or use `mockResolvedValueOnce` (auto-restores).
- - Verify mock is called exactly once unless test requires multiple calls.
+ **Verify**: All required test utilities are imported before test definitions.
- 6. **For learner/storage tests**: Use temp directories and real filesystem
- - Write test data via `fs.writeFileSync(path.join(testDir, 'file.json'), JSON.stringify(data))`.
- - Read and assert: `const result = JSON.parse(fs.readFileSync(path.join(testDir, 'file.json'), 'utf-8'))`.
- - Verify test data persists and is cleaned up after test.
+ ### Step 3: Set up module mocking (if needed)
+ If testing a module that imports other modules you want to mock:
- 7. **Run test and verify coverage**
- - Execute: `pnpm test -- --run src/**/__tests__/your-test.test.ts`.
- - Check coverage: `pnpm test:coverage` (v8 reporter).
- - Verify all happy-path and error-path branches covered (aim for >80% for new tests).
- - If test fails, check setup.ts is loaded and llmCall is mocked.
+ ```typescript
+ vi.mock('../config.js', () => ({
+ loadConfig: vi.fn(),
+ writeConfigFile: vi.fn(),
+ }));
+ ```
+ For complex mocks with test-time factory functions (hoisted):
+ ```typescript
+ const { mockLoadConfig } = vi.hoisted(() => ({
+ mockLoadConfig: vi.fn(),
+ }));
+
+ vi.mock('../config.js', () => ({
+ loadConfig: () => mockLoadConfig(),
+ }));
+ ```
+
+ For unmocking global setup mocks (e.g., to test llm/index.js itself):
+ ```typescript
+ vi.unmock('../index.js');
+ ```
+
+ Place all `vi.mock()` and `vi.unmock()` calls BEFORE importing the module under test.
+
+ **Verify**: Mock declarations appear before the import of the module being tested.
+
+ ### Step 4: Organize tests in describe blocks
+ Group related tests with `describe()`:
+ ```typescript
+ describe('functionName', () => {
+ it('returns X when Y', () => {
+ // test body
+ });
+ });
+ ```
+
+ **Verify**: Each `it()` test has a clear, complete assertion.
+
+ ### Step 5: Manage environment and process state
+ For tests that modify `process.env` or `process.argv`:
+
+ ```typescript
+ describe('config tests', () => {
+ const originalEnv = process.env;
+ const originalArgv = process.argv;
+
+ beforeEach(() => {
+ process.env = { ...originalEnv }; // Copy, not reference
+ process.argv = [...originalArgv];
+ delete process.env.SPECIFIC_VAR; // Explicitly remove vars to test absence
+ });
+
+ afterEach(() => {
+ process.env = originalEnv;
+ process.argv = originalArgv;
+ });
+
+ it('tests env var behavior', () => {
+ process.env.MY_VAR = 'test';
+ // test code
+ });
+ });
+ ```
+
+ **Verify**: `beforeEach` creates a copy of env/argv; `afterEach` restores originals; unused env vars are explicitly deleted with `delete process.env.VAR`.
+
+ ### Step 6: Manage temporary files and directories
+ For file system tests, create temporary directories and clean them up:
+
+ ```typescript
+ describe('file tree', () => {
+ const dirs: string[] = [];
+
+ afterEach(() => {
+ for (const d of dirs) {
+ try { fs.rmSync(d, { recursive: true, force: true }); } catch {}
+ }
+ dirs.length = 0;
+ });
+
+ it('processes files', () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'caliber-test-'));
+ dirs.push(tmp);
+ // use tmp directory
+ });
+ });
+ ```
+
+ **Verify**: All temporary directories are pushed to a cleanup array and removed in `afterEach`.
+
+ ### Step 7: Handle mock state cleanup
+ Before each test, clear mock call history to avoid pollution between tests:
+
+ ```typescript
+ beforeEach(() => {
+ vi.clearAllMocks(); // Reset all mock call counts and return values
+ });
+
+ afterEach(() => {
+ vi.restoreAllMocks(); // Restore original implementations
+ vi.resetModules(); // Reset cached module imports (if you reload modules)
+ });
+ ```
+
+ **Verify**: `beforeEach` calls `vi.clearAllMocks()` for providers and `afterEach` calls `vi.restoreAllMocks()`.
+
+ ### Step 8: Test assertions
+ Write assertions using `expect()`. Match the patterns from existing tests:
+
+ ```typescript
+ // Simple checks
+ expect(value).toBe(expected);
+ expect(array).toContain(item);
+ expect(fn).toThrow('error message');
+
+ // Instance checks
+ expect(obj).toBeInstanceOf(ClassName);
+
+ // Mock checks
+ expect(mockFn).toHaveBeenCalledTimes(1);
+ expect(mockFn).toHaveBeenCalledWith(arg);
+
+ // File system checks
+ expect(fs.existsSync(path)).toBe(true);
+ ```
+
+ **Verify**: Each test has at least one assertion and uses appropriate `expect()` matchers.
+
+ ### Step 9: Run tests
+ Run tests locally before committing:
+ ```bash
+ pnpm test # Run all tests in watch mode
+ pnpm test:coverage # Check coverage thresholds
+ pnpm test -- src/my/path/__tests__/my.test.ts # Run single test file
+ ```
+
+ **Verify**: All tests pass and coverage thresholds are met (lines: 50%, functions: 50%, branches: 50%).
+
## Examples
- **User**: "Write a test for `src/learner/storage.ts` `saveState()` function."
+ ### Example 1: Testing a module with environment variables (config.test.ts pattern)
- **Actions**:
- 1. Create `src/learner/__tests__/storage.test.ts`.
- 2. Import `{ saveState, loadState }` from `../storage` and setup utils.
- 3. In `beforeEach`: `testDir = fs.mkdtempSync(path.join(os.tmpdir(), 'caliber-test-'))`.
- 4. Write test:
- ```typescript
- it('should save state to file', async () => {
- const state = { version: 1, data: { key: 'value' } };
- const filePath = path.join(testDir, 'state.json');
- await saveState(filePath, state);
- const loaded = JSON.parse(fs.readFileSync(filePath, 'utf-8'));
- expect(loaded).toEqual(state);
- });
- ```
- 5. In `afterEach`: `fs.rmSync(testDir, { recursive: true })`.
- 6. Run: `pnpm test -- --run src/learner/__tests__/storage.test.ts`.
+ **User asks**: "Write tests for my config loader that reads from env vars and a config file."
- **Result**: Test passes, file cleaned up, coverage +3%.
+ **Actions taken**:
+ 1. Create `src/lib/__tests__/config.test.ts`
+ 2. Import test framework and fs module
+ 3. Mock `fs` and `os`
+ 4. Save/restore `process.env` in beforeEach/afterEach
+ 5. Delete specific env vars to test absence
+ 6. Test each branch: env vars set, file exists, both, neither
- ---
+ **Result**:
+ ```typescript
+ import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
+ import fs from 'fs';
- **User**: "Add a test for fingerprint `fileTree()` with different directory structures."
+ vi.mock('fs');
+ vi.mock('os', () => ({ default: { homedir: () => '/home/user' } }));
- **Actions**:
- 1. Create `src/fingerprint/__tests__/file-tree.test.ts`.
- 2. Import `{ fileTree }` and `{ vol }` from `memfs`.
- 3. In `beforeEach`: `vol.fromJSON({ '/project/src/index.ts': 'export {}', '/project/package.json': '{}' })`.
- 4. Write test:
- ```typescript
- it('should return nested tree structure', async () => {
- const tree = await fileTree('/project', { exclude: ['node_modules'] });
- expect(tree.children).toHaveLength(2);
- expect(tree.children?.some(n => n.name === 'src')).toBe(true);
- });
- ```
- 5. Run: `pnpm test -- --run src/fingerprint/__tests__/file-tree.test.ts`.
+ import { loadConfig } from '../config.js';
- **Result**: Test passes with memfs-backed virtual filesystem, no I/O side effects.
+ describe('config', () => {
+ const originalEnv = process.env;
+ beforeEach(() => {
+ vi.clearAllMocks();
+ process.env = { ...originalEnv };
+ delete process.env.ANTHROPIC_API_KEY;
+ delete process.env.OPENAI_API_KEY;
+ });
+
+ afterEach(() => {
+ process.env = originalEnv;
+ });
+
+ it('returns env config when ANTHROPIC_API_KEY is set', () => {
+ process.env.ANTHROPIC_API_KEY = 'sk-ant-test';
+ const config = loadConfig();
+ expect(config?.provider).toBe('anthropic');
+ });
+
+ it('returns null when no env vars set', () => {
+ expect(loadConfig()).toBeNull();
+ });
+ });
+ ```
+
+ ### Example 2: Testing file system operations with temp cleanup (file-tree.test.ts pattern)
+
+ **User asks**: "Write tests for file tree analysis."
+
+ **Actions taken**:
+ 1. Create `src/fingerprint/__tests__/file-tree.test.ts`
+ 2. Set up a cleanup array for temp directories
+ 3. Create temp dirs in each test and push to cleanup array
+ 4. Clean up all temp dirs in afterEach
+ 5. Test with various file mtimes and directory structures
+
+ **Result**:
+ ```typescript
+ import { describe, it, expect, afterEach } from 'vitest';
+ import fs from 'fs';
+ import path from 'path';
+ import os from 'os';
+ import { getFileTree } from '../file-tree.js';
+
+ const dirs: string[] = [];
+ afterEach(() => {
+ for (const d of dirs) {
+ try { fs.rmSync(d, { recursive: true, force: true }); } catch {}
+ }
+ dirs.length = 0;
+ });
+
+ describe('getFileTree', () => {
+ it('returns files sorted by mtime descending', () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'caliber-ft-'));
+ dirs.push(tmp);
+
+ fs.writeFileSync(path.join(tmp, 'a.ts'), 'a');
+ fs.writeFileSync(path.join(tmp, 'b.ts'), 'b');
+ const tree = getFileTree(tmp);
+ expect(tree).toHaveLength(2);
+ });
+ });
+ ```
+
+ ### Example 3: Testing with module mocking and factory functions (index.test.ts pattern)
+
+ **User asks**: "Write tests for the provider factory that instantiates different providers based on config."
+
+ **Actions taken**:
+ 1. Create `src/llm/__tests__/index.test.ts`
+ 2. Use `vi.unmock('../index.js')` to test the real module
+ 3. Create mock classes in `vi.hoisted()`
+ 4. Mock dependencies (config, provider classes)
+ 5. Test provider selection logic
+ 6. Use `resetProvider()` between tests to clear cached instances
+
+ **Result**:
+ ```typescript
+ import { describe, it, expect, vi, beforeEach } from 'vitest';
+
+ vi.unmock('../index.js');
+
+ const { mockLoadConfig, MockAnthropicProvider } = vi.hoisted(() => {
+ class MockAnthropicProvider {
+ config: unknown;
+ call = vi.fn();
+ constructor(c: unknown) { this.config = c; }
+ }
+ return {
+ mockLoadConfig: vi.fn(),
+ MockAnthropicProvider,
+ };
+ });
+
+ vi.mock('../config.js', () => ({
+ loadConfig: () => mockLoadConfig(),
+ }));
+
+ vi.mock('../anthropic.js', () => ({
+ AnthropicProvider: MockAnthropicProvider,
+ }));
+
+ import { getProvider, resetProvider } from '../index.js';
+
+ describe('getProvider', () => {
+ beforeEach(() => {
+ vi.clearAllMocks();
+ resetProvider();
+ });
+
+ it('creates AnthropicProvider for anthropic config', () => {
+ mockLoadConfig.mockReturnValue({
+ provider: 'anthropic',
+ model: 'claude-sonnet-4-6',
+ apiKey: 'sk-test',
+ });
+ const provider = getProvider();
+ expect(provider).toBeInstanceOf(MockAnthropicProvider);
+ });
+ });
+ ```
+
## Common Issues
- - **"Cannot find module 'src/test/setup'"** → Verify `src/test/setup.ts` exists and exports test utilities. Check path is relative (e.g., `import { llmCall } from '../../llm'` from `src/commands/__tests__/`).
- - **"llmCall is not mocked" or test calls real API** → Global mock in `setup.ts` may not be loaded. Add `vi.mock('../../llm')` at top of test file OR verify test runs after setup loads. Check vitest config includes `setupFiles: ['src/test/setup.ts']`.
- - **Test hangs or times out** → Likely async/await missing. Ensure test function is `async` and all promises are `await`ed. Use `vi.useFakeTimers()` for setTimeout tests.
- - **memfs vol.reset() doesn't clear filesystem** → Call `vol.reset()` in `beforeEach` BEFORE `vol.fromJSON()`. Verify no other test modifies vol concurrently.
- - **Temp directory not cleaned up (test pollutes filesystem)** → Missing `fs.rmSync(testDir, { recursive: true })` in `afterEach`. Wrap in try-finally if needed.
- - **"Cannot destructure llmCall from mocked module"** → Global mock returns object with `llmCall` property. Import as `import * as llmModule from '../../llm'` then use `llmModule.llmCall`.
- - **Type errors on mock return** → Verify return shape matches `LLMResponse` type: `{ content: Array<{type: string, text: string}>, usage: {input_tokens: number, output_tokens: number} }`.
+ **"Cannot find module" when running tests**
+ - **Cause**: `vi.mock()` called after the import statement
+ - **Fix**: Move all `vi.mock()` and `vi.unmock()` calls to the TOP of the file, before any `import` statements
+
+ **Environment variable persists across tests**
+ - **Cause**: `beforeEach` assigns by reference instead of copying: `process.env = originalEnv`
+ - **Fix**: Create a copy in `beforeEach`: `process.env = { ...originalEnv }; delete process.env.VAR`
+
+ **Temporary files not cleaned up, filling disk**
+ - **Cause**: `afterEach` not called or temp paths not tracked
+ - **Fix**: Use centralized cleanup array: `dirs: string[] = []` pushed to in tests, cleaned in `afterEach` with `fs.rmSync(..., { recursive: true, force: true })`
+
+ **Mock return value from previous test bleeds into next test**
+ - **Cause**: `vi.clearAllMocks()` not called in `beforeEach`
+ - **Fix**: Add `vi.clearAllMocks()` as first statement in `beforeEach`
+
+ **"vi.mocked() is not a function" when accessing mock calls**
+ - **Cause**: Trying to use `vi.mocked()` on a non-mocked module
+ - **Fix**: Ensure the module is mocked with `vi.mock()` before importing, then cast: `const mockFn = vi.mocked(importedFn)`
+
+ **Test passes locally but fails in CI**
+ - **Cause**: Tests rely on real file system or network (not mocked)
+ - **Fix**: Check if temp files are being created in real directories instead of mocked ones. Verify all external calls use `vi.mock()` for fs/http/exec
+
+ **Coverage threshold failures: "lines not covered", "statements not covered"**
+ - **Cause**: Branches or error paths not tested
+ - **Fix**: Run `pnpm test:coverage` locally to see untested lines. Add `it()` tests for error cases, edge conditions, and branches that return different values
+ - **Example**: If `if (x) return 'a'; else return 'b';` has 0% branch coverage, add tests: one with x=true, one with x=false
+
+ **"expected 1 error but got 0" when testing error throws**
+ - **Cause**: Error not actually thrown, or thrown asynchronously
+ - **Fix**: For sync functions: `expect(() => fn()).toThrow('message')`. For async: `expect(async () => { await fn() }).rejects.toThrow('message')` or use `await expect(promise).rejects.toThrow()`
+
+ **Mock factory returns undefined**
+ - **Cause**: `vi.hoisted()` variables used before definition in the same block
+ - **Fix**: Ensure `vi.hoisted()` block returns all factories, and `vi.mock()` blocks use them after the hoisted definition
+
+ **Test flakes (sometimes passes, sometimes fails)**
+ - **Cause**: Timing issues or file system race conditions in cleanup
+ - **Fix**: For file cleanup, use `{ force: true }` in `rmSync`. For timing, avoid `setTimeout`; use `vi.useFakeTimers()` and `vi.runAllTimers()` if needed. Check for leftover files from previous test runs in `beforeEach`