git:20260425.fa623ff to git:20260428.99f2785

339 added, 95 removed. Audit A to A.

What this update changes in the checks

No check changed: every fixed check gives the same answer before and after this update.

A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.

---
name: caliber-testing
- description: Writes Vitest tests following project patterns: __tests__/ dirs, vi.mock() for modules, global LLM mock from src/test/setup.ts, env var save/restore, and describe/it blocks. Use when user says 'write tests', 'add tests', 'test this', or creates *.test.ts files. Do NOT use for non-test code or documentation.
+ description: Writes Vitest tests following project patterns: __tests__/ directories, vi.mock() for module mocking with vi.hoisted() for test-time factories, global LLM mock from src/test/setup.ts, environment variable save/restore in beforeEach/afterEach, vi.clearAllMocks() lifecycle, and test file organization. Use when user says 'write tests', 'add test coverage', 'test this', creates *.test.ts files, or when test failures appear in CI. Do NOT use for non-test code or for debugging without writing tests.
---
# Caliber Testing
## Critical
- - **Test location**: All tests live in `__tests__/` directories at the same level as the code being tested. File names: `<module>.test.ts`.
- - **Global LLM mock**: `src/test/setup.ts` automatically mocks `@anthropic-ai/sdk`, `@anthropic-ai/vertex-sdk`, and `openai`. Do NOT re-mock these in individual tests—use the global mock.
- - **Env var isolation**: Save and restore `process.env` in `beforeEach` / `afterEach`. See example below.
- - **Module mocks**: Use `vi.mock()` at the top of test files with `{ spy: true }` or define default behavior. Always import the real module and use `vi.getMocked()` to assert on calls.
+ - All test files MUST be placed in `__tests__/` directories parallel to source files: `src/[module]/__tests__/[module].test.ts`
+ - Register any new `__tests__/` directories in `vitest.config.ts`'s `include` glob (already configured: `src/**/*.test.ts`)
+ - NEVER mock `src/test/setup.ts` — it is the global LLM provider mock already applied to all tests
+ - Environment variable tests MUST save `process.env` in `beforeEach`, restore in `afterEach`, and explicitly delete env vars to test absence
+ - Temporary file/directory cleanup MUST happen in `afterEach`, not in individual test cleanup. Use `fs.rmSync(dir, { recursive: true, force: true })`
+ - When a test requires unmocking modules mocked in global setup, call `vi.unmock('../module.js')` BEFORE the import statement
+ - Run `pnpm test` locally and `pnpm test:coverage` before committing to verify coverage thresholds (lines: 50, functions: 50, branches: 50, statements: 50)
## Instructions
- 1. **Create test file in `__tests__/`**
- - Path: `src/<feature>/__tests__/<module>.test.ts`
- - Verify the directory exists; create if needed.
+ ### Step 1: Create the test file in the correct directory
+ Create `src/[module]/__tests__/[module].test.ts`. The parent source file is `src/[module]/[module].ts`.
- 2. **Import testing utilities and the module under test**
- ```typescript
- import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
- import { functionToTest } from '../index.js';
- ```
- - Note: Use `.js` import extensions (ESM convention in this project).
+ **Verify**: The `__tests__` directory exists at the same level as the source file being tested.
- 3. **Save/restore env vars if your code reads `process.env`**
- ```typescript
- let originalEnv: NodeJS.ProcessEnv;
- beforeEach(() => {
- originalEnv = { ...process.env };
- });
- afterEach(() => {
- process.env = originalEnv;
- });
- ```
- - Verify this pattern is in place before running tests.
+ ### Step 2: Import test framework
+ At the top of every test file, import from `vitest`:
+ ```typescript
+ import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
+ ```
- 4. **Mock external modules using `vi.mock()` at file top (if needed)**
- ```typescript
- vi.mock('../dependency.js', () => ({
- dependencyFunc: vi.fn().mockResolvedValue({ data: 'test' }),
- }));
- ```
- - Do NOT mock LLM libraries (`@anthropic-ai/sdk`, `openai`, `@anthropic-ai/vertex-sdk`)—the global setup handles them.
- - Use `{ spy: true }` to spy on the original implementation if needed.
+ Add additional imports based on what you're testing:
+ - File system: `import fs from 'fs'; import path from 'path'; import os from 'os';`
+ - Temporary files: Use `fs.mkdtempSync(path.join(os.tmpdir(), 'caliber-prefix-'))`
+ - Exec: `import { execSync } from 'child_process';`
- 5. **Write test cases using `describe` and `it`**
- ```typescript
- describe('functionToTest', () => {
- it('should return expected value when input is valid', () => {
- const result = functionToTest('input');
- expect(result).toBe('expected');
- });
+ **Verify**: All required test utilities are imported before test definitions.
- it('should throw error when input is invalid', () => {
- expect(() => functionToTest(null)).toThrow('Invalid input');
- });
- });
- ```
- - Verify test names are descriptive and follow "should..." convention.
+ ### Step 3: Set up module mocking (if needed)
+ If testing a module that imports other modules you want to mock:
- 6. **For async code, use `async/await` in test functions**
- ```typescript
- it('should fetch data', async () => {
- const result = await fetchData();
- expect(result).toBeDefined();
- });
- ```
- - Verify the test waits for all promises to resolve before assertions.
+ ```typescript
+ vi.mock('../config.js', () => ({
+ loadConfig: vi.fn(),
+ writeConfigFile: vi.fn(),
+ }));
+ ```
- 7. **Run tests and verify all pass**
- ```bash
- npm run test -- src/<feature>/__tests__/<module>.test.ts
- ```
- - Or for the whole feature: `npm run test -- src/<feature>/`
- - Verify exit code is 0 and all assertions pass.
+ For complex mocks with test-time factory functions (hoisted):
+ ```typescript
+ const { mockLoadConfig } = vi.hoisted(() => ({
+ mockLoadConfig: vi.fn(),
+ }));
+ vi.mock('../config.js', () => ({
+ loadConfig: () => mockLoadConfig(),
+ }));
+ ```
+
+ For unmocking global setup mocks (e.g., to test llm/index.js itself):
+ ```typescript
+ vi.unmock('../index.js');
+ ```
+
+ Place all `vi.mock()` and `vi.unmock()` calls BEFORE importing the module under test.
+
+ **Verify**: Mock declarations appear before the import of the module being tested.
+
+ ### Step 4: Organize tests in describe blocks
+ Group related tests with `describe()`:
+ ```typescript
+ describe('functionName', () => {
+ it('returns X when Y', () => {
+ // test body
+ });
+ });
+ ```
+
+ **Verify**: Each `it()` test has a clear, complete assertion.
+
+ ### Step 5: Manage environment and process state
+ For tests that modify `process.env` or `process.argv`:
+
+ ```typescript
+ describe('config tests', () => {
+ const originalEnv = process.env;
+ const originalArgv = process.argv;
+
+ beforeEach(() => {
+ process.env = { ...originalEnv }; // Copy, not reference
+ process.argv = [...originalArgv];
+ delete process.env.SPECIFIC_VAR; // Explicitly remove vars to test absence
+ });
+
+ afterEach(() => {
+ process.env = originalEnv;
+ process.argv = originalArgv;
+ });
+
+ it('tests env var behavior', () => {
+ process.env.MY_VAR = 'test';
+ // test code
+ });
+ });
+ ```
+
+ **Verify**: `beforeEach` creates a copy of env/argv; `afterEach` restores originals; unused env vars are explicitly deleted with `delete process.env.VAR`.
+
+ ### Step 6: Manage temporary files and directories
+ For file system tests, create temporary directories and clean them up:
+
+ ```typescript
+ describe('file tree', () => {
+ const dirs: string[] = [];
+
+ afterEach(() => {
+ for (const d of dirs) {
+ try { fs.rmSync(d, { recursive: true, force: true }); } catch {}
+ }
+ dirs.length = 0;
+ });
+
+ it('processes files', () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'caliber-test-'));
+ dirs.push(tmp);
+ // use tmp directory
+ });
+ });
+ ```
+
+ **Verify**: All temporary directories are pushed to a cleanup array and removed in `afterEach`.
+
+ ### Step 7: Handle mock state cleanup
+ Before each test, clear mock call history to avoid pollution between tests:
+
+ ```typescript
+ beforeEach(() => {
+ vi.clearAllMocks(); // Reset all mock call counts and return values
+ });
+
+ afterEach(() => {
+ vi.restoreAllMocks(); // Restore original implementations
+ vi.resetModules(); // Reset cached module imports (if you reload modules)
+ });
+ ```
+
+ **Verify**: `beforeEach` calls `vi.clearAllMocks()` for providers and `afterEach` calls `vi.restoreAllMocks()`.
+
+ ### Step 8: Test assertions
+ Write assertions using `expect()`. Match the patterns from existing tests:
+
+ ```typescript
+ // Simple checks
+ expect(value).toBe(expected);
+ expect(array).toContain(item);
+ expect(fn).toThrow('error message');
+
+ // Instance checks
+ expect(obj).toBeInstanceOf(ClassName);
+
+ // Mock checks
+ expect(mockFn).toHaveBeenCalledTimes(1);
+ expect(mockFn).toHaveBeenCalledWith(arg);
+
+ // File system checks
+ expect(fs.existsSync(path)).toBe(true);
+ ```
+
+ **Verify**: Each test has at least one assertion and uses appropriate `expect()` matchers.
+
+ ### Step 9: Run tests
+ Run tests locally before committing:
+ ```bash
+ pnpm test # Run all tests in watch mode
+ pnpm test:coverage # Check coverage thresholds
+ pnpm test -- src/my/path/__tests__/my.test.ts # Run single test file
+ ```
+
+ **Verify**: All tests pass and coverage thresholds are met (lines: 50%, functions: 50%, branches: 50%).
+
## Examples
- **User says**: "Write tests for the `detectLanguage` function in `src/fingerprint/code-analysis.ts`."
+ ### Example 1: Testing a module with environment variables (config.test.ts pattern)
- **Actions**:
- 1. Create `src/fingerprint/__tests__/code-analysis.test.ts`.
- 2. Import `detectLanguage` and Vitest utilities.
- 3. Mock file I/O if needed: `vi.mock('fs/promises')` (not LLM).
- 4. Write test cases:
- ```typescript
- import { describe, it, expect, vi } from 'vitest';
- import { detectLanguage } from '../code-analysis.js';
+ **User asks**: "Write tests for my config loader that reads from env vars and a config file."
- vi.mock('fs/promises', () => ({
- readFile: vi.fn(),
- }));
+ **Actions taken**:
+ 1. Create `src/lib/__tests__/config.test.ts`
+ 2. Import test framework and fs module
+ 3. Mock `fs` and `os`
+ 4. Save/restore `process.env` in beforeEach/afterEach
+ 5. Delete specific env vars to test absence
+ 6. Test each branch: env vars set, file exists, both, neither
- describe('detectLanguage', () => {
- it('should detect TypeScript from file extension', () => {
- const lang = detectLanguage('foo.ts');
- expect(lang).toBe('typescript');
- });
+ **Result**:
+ ```typescript
+ import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
+ import fs from 'fs';
- it('should detect Python from file extension', () => {
- const lang = detectLanguage('bar.py');
- expect(lang).toBe('python');
- });
+ vi.mock('fs');
+ vi.mock('os', () => ({ default: { homedir: () => '/home/user' } }));
- it('should return unknown for unrecognized extension', () => {
- const lang = detectLanguage('data.xyz');
- expect(lang).toBe('unknown');
- });
- });
- ```
- 5. Run: `npm run test -- src/fingerprint/__tests__/code-analysis.test.ts`
+ import { loadConfig } from '../config.js';
- **Result**: Tests pass, file is committed, patterns replicated in future tests.
+ describe('config', () => {
+ const originalEnv = process.env;
+ beforeEach(() => {
+ vi.clearAllMocks();
+ process.env = { ...originalEnv };
+ delete process.env.ANTHROPIC_API_KEY;
+ delete process.env.OPENAI_API_KEY;
+ });
+
+ afterEach(() => {
+ process.env = originalEnv;
+ });
+
+ it('returns env config when ANTHROPIC_API_KEY is set', () => {
+ process.env.ANTHROPIC_API_KEY = 'sk-ant-test';
+ const config = loadConfig();
+ expect(config?.provider).toBe('anthropic');
+ });
+
+ it('returns null when no env vars set', () => {
+ expect(loadConfig()).toBeNull();
+ });
+ });
+ ```
+
+ ### Example 2: Testing file system operations with temp cleanup (file-tree.test.ts pattern)
+
+ **User asks**: "Write tests for file tree analysis."
+
+ **Actions taken**:
+ 1. Create `src/fingerprint/__tests__/file-tree.test.ts`
+ 2. Set up a cleanup array for temp directories
+ 3. Create temp dirs in each test and push to cleanup array
+ 4. Clean up all temp dirs in afterEach
+ 5. Test with various file mtimes and directory structures
+
+ **Result**:
+ ```typescript
+ import { describe, it, expect, afterEach } from 'vitest';
+ import fs from 'fs';
+ import path from 'path';
+ import os from 'os';
+ import { getFileTree } from '../file-tree.js';
+
+ const dirs: string[] = [];
+ afterEach(() => {
+ for (const d of dirs) {
+ try { fs.rmSync(d, { recursive: true, force: true }); } catch {}
+ }
+ dirs.length = 0;
+ });
+
+ describe('getFileTree', () => {
+ it('returns files sorted by mtime descending', () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'caliber-ft-'));
+ dirs.push(tmp);
+
+ fs.writeFileSync(path.join(tmp, 'a.ts'), 'a');
+ fs.writeFileSync(path.join(tmp, 'b.ts'), 'b');
+ const tree = getFileTree(tmp);
+ expect(tree).toHaveLength(2);
+ });
+ });
+ ```
+
+ ### Example 3: Testing with module mocking and factory functions (index.test.ts pattern)
+
+ **User asks**: "Write tests for the provider factory that instantiates different providers based on config."
+
+ **Actions taken**:
+ 1. Create `src/llm/__tests__/index.test.ts`
+ 2. Use `vi.unmock('../index.js')` to test the real module
+ 3. Create mock classes in `vi.hoisted()`
+ 4. Mock dependencies (config, provider classes)
+ 5. Test provider selection logic
+ 6. Use `resetProvider()` between tests to clear cached instances
+
+ **Result**:
+ ```typescript
+ import { describe, it, expect, vi, beforeEach } from 'vitest';
+
+ vi.unmock('../index.js');
+
+ const { mockLoadConfig, MockAnthropicProvider } = vi.hoisted(() => {
+ class MockAnthropicProvider {
+ config: unknown;
+ call = vi.fn();
+ constructor(c: unknown) { this.config = c; }
+ }
+ return {
+ mockLoadConfig: vi.fn(),
+ MockAnthropicProvider,
+ };
+ });
+
+ vi.mock('../config.js', () => ({
+ loadConfig: () => mockLoadConfig(),
+ }));
+
+ vi.mock('../anthropic.js', () => ({
+ AnthropicProvider: MockAnthropicProvider,
+ }));
+
+ import { getProvider, resetProvider } from '../index.js';
+
+ describe('getProvider', () => {
+ beforeEach(() => {
+ vi.clearAllMocks();
+ resetProvider();
+ });
+
+ it('creates AnthropicProvider for anthropic config', () => {
+ mockLoadConfig.mockReturnValue({
+ provider: 'anthropic',
+ model: 'claude-sonnet-4-6',
+ apiKey: 'sk-test',
+ });
+ const provider = getProvider();
+ expect(provider).toBeInstanceOf(MockAnthropicProvider);
+ });
+ });
+ ```
+
## Common Issues
- - **"Cannot find module @anthropic-ai/sdk"**: This means `src/test/setup.ts` was not loaded. Verify `vitest.config.ts` has `setupFiles: ['./src/test/setup.ts']` in the config. Check the vitest.config.ts file for correct setup.
- - **"ReferenceError: vi is not defined"**: Add `import { vi } from 'vitest'` at the top of the test file.
- - **"Module not found: src/llm/anthropic.js"**: Check the import path uses `.js` extension (ESM convention). Verify file exists at `src/llm/anthropic.ts` → import as `./anthropic.js`.
- - **"Timeout: async test did not finish"**: Add `async/await` to test function and ensure all promises are awaited. Example: `it('test', async () => { await fn(); expect(...); });`
- - **"process.env is dirty after test"**: Verify `beforeEach/afterEach` are saving and restoring `process.env`. Add: `beforeEach(() => { originalEnv = { ...process.env }; }); afterEach(() => { process.env = originalEnv; });`
- - **"Mock is not being called"**: Use `vi.getMocked(module).functionName` to assert on mocked function calls. Example: `expect(vi.getMocked(fs).readFile).toHaveBeenCalledWith('path')`.
+ **"Cannot find module" when running tests**
+ - **Cause**: `vi.mock()` called after the import statement
+ - **Fix**: Move all `vi.mock()` and `vi.unmock()` calls to the TOP of the file, before any `import` statements
+
+ **Environment variable persists across tests**
+ - **Cause**: `beforeEach` assigns by reference instead of copying: `process.env = originalEnv`
+ - **Fix**: Create a copy in `beforeEach`: `process.env = { ...originalEnv }; delete process.env.VAR`
+
+ **Temporary files not cleaned up, filling disk**
+ - **Cause**: `afterEach` not called or temp paths not tracked
+ - **Fix**: Use centralized cleanup array: `dirs: string[] = []` pushed to in tests, cleaned in `afterEach` with `fs.rmSync(..., { recursive: true, force: true })`
+
+ **Mock return value from previous test bleeds into next test**
+ - **Cause**: `vi.clearAllMocks()` not called in `beforeEach`
+ - **Fix**: Add `vi.clearAllMocks()` as first statement in `beforeEach`
+
+ **"vi.mocked() is not a function" when accessing mock calls**
+ - **Cause**: Trying to use `vi.mocked()` on a non-mocked module
+ - **Fix**: Ensure the module is mocked with `vi.mock()` before importing, then cast: `const mockFn = vi.mocked(importedFn)`
+
+ **Test passes locally but fails in CI**
+ - **Cause**: Tests rely on real file system or network (not mocked)
+ - **Fix**: Check if temp files are being created in real directories instead of mocked ones. Verify all external calls use `vi.mock()` for fs/http/exec
+
+ **Coverage threshold failures: "lines not covered", "statements not covered"**
+ - **Cause**: Branches or error paths not tested
+ - **Fix**: Run `pnpm test:coverage` locally to see untested lines. Add `it()` tests for error cases, edge conditions, and branches that return different values
+ - **Example**: If `if (x) return 'a'; else return 'b';` has 0% branch coverage, add tests: one with x=true, one with x=false
+
+ **"expected 1 error but got 0" when testing error throws**
+ - **Cause**: Error not actually thrown, or thrown asynchronously
+ - **Fix**: For sync functions: `expect(() => fn()).toThrow('message')`. For async: `expect(async () => { await fn() }).rejects.toThrow('message')` or use `await expect(promise).rejects.toThrow()`
+
+ **Mock factory returns undefined**
+ - **Cause**: `vi.hoisted()` variables used before definition in the same block
+ - **Fix**: Ensure `vi.hoisted()` block returns all factories, and `vi.mock()` blocks use them after the hoisted definition
+
+ **Test flakes (sometimes passes, sometimes fails)**
+ - **Cause**: Timing issues or file system race conditions in cleanup
+ - **Fix**: For file cleanup, use `{ force: true }` in `rmSync`. For timing, avoid `setTimeout`; use `vi.useFakeTimers()` and `vi.runAllTimers()` if needed. Check for leftover files from previous test runs in `beforeEach`