caliber-testing · diff
git:20260424.9b749cd to git:20260425.fa623ff
95 added, 341 removed. Audit A to A.
What this update changes in the checks
No check changed: every fixed check gives the same answer before and after this update.
A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.
---
name: caliber-testing
- description: Writes Vitest tests following project patterns: __tests__/ directories, vi.mock() for module mocking with vi.hoisted() for test-time factories, global LLM mock from src/test/setup.ts, environment variable save/restore in beforeEach/afterEach, vi.clearAllMocks() lifecycle, and test file organization. Use when user says 'write tests', 'add test coverage', 'test this', creates *.test.ts files, or when test failures appear in CI. Do NOT use for non-test code or for debugging without writing tests.
- paths:
- - src/**/__tests__/*.test.ts
+ description: Writes Vitest tests following project patterns: __tests__/ dirs, vi.mock() for modules, global LLM mock from src/test/setup.ts, env var save/restore, and describe/it blocks. Use when user says 'write tests', 'add tests', 'test this', or creates *.test.ts files. Do NOT use for non-test code or documentation.
---
# Caliber Testing
## Critical
- - All test files MUST be placed in `__tests__/` directories parallel to source files: `src/[module]/__tests__/[module].test.ts`
- - Register any new `__tests__/` directories in `vitest.config.ts`'s `include` glob (already configured: `src/**/*.test.ts`)
- - NEVER mock `src/test/setup.ts` — it is the global LLM provider mock already applied to all tests
- - Environment variable tests MUST save `process.env` in `beforeEach`, restore in `afterEach`, and explicitly delete env vars to test absence
- - Temporary file/directory cleanup MUST happen in `afterEach`, not in individual test cleanup. Use `fs.rmSync(dir, { recursive: true, force: true })`
- - When a test requires unmocking modules mocked in global setup, call `vi.unmock('../module.js')` BEFORE the import statement
- - Run `pnpm test` locally and `pnpm test:coverage` before committing to verify coverage thresholds (lines: 50, functions: 50, branches: 50, statements: 50)
+ - **Test location**: All tests live in `__tests__/` directories at the same level as the code being tested. File names: `<module>.test.ts`.
+ - **Global LLM mock**: `src/test/setup.ts` automatically mocks `@anthropic-ai/sdk`, `@anthropic-ai/vertex-sdk`, and `openai`. Do NOT re-mock these in individual tests—use the global mock.
+ - **Env var isolation**: Save and restore `process.env` in `beforeEach` / `afterEach`. See example below.
+ - **Module mocks**: Use `vi.mock()` at the top of test files with `{ spy: true }` or define default behavior. Always import the real module and use `vi.getMocked()` to assert on calls.
## Instructions
- ### Step 1: Create the test file in the correct directory
- Create `src/[module]/__tests__/[module].test.ts`. The parent source file is `src/[module]/[module].ts`.
-
- **Verify**: The `__tests__` directory exists at the same level as the source file being tested.
-
- ### Step 2: Import test framework
- At the top of every test file, import from `vitest`:
- ```typescript
- import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
- ```
-
- Add additional imports based on what you're testing:
- - File system: `import fs from 'fs'; import path from 'path'; import os from 'os';`
- - Temporary files: Use `fs.mkdtempSync(path.join(os.tmpdir(), 'caliber-prefix-'))`
- - Exec: `import { execSync } from 'child_process';`
-
- **Verify**: All required test utilities are imported before test definitions.
-
- ### Step 3: Set up module mocking (if needed)
- If testing a module that imports other modules you want to mock:
-
- ```typescript
- vi.mock('../config.js', () => ({
- loadConfig: vi.fn(),
- writeConfigFile: vi.fn(),
- }));
- ```
-
- For complex mocks with test-time factory functions (hoisted):
- ```typescript
- const { mockLoadConfig } = vi.hoisted(() => ({
- mockLoadConfig: vi.fn(),
- }));
-
- vi.mock('../config.js', () => ({
- loadConfig: () => mockLoadConfig(),
- }));
- ```
-
- For unmocking global setup mocks (e.g., to test llm/index.js itself):
- ```typescript
- vi.unmock('../index.js');
- ```
-
- Place all `vi.mock()` and `vi.unmock()` calls BEFORE importing the module under test.
-
- **Verify**: Mock declarations appear before the import of the module being tested.
-
- ### Step 4: Organize tests in describe blocks
- Group related tests with `describe()`:
- ```typescript
- describe('functionName', () => {
- it('returns X when Y', () => {
- // test body
- });
- });
- ```
-
- **Verify**: Each `it()` test has a clear, complete assertion.
-
- ### Step 5: Manage environment and process state
- For tests that modify `process.env` or `process.argv`:
-
- ```typescript
- describe('config tests', () => {
- const originalEnv = process.env;
- const originalArgv = process.argv;
-
- beforeEach(() => {
- process.env = { ...originalEnv }; // Copy, not reference
- process.argv = [...originalArgv];
- delete process.env.SPECIFIC_VAR; // Explicitly remove vars to test absence
- });
-
- afterEach(() => {
- process.env = originalEnv;
- process.argv = originalArgv;
- });
-
- it('tests env var behavior', () => {
- process.env.MY_VAR = 'test';
- // test code
- });
- });
- ```
-
- **Verify**: `beforeEach` creates a copy of env/argv; `afterEach` restores originals; unused env vars are explicitly deleted with `delete process.env.VAR`.
-
- ### Step 6: Manage temporary files and directories
- For file system tests, create temporary directories and clean them up:
-
- ```typescript
- describe('file tree', () => {
- const dirs: string[] = [];
-
- afterEach(() => {
- for (const d of dirs) {
- try { fs.rmSync(d, { recursive: true, force: true }); } catch {}
- }
- dirs.length = 0;
- });
-
- it('processes files', () => {
- const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'caliber-test-'));
- dirs.push(tmp);
- // use tmp directory
- });
- });
- ```
-
- **Verify**: All temporary directories are pushed to a cleanup array and removed in `afterEach`.
-
- ### Step 7: Handle mock state cleanup
- Before each test, clear mock call history to avoid pollution between tests:
-
- ```typescript
- beforeEach(() => {
- vi.clearAllMocks(); // Reset all mock call counts and return values
- });
-
- afterEach(() => {
- vi.restoreAllMocks(); // Restore original implementations
- vi.resetModules(); // Reset cached module imports (if you reload modules)
- });
- ```
-
- **Verify**: `beforeEach` calls `vi.clearAllMocks()` for providers and `afterEach` calls `vi.restoreAllMocks()`.
-
- ### Step 8: Test assertions
- Write assertions using `expect()`. Match the patterns from existing tests:
+ 1. **Create test file in `__tests__/`**
+ - Path: `src/<feature>/__tests__/<module>.test.ts`
+ - Verify the directory exists; create if needed.
- ```typescript
- // Simple checks
- expect(value).toBe(expected);
- expect(array).toContain(item);
- expect(fn).toThrow('error message');
+ 2. **Import testing utilities and the module under test**
+ ```typescript
+ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
+ import { functionToTest } from '../index.js';
+ ```
+ - Note: Use `.js` import extensions (ESM convention in this project).
- // Instance checks
- expect(obj).toBeInstanceOf(ClassName);
+ 3. **Save/restore env vars if your code reads `process.env`**
+ ```typescript
+ let originalEnv: NodeJS.ProcessEnv;
+ beforeEach(() => {
+ originalEnv = { ...process.env };
+ });
+ afterEach(() => {
+ process.env = originalEnv;
+ });
+ ```
+ - Verify this pattern is in place before running tests.
- // Mock checks
- expect(mockFn).toHaveBeenCalledTimes(1);
- expect(mockFn).toHaveBeenCalledWith(arg);
+ 4. **Mock external modules using `vi.mock()` at file top (if needed)**
+ ```typescript
+ vi.mock('../dependency.js', () => ({
+ dependencyFunc: vi.fn().mockResolvedValue({ data: 'test' }),
+ }));
+ ```
+ - Do NOT mock LLM libraries (`@anthropic-ai/sdk`, `openai`, `@anthropic-ai/vertex-sdk`)—the global setup handles them.
+ - Use `{ spy: true }` to spy on the original implementation if needed.
- // File system checks
- expect(fs.existsSync(path)).toBe(true);
- ```
+ 5. **Write test cases using `describe` and `it`**
+ ```typescript
+ describe('functionToTest', () => {
+ it('should return expected value when input is valid', () => {
+ const result = functionToTest('input');
+ expect(result).toBe('expected');
+ });
- **Verify**: Each test has at least one assertion and uses appropriate `expect()` matchers.
+ it('should throw error when input is invalid', () => {
+ expect(() => functionToTest(null)).toThrow('Invalid input');
+ });
+ });
+ ```
+ - Verify test names are descriptive and follow "should..." convention.
- ### Step 9: Run tests
- Run tests locally before committing:
- ```bash
- pnpm test # Run all tests in watch mode
- pnpm test:coverage # Check coverage thresholds
- pnpm test -- src/my/path/__tests__/my.test.ts # Run single test file
- ```
+ 6. **For async code, use `async/await` in test functions**
+ ```typescript
+ it('should fetch data', async () => {
+ const result = await fetchData();
+ expect(result).toBeDefined();
+ });
+ ```
+ - Verify the test waits for all promises to resolve before assertions.
- **Verify**: All tests pass and coverage thresholds are met (lines: 50%, functions: 50%, branches: 50%).
+ 7. **Run tests and verify all pass**
+ ```bash
+ npm run test -- src/<feature>/__tests__/<module>.test.ts
+ ```
+ - Or for the whole feature: `npm run test -- src/<feature>/`
+ - Verify exit code is 0 and all assertions pass.
## Examples
- ### Example 1: Testing a module with environment variables (config.test.ts pattern)
-
- **User asks**: "Write tests for my config loader that reads from env vars and a config file."
-
- **Actions taken**:
- 1. Create `src/lib/__tests__/config.test.ts`
- 2. Import test framework and fs module
- 3. Mock `fs` and `os`
- 4. Save/restore `process.env` in beforeEach/afterEach
- 5. Delete specific env vars to test absence
- 6. Test each branch: env vars set, file exists, both, neither
-
- **Result**:
- ```typescript
- import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
- import fs from 'fs';
-
- vi.mock('fs');
- vi.mock('os', () => ({ default: { homedir: () => '/home/user' } }));
-
- import { loadConfig } from '../config.js';
-
- describe('config', () => {
- const originalEnv = process.env;
-
- beforeEach(() => {
- vi.clearAllMocks();
- process.env = { ...originalEnv };
- delete process.env.ANTHROPIC_API_KEY;
- delete process.env.OPENAI_API_KEY;
- });
-
- afterEach(() => {
- process.env = originalEnv;
- });
-
- it('returns env config when ANTHROPIC_API_KEY is set', () => {
- process.env.ANTHROPIC_API_KEY = 'sk-ant-test';
- const config = loadConfig();
- expect(config?.provider).toBe('anthropic');
- });
-
- it('returns null when no env vars set', () => {
- expect(loadConfig()).toBeNull();
- });
- });
- ```
-
- ### Example 2: Testing file system operations with temp cleanup (file-tree.test.ts pattern)
-
- **User asks**: "Write tests for file tree analysis."
-
- **Actions taken**:
- 1. Create `src/fingerprint/__tests__/file-tree.test.ts`
- 2. Set up a cleanup array for temp directories
- 3. Create temp dirs in each test and push to cleanup array
- 4. Clean up all temp dirs in afterEach
- 5. Test with various file mtimes and directory structures
-
- **Result**:
- ```typescript
- import { describe, it, expect, afterEach } from 'vitest';
- import fs from 'fs';
- import path from 'path';
- import os from 'os';
- import { getFileTree } from '../file-tree.js';
-
- const dirs: string[] = [];
- afterEach(() => {
- for (const d of dirs) {
- try { fs.rmSync(d, { recursive: true, force: true }); } catch {}
- }
- dirs.length = 0;
- });
-
- describe('getFileTree', () => {
- it('returns files sorted by mtime descending', () => {
- const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'caliber-ft-'));
- dirs.push(tmp);
-
- fs.writeFileSync(path.join(tmp, 'a.ts'), 'a');
- fs.writeFileSync(path.join(tmp, 'b.ts'), 'b');
- const tree = getFileTree(tmp);
- expect(tree).toHaveLength(2);
- });
- });
- ```
-
- ### Example 3: Testing with module mocking and factory functions (index.test.ts pattern)
-
- **User asks**: "Write tests for the provider factory that instantiates different providers based on config."
-
- **Actions taken**:
- 1. Create `src/llm/__tests__/index.test.ts`
- 2. Use `vi.unmock('../index.js')` to test the real module
- 3. Create mock classes in `vi.hoisted()`
- 4. Mock dependencies (config, provider classes)
- 5. Test provider selection logic
- 6. Use `resetProvider()` between tests to clear cached instances
-
- **Result**:
- ```typescript
- import { describe, it, expect, vi, beforeEach } from 'vitest';
-
- vi.unmock('../index.js');
+ **User says**: "Write tests for the `detectLanguage` function in `src/fingerprint/code-analysis.ts`."
- const { mockLoadConfig, MockAnthropicProvider } = vi.hoisted(() => {
- class MockAnthropicProvider {
- config: unknown;
- call = vi.fn();
- constructor(c: unknown) { this.config = c; }
- }
- return {
- mockLoadConfig: vi.fn(),
- MockAnthropicProvider,
- };
- });
+ **Actions**:
+ 1. Create `src/fingerprint/__tests__/code-analysis.test.ts`.
+ 2. Import `detectLanguage` and Vitest utilities.
+ 3. Mock file I/O if needed: `vi.mock('fs/promises')` (not LLM).
+ 4. Write test cases:
+ ```typescript
+ import { describe, it, expect, vi } from 'vitest';
+ import { detectLanguage } from '../code-analysis.js';
- vi.mock('../config.js', () => ({
- loadConfig: () => mockLoadConfig(),
- }));
+ vi.mock('fs/promises', () => ({
+ readFile: vi.fn(),
+ }));
- vi.mock('../anthropic.js', () => ({
- AnthropicProvider: MockAnthropicProvider,
- }));
+ describe('detectLanguage', () => {
+ it('should detect TypeScript from file extension', () => {
+ const lang = detectLanguage('foo.ts');
+ expect(lang).toBe('typescript');
+ });
- import { getProvider, resetProvider } from '../index.js';
+ it('should detect Python from file extension', () => {
+ const lang = detectLanguage('bar.py');
+ expect(lang).toBe('python');
+ });
- describe('getProvider', () => {
- beforeEach(() => {
- vi.clearAllMocks();
- resetProvider();
- });
+ it('should return unknown for unrecognized extension', () => {
+ const lang = detectLanguage('data.xyz');
+ expect(lang).toBe('unknown');
+ });
+ });
+ ```
+ 5. Run: `npm run test -- src/fingerprint/__tests__/code-analysis.test.ts`
- it('creates AnthropicProvider for anthropic config', () => {
- mockLoadConfig.mockReturnValue({
- provider: 'anthropic',
- model: 'claude-sonnet-4-6',
- apiKey: 'sk-test',
- });
- const provider = getProvider();
- expect(provider).toBeInstanceOf(MockAnthropicProvider);
- });
- });
- ```
+ **Result**: Tests pass, file is committed, patterns replicated in future tests.
## Common Issues
- **"Cannot find module" when running tests**
- - **Cause**: `vi.mock()` called after the import statement
- - **Fix**: Move all `vi.mock()` and `vi.unmock()` calls to the TOP of the file, before any `import` statements
-
- **Environment variable persists across tests**
- - **Cause**: `beforeEach` assigns by reference instead of copying: `process.env = originalEnv`
- - **Fix**: Create a copy in `beforeEach`: `process.env = { ...originalEnv }; delete process.env.VAR`
-
- **Temporary files not cleaned up, filling disk**
- - **Cause**: `afterEach` not called or temp paths not tracked
- - **Fix**: Use centralized cleanup array: `dirs: string[] = []` pushed to in tests, cleaned in `afterEach` with `fs.rmSync(..., { recursive: true, force: true })`
-
- **Mock return value from previous test bleeds into next test**
- - **Cause**: `vi.clearAllMocks()` not called in `beforeEach`
- - **Fix**: Add `vi.clearAllMocks()` as first statement in `beforeEach`
-
- **"vi.mocked() is not a function" when accessing mock calls**
- - **Cause**: Trying to use `vi.mocked()` on a non-mocked module
- - **Fix**: Ensure the module is mocked with `vi.mock()` before importing, then cast: `const mockFn = vi.mocked(importedFn)`
-
- **Test passes locally but fails in CI**
- - **Cause**: Tests rely on real file system or network (not mocked)
- - **Fix**: Check if temp files are being created in real directories instead of mocked ones. Verify all external calls use `vi.mock()` for fs/http/exec
-
- **Coverage threshold failures: "lines not covered", "statements not covered"**
- - **Cause**: Branches or error paths not tested
- - **Fix**: Run `pnpm test:coverage` locally to see untested lines. Add `it()` tests for error cases, edge conditions, and branches that return different values
- - **Example**: If `if (x) return 'a'; else return 'b';` has 0% branch coverage, add tests: one with x=true, one with x=false
-
- **"expected 1 error but got 0" when testing error throws**
- - **Cause**: Error not actually thrown, or thrown asynchronously
- - **Fix**: For sync functions: `expect(() => fn()).toThrow('message')`. For async: `expect(async () => { await fn() }).rejects.toThrow('message')` or use `await expect(promise).rejects.toThrow()`
-
- **Mock factory returns undefined**
- - **Cause**: `vi.hoisted()` variables used before definition in the same block
- - **Fix**: Ensure `vi.hoisted()` block returns all factories, and `vi.mock()` blocks use them after the hoisted definition
-
- **Test flakes (sometimes passes, sometimes fails)**
- - **Cause**: Timing issues or file system race conditions in cleanup
- - **Fix**: For file cleanup, use `{ force: true }` in `rmSync`. For timing, avoid `setTimeout`; use `vi.useFakeTimers()` and `vi.runAllTimers()` if needed. Check for leftover files from previous test runs in `beforeEach`
+ - **"Cannot find module @anthropic-ai/sdk"**: This means `src/test/setup.ts` was not loaded. Verify `vitest.config.ts` has `setupFiles: ['./src/test/setup.ts']` in the config. Check the vitest.config.ts file for correct setup.
+ - **"ReferenceError: vi is not defined"**: Add `import { vi } from 'vitest'` at the top of the test file.
+ - **"Module not found: src/llm/anthropic.js"**: Check the import path uses `.js` extension (ESM convention). Verify file exists at `src/llm/anthropic.ts` → import as `./anthropic.js`.
+ - **"Timeout: async test did not finish"**: Add `async/await` to test function and ensure all promises are awaited. Example: `it('test', async () => { await fn(); expect(...); });`
+ - **"process.env is dirty after test"**: Verify `beforeEach/afterEach` are saving and restoring `process.env`. Add: `beforeEach(() => { originalEnv = { ...process.env }; }); afterEach(() => { process.env = originalEnv; });`
+ - **"Mock is not being called"**: Use `vi.getMocked(module).functionName` to assert on mocked function calls. Example: `expect(vi.getMocked(fs).readFile).toHaveBeenCalledWith('path')`.