git:20260329.8cf93fd to git:20260330.4e5d8f6

158 added, 105 removed. Audit A to C.

What this update changes in the checks

This update newly trips 2 checks:

  • No instruction to read or print local credentials
    matched: read from process.env
  • No script tag
    matched: <script

A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.

---
name: caliber-testing
description: Writes Vitest tests following project patterns: __tests__/ directories, vi.mock() for module mocking, global LLM mock from src/test/setup.ts, environment variable save/restore, and test file organization. Use when user says 'write tests', 'add test coverage', 'test this', creates *.test.ts files, or when test failures appear in CI. Do NOT use for non-test code or for debugging without writing tests.
---
- # Caliber Testing
+ # caliber-testing
## Critical
- - **Never test internal implementation details.** Test behavior and side effects only.
- - **Always use vi.mock() with default implementation** for external dependencies (LLM APIs, file system, git commands).
- - **Global LLM mock is pre-configured in `src/test/setup.ts`** — you do NOT need to mock LLM calls in individual tests unless overriding the default.
- - **Environment variables must be saved/restored per-test** using `beforeEach`/`afterEach`. Do NOT assume env state between tests.
- - **Test files live in `__tests__/` directories** adjacent to the code they test (e.g., `src/scoring/__tests__/accuracy.test.ts` for `src/scoring/accuracy.ts`).
+ - **Test file location**: Always place tests in `__tests__/` subdirectory of the module being tested. File name: `module-name.test.ts`. Example: `src/llm/__tests__/anthropic.test.ts` for `src/llm/anthropic.ts`.
+ - **Global LLM mock**: Use the mock from `src/test/setup.ts`. It is auto-loaded by Vitest. Do NOT create duplicate LLM mocks in individual test files.
+ - **Avoid over-mocking**: Mock only external dependencies (HTTP, file I/O, LLM calls). Never mock internal modules or pure logic.
+ - **Environment cleanup**: Always save and restore `process.env` between tests to prevent cross-test pollution. Use the pattern in Step 4.
+ - **No hardcoded test data**: Use `beforeEach` to set up fresh mocks and env state for each test.
## Instructions
- 1. **Create the test file** in the correct location: `src/<module>/__tests__/<feature>.test.ts`
- - Verify the `__tests__/` directory exists; create if missing.
+ ### Step 1: Set up the test file structure
+ Create `src/<module>/__tests__/<module-name>.test.ts`. Import `describe`, `it`, `expect`, `beforeEach`, `afterEach` from `vitest`, and import the module under test.
- 2. **Import test utilities** at the top of the file:
- ```typescript
- import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
- import { SomeModule } from '../module.js'; // ESM with .js extension
- ```
- - Always use `.js` extensions in imports (ESM convention).
- - Import only what you need to test.
+ ```typescript
+ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
+ import { functionToTest } from '../module-name.js';
+ ```
- 3. **Set up mocks for external dependencies** using `vi.mock()` before imports:
- ```typescript
- vi.mock('fs/promises');
- vi.mock('../some-internal-module.js');
- ```
- - For LLM calls: do NOT mock — the global mock from `src/test/setup.ts` handles it.
- - For file I/O, git commands, spawn/exec: mock these modules.
+ Verify: The test file is in the correct `__tests__/` directory parallel to the module it tests.
- 4. **Save/restore environment variables per-test**:
- ```typescript
- describe('MyModule', () => {
- beforeEach(() => {
- process.env.MY_VAR = 'test-value';
- });
- afterEach(() => {
- delete process.env.MY_VAR;
- });
- });
- ```
- - Do this even if the test doesn't directly set env vars — tests may inherit from CI or prior tests.
+ ### Step 2: Mock external dependencies with vi.mock()
+ At the top of the test file, before `describe`, mock external modules using `vi.mock()`. This applies to:
+ - HTTP calls (use `vi.mock()` to intercept fetch or axios)
+ - File I/O (use `vi.mock('fs')` or similar)
+ - LLM providers (already mocked globally via `src/test/setup.ts`, but document the assumption)
- 5. **Write focused test cases** with clear intent:
- ```typescript
- it('should return true when condition is met', async () => {
- const result = await myFunction({ input: 'data' });
- expect(result).toBe(true);
- });
- ```
- - One assertion per test when possible; group related assertions in one test.
- - Use descriptive test names that explain the behavior.
- - Verify side effects (file writes, logs, state changes) when relevant.
+ Example:
+ ```typescript
+ vi.mock('fs', () => ({
+ readFileSync: vi.fn(() => 'mock content'),
+ }));
+ ```
- 6. **Run tests locally before committing**:
- ```bash
- npm run test # all tests
- npx vitest run src/scoring/__tests__/accuracy.test.ts # single file
- npm run test -- --coverage # coverage report
- ```
- - Verify all tests pass and coverage is acceptable (>80% for critical paths).
+ Verify: All external dependencies are mocked. Internal module calls are NOT mocked.
- ## Examples
+ ### Step 3: Use beforeEach and afterEach for setup and cleanup
+ For each test suite, create `beforeEach` to reset mocks and set up test state. Create `afterEach` to clean up.
- ### User says: "Write tests for the accuracy scoring check"
+ ```typescript
+ describe('module-name', () => {
+ beforeEach(() => {
+ vi.clearAllMocks();
+ });
- **File created**: `src/scoring/__tests__/accuracy.test.ts`
+ afterEach(() => {
+ vi.resetAllMocks();
+ });
+ it('should do something', () => {
+ // test body
+ });
+ });
+ ```
+
+ Verify: `vi.clearAllMocks()` is called in `beforeEach` to reset mock call counts.
+
+ ### Step 4: Save and restore process.env for env-dependent tests
+ If a test modifies `process.env`, save the original state and restore it. Use this pattern:
+
```typescript
- import { describe, it, expect, beforeEach, afterEach } from 'vitest';
- import { checkAccuracy } from '../checks/accuracy.js';
+ describe('env-dependent module', () => {
+ const originalEnv = process.env;
- describe('checkAccuracy', () => {
beforeEach(() => {
- process.env.CALIBER_LEARN_ENABLED = 'false';
+ process.env = { ...originalEnv };
+ vi.clearAllMocks();
});
+
afterEach(() => {
- delete process.env.CALIBER_LEARN_ENABLED;
+ process.env = originalEnv;
+ vi.resetAllMocks();
});
- it('should return 100 when CLAUDE.md matches generated content', async () => {
- const result = await checkAccuracy({
- existing: { path: 'CLAUDE.md', content: 'test' },
- generated: 'test',
- });
- expect(result.score).toBe(100);
+ it('should read from process.env.VARIABLE', () => {
+ process.env.VARIABLE = 'test-value';
+ // test assertion
});
+ });
+ ```
- it('should return 0 when content differs significantly', async () => {
- const result = await checkAccuracy({
- existing: { path: 'CLAUDE.md', content: 'old' },
- generated: 'new',
- });
- expect(result.score).toBe(0);
- });
+ Verify: `process.env` is restored to its original state after each test.
+
+ ### Step 5: Write assertions that match project conventions
+ Use `expect()` with clear matchers. For async functions, use `await` or `.resolves`.
+
+ ```typescript
+ it('should return the expected value', async () => {
+ const result = await functionToTest('input');
+ expect(result).toEqual({ key: 'value' });
});
+
+ it('should throw on invalid input', async () => {
+ await expect(functionToTest(null)).rejects.toThrow('Invalid input');
+ });
```
- ### User says: "Test the fingerprint collector"
+ Verify: Assertions are specific (not just checking truthiness) and async functions use `await` or `.resolves`.
- **File created**: `src/fingerprint/__tests__/index.test.ts`
+ ### Step 6: Run and verify the tests
+ Run the specific test file to ensure it passes:
+ ```bash
+ npm run test -- src/<module>/__tests__/<module-name>.test.ts
+ ```
+ Or run all tests:
+ ```bash
+ npm run test
+ ```
+
+ Verify: All tests pass. Check test coverage with `npm run test -- --coverage`.
+
+ ## Examples
+
+ ### Example: Testing an LLM provider module
+
+ User says: "Write tests for src/llm/anthropic.ts"
+
+ **Actions:**
+ 1. Create `src/llm/__tests__/anthropic.test.ts`
+ 2. Import `describe`, `it`, `expect`, `beforeEach`, `afterEach`, `vi` from vitest
+ 3. Mock external HTTP calls (fetch is mocked globally via setup)
+ 4. Write tests for `call()` and `stream()` methods
+ 5. Save/restore `process.env` for API key tests
+ 6. Run: `npm run test -- src/llm/__tests__/anthropic.test.ts`
+
+ **Result:**
```typescript
- import { describe, it, expect, vi, beforeEach } from 'vitest';
- import * as fs from 'fs/promises';
- import { collectFingerprint } from '../index.js';
+ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
+ import { AnthropicProvider } from '../anthropic.js';
- vi.mock('fs/promises');
- vi.mock('../file-tree.js');
- vi.mock('../code-analysis.js');
+ describe('AnthropicProvider', () => {
+ const originalEnv = process.env;
- describe('collectFingerprint', () => {
beforeEach(() => {
+ process.env = { ...originalEnv, ANTHROPIC_API_KEY: 'test-key' };
vi.clearAllMocks();
});
- it('should return a fingerprint object with all expected fields', async () => {
- const result = await collectFingerprint('/test/project');
- expect(result).toHaveProperty('framework');
- expect(result).toHaveProperty('language');
- expect(result).toHaveProperty('files');
+ afterEach(() => {
+ process.env = originalEnv;
+ vi.resetAllMocks();
});
+
+ it('should call the API with correct parameters', async () => {
+ const provider = new AnthropicProvider();
+ const response = await provider.call({ messages: [{ role: 'user', content: 'test' }] });
+ expect(response).toBeDefined();
+ });
});
```
+ ### Example: Testing a utility function
+
+ User says: "Add tests for src/utils/sanitize.ts"
+
+ **Actions:**
+ 1. Create `src/utils/__tests__/sanitize.test.ts`
+ 2. Import the sanitize function
+ 3. Write tests for different input types
+ 4. No mocking needed (pure logic)
+ 5. Run: `npm run test -- src/utils/__tests__/sanitize.test.ts`
+
+ **Result:**
+ ```typescript
+ import { describe, it, expect } from 'vitest';
+ import { sanitize } from '../sanitize.js';
+
+ describe('sanitize', () => {
+ it('should remove dangerous characters', () => {
+ expect(sanitize('<script>alert()</script>')).toEqual('scriptalert()script');
+ });
+
+ it('should handle null input', () => {
+ expect(sanitize(null)).toEqual('');
+ });
+ });
+ ```
+
## Common Issues
- **"Cannot find module '__mocks__'"**
- - Issue: Attempting to use Jest-style `__mocks__/` directory.
- - Fix: Use `vi.mock('module-name')` inline in your test file instead. Caliber uses Vitest, not Jest.
+ **Issue: "Cannot find module" error when importing from test**
+ - **Cause**: Missing `.js` extension in ESM imports
+ - **Fix**: Ensure all imports in test files use `.js` extension: `import { foo } from '../module.js'`
- **"ReferenceError: process is not defined"**
- - Issue: Test is running in a browser-like environment.
- - Fix: Add `environment: 'node'` to `vitest.config.ts` for that test file, or ensure `src/test/setup.ts` is loaded.
+ **Issue: "process.env.VARIABLE is undefined" in tests**
+ - **Cause**: Environment variable not set up in test or not restored properly
+ - **Fix**: Use the save/restore pattern from Step 4. Set `process.env.VARIABLE = 'value'` in `beforeEach`.
- **"Test timeout of 10000ms exceeded"**
- - Issue: Async operation (file I/O, network) is waiting on a real module instead of a mock.
- - Fix: Verify you've called `vi.mock('module-name')` before importing the code under test.
- - Check: Run `npx vitest --reporter=verbose` to see which module is blocking.
+ **Issue: "vi.mock is not defined" or mock not working**
+ - **Cause**: Attempting to mock after imports, or mocking an already-imported module
+ - **Fix**: Call `vi.mock()` at the top of the file, before any imports of the mocked module.
- **"Expected mock to be called but it wasn't"**
- - Issue: Mock is defined but the code path doesn't execute.
- - Fix: Verify the test input reaches the mocked function. Add `console.log()` before the call, or use `vi.mocked(mockFn).mock.calls` to inspect.
+ **Issue: Tests pass locally but fail in CI**
+ - **Cause**: Mock state or env vars not cleaned up between tests
+ - **Fix**: Add `vi.clearAllMocks()` in `beforeEach` and restore `process.env` in `afterEach`.
- **"env var set in test is visible in next test"**
- - Issue: Missing `afterEach(() => delete process.env.VAR)` or similar cleanup.
- - Fix: Always pair `beforeEach` with `afterEach` for env vars. Use the pattern shown in Step 4.
+ **Issue: "Timeout of 5000ms exceeded"**
+ - **Cause**: Async test or mock not resolving
+ - **Fix**: Ensure mocks return resolved promises. Use `vi.fn().mockResolvedValue()` for async mocks.
- **"LLM mock is not being used"**
- - Issue: Test is importing LLM module before the mock is active.
- - Fix: Do NOT manually mock LLM modules — the global mock in `src/test/setup.ts` handles all LLM calls. If override is needed, call `vi.mocked(mockFn).mockResolvedValueOnce(...)` in your test.
+ **Issue: Tests reference global LLM mock but it's not available**
+ - **Cause**: Test setup not loading `src/test/setup.ts`
+ - **Fix**: Verify `vitest.config.ts` includes `setupFiles: ['src/test/setup.ts']`. This is already configured in the project.