caliber-testing · git:20260329.8cf93fd · 2026-03-29 · sha256 f9c44e627d908040

caliber-testing git:20260329.8cf93fdA

Immutable. This exact content never changes and is served at /api/v1/blob/f9c44e627d908040.

No tool to install first
claude plugin marketplace add https://markdownregistry.com/mp/a/art_nmkoi2anid6qlec5/f9c44e627d9080407df5289f620cfaeb306b9c5b23d6c51f96a4c6f88167512a.json

Then /plugin install caliber-testing@markdownregistry-nmkoi2anid6qlec5 in Claude Code. It installs this exact version, the one graded A here, pinned to the commit it was graded at, and leaves it there.

---
name: caliber-testing
description: Writes Vitest tests following project patterns: __tests__/ directories, vi.mock() for module mocking, global LLM mock from src/test/setup.ts, environment variable save/restore, and test file organization. Use when user says 'write tests', 'add test coverage', 'test this', creates *.test.ts files, or when test failures appear in CI. Do NOT use for non-test code or for debugging without writing tests.
---
# Caliber Testing

## Critical

- **Never test internal implementation details.** Test behavior and side effects only.
- **Always use vi.mock() with default implementation** for external dependencies (LLM APIs, file system, git commands).
- **Global LLM mock is pre-configured in `src/test/setup.ts`** — you do NOT need to mock LLM calls in individual tests unless overriding the default.
- **Environment variables must be saved/restored per-test** using `beforeEach`/`afterEach`. Do NOT assume env state between tests.
- **Test files live in `__tests__/` directories** adjacent to the code they test (e.g., `src/scoring/__tests__/accuracy.test.ts` for `src/scoring/accuracy.ts`).

## Instructions

1. **Create the test file** in the correct location: `src/<module>/__tests__/<feature>.test.ts`
   - Verify the `__tests__/` directory exists; create if missing.

2. **Import test utilities** at the top of the file:
   ```typescript
   import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
   import { SomeModule } from '../module.js'; // ESM with .js extension
   ```
   - Always use `.js` extensions in imports (ESM convention).
   - Import only what you need to test.

3. **Set up mocks for external dependencies** using `vi.mock()` before imports:
   ```typescript
   vi.mock('fs/promises');
   vi.mock('../some-internal-module.js');
   ```
   - For LLM calls: do NOT mock — the global mock from `src/test/setup.ts` handles it.
   - For file I/O, git commands, spawn/exec: mock these modules.

4. **Save/restore environment variables per-test**:
   ```typescript
   describe('MyModule', () => {
     beforeEach(() => {
       process.env.MY_VAR = 'test-value';
     });
     afterEach(() => {
       delete process.env.MY_VAR;
     });
   });
   ```
   - Do this even if the test doesn't directly set env vars — tests may inherit from CI or prior tests.

5. **Write focused test cases** with clear intent:
   ```typescript
   it('should return true when condition is met', async () => {
     const result = await myFunction({ input: 'data' });
     expect(result).toBe(true);
   });
   ```
   - One assertion per test when possible; group related assertions in one test.
   - Use descriptive test names that explain the behavior.
   - Verify side effects (file writes, logs, state changes) when relevant.

6. **Run tests locally before committing**:
   ```bash
   npm run test                                          # all tests
   npx vitest run src/scoring/__tests__/accuracy.test.ts # single file
   npm run test -- --coverage                           # coverage report
   ```
   - Verify all tests pass and coverage is acceptable (>80% for critical paths).

## Examples

### User says: "Write tests for the accuracy scoring check"

**File created**: `src/scoring/__tests__/accuracy.test.ts`

```typescript
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import { checkAccuracy } from '../checks/accuracy.js';

describe('checkAccuracy', () => {
  beforeEach(() => {
    process.env.CALIBER_LEARN_ENABLED = 'false';
  });
  afterEach(() => {
    delete process.env.CALIBER_LEARN_ENABLED;
  });

  it('should return 100 when CLAUDE.md matches generated content', async () => {
    const result = await checkAccuracy({
      existing: { path: 'CLAUDE.md', content: 'test' },
      generated: 'test',
    });
    expect(result.score).toBe(100);
  });

  it('should return 0 when content differs significantly', async () => {
    const result = await checkAccuracy({
      existing: { path: 'CLAUDE.md', content: 'old' },
      generated: 'new',
    });
    expect(result.score).toBe(0);
  });
});
```

### User says: "Test the fingerprint collector"

**File created**: `src/fingerprint/__tests__/index.test.ts`

```typescript
import { describe, it, expect, vi, beforeEach } from 'vitest';
import * as fs from 'fs/promises';
import { collectFingerprint } from '../index.js';

vi.mock('fs/promises');
vi.mock('../file-tree.js');
vi.mock('../code-analysis.js');

describe('collectFingerprint', () => {
  beforeEach(() => {
    vi.clearAllMocks();
  });

  it('should return a fingerprint object with all expected fields', async () => {
    const result = await collectFingerprint('/test/project');
    expect(result).toHaveProperty('framework');
    expect(result).toHaveProperty('language');
    expect(result).toHaveProperty('files');
  });
});
```

## Common Issues

**"Cannot find module '__mocks__'"**
- Issue: Attempting to use Jest-style `__mocks__/` directory.
- Fix: Use `vi.mock('module-name')` inline in your test file instead. Caliber uses Vitest, not Jest.

**"ReferenceError: process is not defined"**
- Issue: Test is running in a browser-like environment.
- Fix: Add `environment: 'node'` to `vitest.config.ts` for that test file, or ensure `src/test/setup.ts` is loaded.

**"Test timeout of 10000ms exceeded"**
- Issue: Async operation (file I/O, network) is waiting on a real module instead of a mock.
- Fix: Verify you've called `vi.mock('module-name')` before importing the code under test.
- Check: Run `npx vitest --reporter=verbose` to see which module is blocking.

**"Expected mock to be called but it wasn't"**
- Issue: Mock is defined but the code path doesn't execute.
- Fix: Verify the test input reaches the mocked function. Add `console.log()` before the call, or use `vi.mocked(mockFn).mock.calls` to inspect.

**"env var set in test is visible in next test"**
- Issue: Missing `afterEach(() => delete process.env.VAR)` or similar cleanup.
- Fix: Always pair `beforeEach` with `afterEach` for env vars. Use the pattern shown in Step 4.

**"LLM mock is not being used"**
- Issue: Test is importing LLM module before the mock is active.
- Fix: Do NOT manually mock LLM modules — the global mock in `src/test/setup.ts` handles all LLM calls. If override is needed, call `vi.mocked(mockFn).mockResolvedValueOnce(...)` in your test.