llm-provider · diff

git:20260329.8cf93fd to git:20260424.9b749cd

224 added, 58 removed. Audit A to A.

What this update changes in the checks

No check changed: every fixed check gives the same answer before and after this update.

A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.

---
name: llm-provider
- description: Adds a new LLM provider implementing LLMProvider interface from src/llm/types.ts with call() and stream() methods. Integrates config in src/llm/config.ts and factory in src/llm/index.ts. Use when adding a new AI backend, integrating a new model API, or extending provider support. Do NOT use for modifying existing providers or debugging provider issues.
+ description: Adds a new LLM provider implementing LLMProvider interface with call() and stream() methods. Integrates with provider factory in src/llm/index.ts, config detection in src/llm/config.ts, and error handling via tracking and recovery. Use when adding a new model backend, integrating a third-party LLM API, or extending LLM platform support. Do NOT use for fixing bugs in existing providers, modifying existing provider behavior, or changing the LLMProvider interface.
+ paths:
+ - src/llm/**/*.ts
+ - src/llm/__tests__/**/*.ts
---
# LLM Provider
## Critical
- - **All providers must implement `LLMProvider` interface** from `src/llm/types.ts`: `call(messages, params)` and `stream(messages, params)` returning `AsyncIterable<StreamChunk>`
- - **No partial implementations**: Both `call()` and `stream()` must work. Streaming is not optional.
- - **StreamChunk format**: `{ type: 'text' | 'error' | 'usage'; value: string; usage?: { input: number; output: number } }`
- - **Error handling**: Catch provider-specific errors, map to `ChatError` from `src/llm/types.ts` with `code` (e.g., `'auth'`, `'rate_limit'`, `'network'`) and `message`
- - **Model validation**: Call `validateModel(modelId)` in config before accepting the model. Refer to `src/llm/config.ts` for pattern.
+ 1. **All providers MUST implement the LLMProvider interface** from src/llm/types.ts with three methods:
+ - call(options: LLMCallOptions): Promise<string> — single non-streaming call returning text
+ - stream(options: LLMStreamOptions, callbacks: LLMStreamCallbacks): Promise<void> — streaming call invoking callbacks
+ - listModels?(): Promise<string[]> — optional; list available models from the API
+ 2. **Initialize client in constructor and store defaultModel from config**. Example: `this.client = new YourSDK({ apiKey: config.apiKey })`. Never lazy-initialize on first call — providers are instantiated once and cached in src/llm/index.ts.
+
+ 3. **For EVERY response in call() and stream(), invoke trackUsage(model, usage)** from src/llm/usage.js before returning/ending. This is mandatory — it captures token metrics for CLI telemetry and cost analysis. If the API doesn't return usage data, estimate via estimateTokens(text), which assumes ~4 chars per token.
+
+ 4. **Both call() and stream() must respect the model parameter** using pattern: `options.model || this.defaultModel`. Never hardcode model names. Callers supply model overrides via LLMCallOptions.model.
+
+ 5. **Error handling: catch all errors, preserve error messages unchanged**. The retry logic in src/llm/index.ts handles transient errors (ECONNRESET, socket hang up, 529 overload). For seat-based providers (Cursor, Claude CLI), wrap stderr via parseSeatBasedError() for user-friendly messages.
+
+ 6. **Always update ProviderType union** (Step 2), DEFAULT_MODELS (Step 4), and createProvider() switch case (Step 5) in lock-step. Missing any one breaks the build or causes runtime Unknown provider error.
+
## Instructions
- 1. **Define provider file** at `src/llm/{provider-name}.ts`
- - Import `{ LLMProvider, ChatMessage, ChatParams, StreamChunk, ChatError }` from `src/llm/types.ts`
- - Export class `{ProviderName}Provider implements LLMProvider`
- - Constructor takes `{ apiKey?: string; model: string; baseUrl?: string }` matching config structure
- - Store config in instance: `this.apiKey = apiKey || process.env.{PROVIDER_API_KEY}`
- - Verify API key exists on first call, throw `ChatError` with `code: 'auth'` if missing
+ ### Step 1: Create provider class file
+ Verify directory exists: `ls -la src/llm/`. Create `src/llm/your-provider.ts`. Match existing provider patterns (src/llm/anthropic.ts, src/llm/openai-compat.ts).
- 2. **Implement `call()` method**
- - Signature: `async call(messages: ChatMessage[], params: ChatParams): Promise<string>`
- - Make HTTP request to provider API with messages and params (temperature, max_tokens, etc.)
- - Extract text from response, return as single string
- - On error (network, auth, rate limit), throw `ChatError` with appropriate `code` and `message`
- - Verify this works before proceeding
+ Minimal structure:
+ ```typescript
+ import type { LLMProvider, LLMCallOptions, LLMStreamOptions, LLMStreamCallbacks, LLMConfig, TokenUsage } from './types.js';
+ import { trackUsage } from './usage.js';
+ import { estimateTokens } from './utils.js';
- 3. **Implement `stream()` method**
- - Signature: `async *stream(messages: ChatMessage[], params: ChatParams): AsyncIterable<StreamChunk>`
- - Use provider's streaming endpoint (e.g., SSE, WebSocket, chunked response)
- - Yield `{ type: 'text', value: '<chunk>' }` for each text delta
- - Yield `{ type: 'usage', value: '', usage: { input, output } }` at end if available
- - On error, yield `{ type: 'error', value: '<error message>' }` and return
- - Test streaming with `for await (const chunk of stream(...)) { console.log(chunk) }`
+ export class YourProviderProvider implements LLMProvider {
+ private client: YourSDKType;
+ private defaultModel: string;
- 4. **Add config in `src/llm/config.ts`**
- - Import `{ProviderName}Provider` in `getProvider(config, model)` function
- - Add condition: `if (config.provider === '{provider-slug}') return new {ProviderName}Provider({ apiKey: config.apiKey, model, baseUrl: config.baseUrl })`
- - Add case in `validateModel()`: check against provider's official model list or hardcode supported models
- - Export provider slug in `SUPPORTED_PROVIDERS` array if it exists
- - Verify config function accepts and routes your provider
+ constructor(config: LLMConfig) {
+ if (!config.apiKey) throw new Error('API key required');
+ this.client = new YourSDK({ apiKey: config.apiKey, ...(config.baseUrl && { baseURL: config.baseUrl }) });
+ this.defaultModel = config.model;
+ }
- 5. **Register in factory at `src/llm/index.ts`**
- - Import provider in `getProvider()` function
- - Add to the conditional chain matching provider name from config
- - Run `npm run build && npm run test` to verify factory picks up provider
+ async call(options: LLMCallOptions): Promise<string> {
+ const model = options.model || this.defaultModel;
+ const response = await this.client.messages.create({ model, max_tokens: options.maxTokens || 4096, system: options.system, messages: [{ role: 'user', content: options.prompt }] });
+ trackUsage(model, { inputTokens: response.usage?.input_tokens || 0, outputTokens: response.usage?.output_tokens || 0 });
+ return response.content?.[0]?.text || '';
+ }
- 6. **Add tests in `src/llm/__tests__/{provider-name}.test.ts`**
- - Mock API responses using `vitest.mock()` or `fetch` stub
- - Test `call()`: verify message formatting, response parsing, error handling
- - Test `stream()`: verify chunk parsing, usage reporting, error yields
- - Test config validation: invalid model, missing API key
- - Run `npx vitest run src/llm/__tests__/{provider-name}.test.ts`
+ async stream(options: LLMStreamOptions, callbacks: LLMStreamCallbacks): Promise<void> {
+ const model = options.model || this.defaultModel;
+ const messages = [...(options.messages || []), { role: 'user' as const, content: options.prompt }];
+ try {
+ const stream = await this.client.stream({ model, max_tokens: options.maxTokens || 10240, system: options.system, messages });
+ let stopReason: string | undefined, usage: TokenUsage | undefined;
+ for await (const chunk of stream) {
+ if (chunk.delta?.text) callbacks.onText(chunk.delta.text);
+ if (chunk.delta?.stop_reason) stopReason = chunk.delta.stop_reason;
+ if (chunk.usage) usage = { inputTokens: chunk.usage.input_tokens, outputTokens: chunk.usage.output_tokens };
+ }
+ if (usage) trackUsage(model, usage);
+ callbacks.onEnd({ stopReason, usage });
+ } catch (error) { callbacks.onError(error instanceof Error ? error : new Error(String(error))); }
+ }
+ }
+ ```
+ Verify: File exports the class; imports match existing providers.
+
+ ### Step 2: Add to ProviderType union
+ Edit `src/llm/types.ts` line 1. Add your provider in kebab-case:
+ ```typescript
+ export type ProviderType = 'anthropic' | 'vertex' | 'openai' | 'cursor' | 'claude-cli' | 'your-provider';
+ ```
+ Verify: `npx tsc --noEmit` shows no ProviderType errors.
+
+ ### Step 3: Add config fields
+ If your provider needs fields beyond apiKey, model, baseUrl, extend LLMConfig in src/llm/types.ts:
+ ```typescript
+ export interface LLMConfig {
+ provider: ProviderType;
+ model: string;
+ fastModel?: string;
+ apiKey?: string;
+ baseUrl?: string;
+ yourProviderSecret?: string;
+ }
+ ```
+
+ ### Step 4: Update config.ts
+ Edit `src/llm/config.ts`:
+
+ **Line 9:** Add to DEFAULT_MODELS:
+ ```typescript
+ export const DEFAULT_MODELS: Record<ProviderType, string> = {
+ anthropic: 'claude-sonnet-4-6',
+ vertex: 'claude-sonnet-4-6',
+ openai: 'gpt-5.4-mini',
+ cursor: 'sonnet-4.6',
+ 'claude-cli': 'default',
+ 'your-provider': 'your-provider/default-model',
+ };
+ ```
+
+ **Line 17:** Add to MODEL_CONTEXT_WINDOWS if known:
+ ```typescript
+ export const MODEL_CONTEXT_WINDOWS: Record<string, number> = {
+ 'your-provider/model-name': 128_000,
+ };
+ ```
+
+ **Line 59:** In resolveFromEnv(), add env detection before final return null:
+ ```typescript
+ if (process.env.YOUR_PROVIDER_API_KEY) {
+ return {
+ provider: 'your-provider',
+ apiKey: process.env.YOUR_PROVIDER_API_KEY,
+ model: process.env.CALIBER_MODEL || DEFAULT_MODELS['your-provider'],
+ baseUrl: process.env.YOUR_PROVIDER_BASE_URL,
+ };
+ }
+ ```
+
+ **Line 115:** In readConfigFile() validation, add 'your-provider' to includes list.
+
+ Verify: `npm run test -- src/llm/__tests__/ -t config` confirms env var detection works.
+
+ ### Step 5: Register in factory
+ Edit `src/llm/index.ts`. Add import (line ~4):
+ ```typescript
+ import { YourProviderProvider } from './your-provider.js';
+ ```
+
+ In createProvider() switch (line ~24), add before default case:
+ ```typescript
+ case 'your-provider':
+ return new YourProviderProvider(config);
+ ```
+
+ Verify: `npx tsc --noEmit` passes; no type errors on switch cases.
+
+ ### Step 6: Write tests
+ Create `src/llm/__tests__/your-provider.test.ts`:
+ ```typescript
+ import { describe, it, expect, beforeEach } from 'vitest';
+ import { YourProviderProvider } from '../your-provider.js';
+
+ describe('YourProviderProvider', () => {
+ let provider: YourProviderProvider;
+ beforeEach(() => {
+ provider = new YourProviderProvider({ provider: 'your-provider', model: 'test', apiKey: 'test' });
+ });
+
+ it('implements LLMProvider interface', () => {
+ expect(typeof provider.call).toBe('function');
+ expect(typeof provider.stream).toBe('function');
+ });
+
+ it('call() returns string', async () => {
+ const result = await provider.call({ system: 'helpful', prompt: 'hi' });
+ expect(typeof result).toBe('string');
+ });
+
+ it('stream() invokes callbacks', async () => {
+ const texts: string[] = [];
+ let ended = false;
+ await provider.stream({ system: 'helpful', prompt: 'hi' }, {
+ onText: (t) => texts.push(t),
+ onEnd: () => { ended = true; },
+ onError: () => {},
+ });
+ expect(ended).toBe(true);
+ });
+ });
+ ```
+
+ Verify: `npm run test -- src/llm/__tests__/your-provider.test.ts` passes.
+
+ ### Step 7: Integration test
+ Run factory tests with your provider env var:
+ ```bash
+ YOUR_PROVIDER_API_KEY=test npm run test -- src/llm/__tests__/index.test.ts
+ ```
+ Verify: getProvider() instantiates your provider; llmCall() dispatches correctly.
+
## Examples
- **User says**: "Add support for Groq as a new LLM provider."
+ ### Example 1: Local LM Studio server
+ User says: "I need caliber to use my local LM Studio instance."
- **Actions**:
- 1. Create `src/llm/groq.ts` with `GroqProvider` class
- 2. Implement `call()` calling `https://api.groq.com/openai/v1/chat/completions` with OpenAI-compatible format
- 3. Implement `stream()` using same endpoint with `stream: true`
- 4. In `src/llm/config.ts`, add `if (config.provider === 'groq') return new GroqProvider(...)`
- 5. In `src/llm/index.ts`, import and route `groq` provider in `getProvider()`
- 6. Create tests mocking Groq API responses
- 7. Verify: `npm run test -- src/llm/__tests__/groq.test.ts` passes
+ Actions: Create src/llm/lm-studio.ts extending OpenAICompatProvider. Add 'lm-studio' to ProviderType. In config.ts:
+ ```typescript
+ if (process.env.LM_STUDIO_BASE_URL) {
+ return { provider: 'lm-studio', apiKey: '', model: 'local', baseUrl: process.env.LM_STUDIO_BASE_URL };
+ }
+ ```
+ Register in createProvider() case. User: `export LM_STUDIO_BASE_URL=http://localhost:8000/v1`. Result: caliber uses local LM Studio; tokens estimated via estimateTokens().
- **Result**: Caliber can now use Groq models via `{ provider: 'groq', apiKey: '...', model: 'mixtral-8x7b-32768' }`
+ ### Example 2: Ollama (seat-based)
+ User says: "Ollama is auto-detected; no API key needed."
+ Actions: Create src/llm/ollama.ts extending OpenAICompatProvider. Add 'ollama' to ProviderType and SEAT_BASED_PROVIDERS. In config.ts:
+ ```typescript
+ if (process.env.OLLAMA_HOST) {
+ return { provider: 'ollama', model: 'mistral', baseUrl: process.env.OLLAMA_HOST || 'http://localhost:11434/v1' };
+ }
+ ```
+ Result: Offline per-machine LLM without API keys.
+
## Common Issues
- - **"Provider not recognized" in factory**: Verify provider slug matches exactly in `getProvider()` condition AND in config file. Check spelling and case sensitivity.
- - **"TypeError: stream is not async iterable"**: Ensure `stream()` is a generator function (uses `async *` and `yield`). Test with `for await` loop before deploying.
- - **"API key is undefined"**: Verify environment variable name in provider constructor matches what user sets. Log `apiKey` value in error message: `throw new ChatError('auth', 'API key missing: check {ENV_VAR_NAME}')`
- - **"Stream stops early or yields garbage"**: Check provider's response format (JSON lines, SSE, etc.). Log raw response chunk: `console.error('Raw chunk:', chunk)` to debug parsing.
- - **"Model validation fails but model is valid"**: Ensure `validateModel()` in config covers all supported models for this provider. If list is dynamic, call provider's models endpoint and cache.
- - **Type errors on ChatError**: Verify import is `from 'src/llm/types.js'` (with `.js` extension for ESM).
- - **Tests fail with "fetch is not defined"**: Add `import { fetch } from 'node-fetch'` or mock globally in `src/test/setup.ts`.
+ **Unknown provider: your-provider**
+ - Cause: ProviderType updated but createProvider() case missing.
+ - Fix: Add case in src/llm/index.ts and import the class.
+
+ **Cannot find module './your-provider.js'**
+ - Cause: File named your_provider.ts (underscore) not your-provider.ts (kebab).
+ - Fix: Rename file to use kebab-case.
+
+ **API key is required for YourProvider**
+ - Cause: Env var YOUR_PROVIDER_API_KEY not set; resolveFromEnv() didn't detect it.
+ - Fix: Verify env var name in config.ts matches. Test: `YOUR_PROVIDER_API_KEY=test npm run test -- src/llm/__tests__/index.test.ts`.
+
+ **LLM response did not include usage tokens**
+ - Cause: Provider API doesn't return usage (local models).
+ - Fix: Estimate tokens: `trackUsage(model, { inputTokens: estimateTokens(options.system + options.prompt), outputTokens: estimateTokens(response.text) });`
+
+ **Stream callbacks never fire; onEnd not called**
+ - Cause: Async iterator not fully consumed before method returns.
+ - Fix: Ensure iteration completes before onEnd(): `for await (const chunk of stream) { } callbacks.onEnd({ stopReason, usage });`
+
+ **My model parameter is ignored**
+ - Cause: call()/stream() doesn't use `options.model || this.defaultModel`.
+ - Fix: Replace hardcoded model: `const model = options.model || this.defaultModel; const response = await this.client.create({ model, ... });`
+
+ **trackUsage() is never called**
+ - Cause: Forgot to call trackUsage() in call() or stream().
+ - Fix: Add after every response: `trackUsage(model, { inputTokens: ..., outputTokens: ... });`
+
+ **Type error: Provider doesn't implement LLMProvider**
+ - Cause: Missing method or wrong signature.
+ - Fix: Verify all required methods exist with exact signatures from src/llm/types.ts. Copy-paste from anthropic.ts as reference.