llm-provider · git:20260319.c97e02c · 2026-03-19 · sha256 e09951931568e261

llm-provider git:20260319.c97e02cA

Immutable. This exact content never changes and is served at /api/v1/blob/e09951931568e261.

No tool to install first
claude plugin marketplace add https://markdownregistry.com/mp/a/art_neeitkenrgyxfdnl/e09951931568e261fad85f621099f314eb7cb64a95a92859fa0c0c38fc79453e.json

Then /plugin install llm-provider@markdownregistry-neeitkenrgyxfdnl in Claude Code. It installs this exact version, the one graded A here, pinned to the commit it was graded at, and leaves it there.

---
name: llm-provider
description: Implements or modifies an LLM provider in src/llm/ by implementing the LLMProvider interface (call() + stream() methods). All provider calls route through src/llm/index.ts (llmCall, llmJsonCall). Use when user says 'add provider', 'new LLM', 'support model X', or modifies src/llm/ files. Do NOT use for calling LLM from commands—use existing llmCall/llmJsonCall instead.
---
# LLM Provider Implementation

## Critical

- **All providers MUST implement `LLMProvider` interface** from `src/llm/types.ts`: `call(params: LLMCallParams): Promise<string>` and `stream(params: LLMCallParams): AsyncGenerator<string>`.
- **Provider registration happens in `src/llm/index.ts`**: Add provider instance to `llmCall()` and `llmJsonCall()` switch statement.
- **Error handling is mandatory**: Wrap API calls in try/catch. Transient errors (timeout, rate limit, network) must be in `TRANSIENT_ERRORS` set in `src/llm/index.ts` for retry logic. Seat-based errors detected via `parseSeatBasedError()` from `src/llm/seat-based-errors.ts`.
- **Token estimation required**: Implement `estimateTokens()` helper in provider file; export for usage tracking in `src/llm/usage.ts`.
- **No direct provider instantiation in commands**: All LLM calls go through `llmCall()` or `llmJsonCall()` in `src/llm/index.ts`—commands never import provider classes directly.

## Instructions

1. **Create provider file in `src/llm/`** (e.g., `src/llm/my-provider.ts`).
   - Import `LLMProvider`, `LLMCallParams`, `LLMResponse` from `src/llm/types.ts`.
   - Import `estimateTokens` utility function (see Step 3).
   - Verify: File exists and TypeScript compiles via `npx tsc --noEmit`.

2. **Implement `LLMProvider` interface** with two methods:
   ```typescript
   export class MyProvider implements LLMProvider {
     async call(params: LLMCallParams): Promise<string> {
       try {
         // API call logic here
         // Return full response text
       } catch (error) {
         throw new Error(`MyProvider error: ${error.message}`);
       }
     }
     async *stream(params: LLMCallParams): AsyncGenerator<string> {
       try {
         // Streaming logic: yield chunks as they arrive
       } catch (error) {
         throw new Error(`MyProvider stream error: ${error.message}`);
       }
     }
   }
   ```
   - Verify: Both methods accept `LLMCallParams` (model, messages, temperature, maxTokens, system).

3. **Export `estimateTokens()` function** in provider file:
   ```typescript
   export function estimateTokensMyProvider(messages: Message[]): number {
     // Rough estimate: 1 token ≈ 4 chars
     return Math.ceil(JSON.stringify(messages).length / 4);
   }
   ```
   - Verify: Function signature matches `(messages: Message[]) => number`.

4. **Update `src/llm/index.ts`**:
   - Import provider class and `estimateTokens` function: `import { MyProvider, estimateTokensMyProvider } from './my-provider'`.
   - Create instance: `const myProvider = new MyProvider()`.
   - Add to `llmCall()` switch: `case 'my-provider': return myProvider.call(params)`.
   - Add to `llmJsonCall()` switch: `case 'my-provider': return myProvider.stream(params)`.
   - Add model to `TRANSIENT_ERRORS` if provider has transient errors: `TRANSIENT_ERRORS.add('my-model-name')`.
   - Verify: `npm run build` succeeds without type errors.

5. **Handle provider-specific errors** in `src/llm/seat-based-errors.ts`:
   - If provider has seat-based or quota errors, add detection logic to `parseSeatBasedError(error, model)`.
   - Example: Check `error.code === 'RATE_LIMIT_EXCEEDED'` for your provider.
   - Verify: Run `npm run test` — error detection tests pass.

6. **Add provider configuration** to `src/llm/config.ts`:
   - Define env var requirement: `MYSERVICE_API_KEY` or similar.
   - Export a validation function: `export function validateMyProviderConfig(): boolean { return !!process.env.MYSERVICE_API_KEY; }`.
   - Verify: Config check passes when env vars are set.

7. **Register in provider resolution** (`src/llm/config.ts` or entry point):
   - Add condition: `if (process.env.MYSERVICE_API_KEY && validateMyProviderConfig()) { return 'my-provider'; }`.
   - Verify: Provider is detected when env var is present by checking `src/llm/config.ts` provider order.

8. **Test provider locally**:
   - Run `npm run test -- src/llm/__tests__/my-provider.test.ts` (create test file in `src/llm/__tests__/`).
   - Test file must cover: `call()` success/failure, `stream()` token yielding, `estimateTokens()` accuracy.
   - Verify: `npm run test:coverage` shows >80% coverage for provider file.

## Examples

**User says**: "Add support for Claude 3.5 Sonnet via custom endpoint"

**Actions**:
1. Create `src/llm/custom-endpoint.ts` implementing `LLMProvider`.
2. In `call()` method: POST to custom endpoint with `Authorization: Bearer ${process.env.CUSTOM_ENDPOINT_KEY}`.
3. In `stream()` method: POST with `stream: true`, yield chunks from SSE response.
4. Export `estimateTokensCustomEndpoint()` using same 4-char-per-token logic.
5. Update `src/llm/index.ts`: import provider, add to switch cases.
6. Update `src/llm/config.ts`: validate `CUSTOM_ENDPOINT_URL` and `CUSTOM_ENDPOINT_KEY` env vars.
7. Register in provider resolution: check `CUSTOM_ENDPOINT_URL` before Anthropic fallback.
8. Test: `npm run test -- src/llm/__tests__/custom-endpoint.test.ts`.

**Result**: `caliber init` detects custom endpoint when `CUSTOM_ENDPOINT_URL` is set and uses it for all LLM calls.

## Common Issues

**"Provider is not registered in llmCall()"**
- Verify `src/llm/index.ts` has your provider's `case 'your-provider-name':` in both `llmCall()` and `llmJsonCall()` switch statements.
- Check: `npm run build` passes without errors.
- Fix: Ensure case names match exactly in provider registration and in `config.ts` return value.

**"ENOENT: no such file or directory in src/llm/ after adding provider"**
- The provider file path in import statement does not match actual file.
- Fix: Run `ls src/llm/` to verify file exists, then check import path matches exactly (including extension `.ts`).

**"Type error: 'MyProvider' does not implement 'LLMProvider'"**
- Missing or incorrect method signatures.
- Fix: Verify `call()` returns `Promise<string>` and `stream()` is `AsyncGenerator<string>`. Both must accept `LLMCallParams` with no required properties besides `model` and `messages`.

**"llmCall() defaults to Anthropic even when my provider env var is set"**
- Provider detection in `src/llm/config.ts` is not being reached.
- Fix: Check the condition order in `getProvider()` — your provider check must come BEFORE Anthropic check. Move your `if (process.env.YOUR_KEY)` block above `if (process.env.ANTHROPIC_API_KEY)`.

**"Transient errors are retried infinitely"**
- Error is in `TRANSIENT_ERRORS` set but provider throws it on every call.
- Fix: Verify `TRANSIENT_ERRORS` contains exact error message or error code. Add logging: `console.error('Transient error:', error.code, error.message)` to debug. Remove error from `TRANSIENT_ERRORS` if it's not actually transient (fix the underlying issue first).

**"Token estimation is wildly off, causing budget overruns"**
- The 4-char-per-token estimate is inaccurate for your model.
- Fix: Use actual token counts from provider API responses (e.g., `response.usage.input_tokens`). Update `estimateTokens()` to return cached actual count or use provider's `countTokens()` API if available. Run `npm run test -- src/llm/usage.test.ts` to validate.