llm-provider · git:20260323.a091579 · 2026-03-23 · sha256 21cfb9dd93f9134b
llm-provider git:20260323.a091579A
Immutable. This exact content never changes and is served at /api/v1/blob/21cfb9dd93f9134b.
No tool to install first
claude plugin marketplace add https://markdownregistry.com/mp/a/art_neeitkenrgyxfdnl/21cfb9dd93f9134b01820454b7c66935f601ffe66e51cc06dc847bafdf4ee9e7.json
Then /plugin install llm-provider@markdownregistry-neeitkenrgyxfdnl in Claude Code. It installs this exact version, the one graded A here, pinned to the commit it was graded at, and leaves it there.
---
name: llm-provider
description: Implements or modifies an LLM provider in src/llm/ by implementing the LLMProvider interface (call() + stream() methods) from src/llm/types.ts. All provider calls route through src/llm/index.ts (llmCall, llmJsonCall). Use when user says 'add provider', 'new LLM', 'support model X', or modifies src/llm/ files. Do NOT use for calling LLM from commands — use existing llmCall/llmJsonCall instead.
---
# LLM Provider Implementation
## Critical
1. **All new providers MUST implement the `LLMProvider` interface** from `src/llm/types.ts`:
```typescript
interface LLMProvider {
call(req: LLMRequest): Promise<LLMResponse>;
stream(req: LLMRequest): AsyncGenerator<LLMChunk, void>;
}
```
2. **Register provider in `src/llm/index.ts`** via the `llmCall()` and `llmJsonCall()` exports — this is the only entrypoint commands use.
3. **Handle `TRANSIENT_ERRORS`** (network, rate-limit, timeout) by catching and re-throwing as `Error` with message matching pattern in `src/llm/model-recovery.ts`.
4. **Never catch `SeatBasedError`** — let it propagate; use `parseSeatBasedError()` from `src/llm/seat-based-errors.ts` in caller if needed.
5. **Verify token estimation** with `estimateTokens()` from `src/llm/utils.ts` for cost tracking via `trackUsage()` in `src/llm/usage.ts`.
## Instructions
1. **Create provider file in `src/llm/`** (e.g., `src/llm/my-provider.ts`).
- Verify it exports a class implementing `LLMProvider`.
- Validate constructor accepts config from `src/llm/config.ts` (type `LLMConfig`).
2. **Implement `call(req: LLMRequest): Promise<LLMResponse>`**.
- Extract request fields: `messages`, `model`, `maxTokens`, `temperature`, `systemPrompt`, `jsonMode`.
- Call provider API with correct field mappings (e.g., Anthropic uses `max_tokens`, OpenAI uses `max_tokens`).
- On success: return `{ content: string, inputTokens: number, outputTokens: number, model: string }`.
- On transient error: throw `Error` with message containing 'timeout', 'rate limit', or 'connection'.
- On auth/quota error: throw error with message matching `seat-based-errors.ts` patterns (e.g., 'quota', 'billing').
- Verify response parsing with `extractJson()` from `src/llm/utils.ts` if `jsonMode: true`.
3. **Implement `stream(req: LLMRequest): AsyncGenerator<LLMChunk, void>`**.
- Yield chunks as `{ delta: string, inputTokens?: number, outputTokens?: number }`.
- Track final token counts; yield final chunk with cumulative counts.
- On error: throw the same error types as `call()`.
- Use `src/ai/stream-parser.ts` to parse tool calls if needed.
4. **Export provider from `src/llm/index.ts`**.
- Add import: `import { MyProvider } from './my-provider'`.
- Update `getProvider()` function to instantiate your provider based on config type.
- Verify `llmCall()` and `llmJsonCall()` delegate to `provider.call()`.
5. **Add provider to `src/llm/config.ts`** if it requires new config fields.
- Update `LLMConfig` type union to include new provider config type.
- Export type so callers can pass correct config shape.
6. **Test with `vitest`** in `src/llm/__tests__/`.
- Mock API calls; verify `call()` returns correct `LLMResponse` shape.
- Verify `stream()` yields correct `LLMChunk` objects.
- Test transient error handling: mock network timeout, verify error message matches `TRANSIENT_ERRORS` pattern.
- Run `npm run test -- src/llm/__tests__/my-provider.test.ts` to validate.
## Examples
**User**: "Add support for OpenAI-compatible endpoints."
**Actions**:
1. Create `src/llm/openai-compat.ts` (already exists as reference).
2. Import `LLMProvider`, `LLMRequest`, `LLMResponse`, `LLMConfig` from `src/llm/types.ts`.
3. Define class `OpenAICompatProvider implements LLMProvider`.
4. In `call()`: map `req.messages` to OpenAI format, call provider, extract tokens, return `LLMResponse`.
5. In `stream()`: iterate provider stream, yield `LLMChunk` per delta.
6. Export from `src/llm/index.ts`; update `getProvider()` to instantiate based on config.
7. Test: `npm run test -- src/llm/__tests__/openai-compat.test.ts`.
**Result**: Commands using `llmCall({ messages, model })` automatically route to OpenAI-compatible endpoint based on config; token counts tracked via `trackUsage()`.
## Anti-patterns
1. **DO NOT** call provider API directly from commands — use `llmCall()` from `src/llm/index.ts`. CORRECT: `import { llmCall } from './llm'; const res = await llmCall({ messages, model });`
2. **DO NOT** handle `SeatBasedError` inside provider — let it bubble up. Commands catch it via `try/catch` and call `parseSeatBasedError()`. CORRECT: throw raw error; let caller parse with `parseSeatBasedError(error.message)`.
3. **DO NOT** forget to track usage for cost reporting — always return `inputTokens` and `outputTokens` in `LLMResponse`. CORRECT: after API call, estimate tokens with `estimateTokens()` or extract from provider response, call `trackUsage()` in caller context.
## Common Issues
- **Error: "Provider not registered in llmCall()"**
- Verify import statement in `src/llm/index.ts`: `import { MyProvider } from './my-provider'`.
- Verify `getProvider()` function instantiates your provider: `if (config.type === 'my-provider') return new MyProvider(config)`.
- Run `npm run build` and verify no TypeScript errors.
- **Error: "Transient error not caught, request retried infinitely"**
- Check error message from provider matches one of `TRANSIENT_ERRORS` patterns in `src/llm/model-recovery.ts` (e.g., 'timeout', 'ECONNREFUSED', 'rate_limit_exceeded').
- If not: wrap provider error in `new Error()` with message containing transient keyword, e.g., `throw new Error('timeout calling provider')`.
- Verify `model-recovery.ts` retry logic kicks in: `npm run test -- src/llm/__tests__/model-recovery.test.ts`.
- **Error: "Token count mismatch, usage not tracked"**
- Verify `call()` returns `inputTokens` and `outputTokens` as numbers (not strings).
- If provider doesn't return token counts: use `estimateTokens(messages, model)` from `src/llm/utils.ts` to estimate before API call.
- Verify caller invokes `trackUsage()` after `llmCall()` succeeds (check `src/ai/generate.ts` for pattern).