llm-provider · git:20260330.4e5d8f6 · 2026-03-30 · sha256 aab0055275f839ab

llm-provider git:20260330.4e5d8f6A

Immutable. This exact content never changes and is served at /api/v1/blob/aab0055275f839ab.

No tool to install first
claude plugin marketplace add https://markdownregistry.com/mp/a/art_neeitkenrgyxfdnl/aab0055275f839ab4554721f118c391b4df7f6576d0d453fa6d1935e71a73b59.json

Then /plugin install llm-provider@markdownregistry-neeitkenrgyxfdnl in Claude Code. It installs this exact version, the one graded A here, pinned to the commit it was graded at, and leaves it there.

---
name: llm-provider
description: Adds a new LLM provider implementing LLMProvider interface with call() and stream() methods. Integrates config in src/llm/config.ts, factory in src/llm/index.ts, and error handling. Use when adding a new provider backend, integrating a model API, or extending LLM capabilities. Do NOT use for modifying existing providers or fixing provider bugs.
---
# llm-provider

## Critical

- **Interface compliance**: Every provider MUST implement `LLMProvider` from `src/llm/types.ts` with exactly two methods: `call(params)` and `stream(params)`. Both return `Promise<LLMResponse>` and `AsyncIterable<StreamChunk>` respectively.
- **No hardcoded model mappings**: Detection and selection MUST be LLM-driven or user-selected, never static. Seat-based providers (Cursor, Claude CLI) must skip validation in `validateModel()` checks.
- **Error handling**: All network/auth errors MUST be caught and wrapped in `LLMError` with code (e.g., `'auth'`, `'rate_limit'`, `'model_not_found'`). Provide actionable error messages.
- **Stream format**: Stream chunks MUST follow `{ type: 'text' | 'stop', text?: string, stop_reason?: string }`. The `stream-parser.ts` expects exact format.
- **Fast model scoping**: `ANTHROPIC_SMALL_FAST_MODEL` env var is scoped ONLY to `anthropic` and `vertex` providers in `getFastModel()`. Other providers must not read this var.

## Instructions

### Step 1: Create the provider file
**File**: `src/llm/{provider-name}.ts`

Start with the exact structure from existing providers (e.g., `anthropic.ts`, `openai-compat.ts`):

```typescript
import { LLMProvider, LLMResponse, StreamChunk, LLMError } from './types.js';

export class MyProvider implements LLMProvider {
  private apiKey: string;
  private baseUrl?: string;

  constructor(apiKey: string, baseUrl?: string) {
    if (!apiKey) throw new Error('API key required for MyProvider');
    this.apiKey = apiKey;
    this.baseUrl = baseUrl;
  }

  async call(params: {
    model: string;
    messages: Array<{ role: 'user' | 'assistant'; content: string }>;
    temperature?: number;
    max_tokens?: number;
    system?: string;
  }): Promise<LLMResponse> {
    // Implementation
  }

  async *stream(params: {
    model: string;
    messages: Array<{ role: 'user' | 'assistant'; content: string }>;
    temperature?: number;
    max_tokens?: number;
    system?: string;
  }): AsyncIterable<StreamChunk> {
    // Implementation
  }
}
```

**Verify**: Both methods exist. Constructor validates required config. No hardcoded API URLs (use env vars or constructor params).

### Step 2: Export from LLM index
**File**: `src/llm/index.ts`

Add import and factory export:

```typescript
import { MyProvider } from './my-provider.js';

export { MyProvider };
export type { /* types if needed */ };

// In the getLLMProvider() factory function:
case 'my-provider':
  return new MyProvider(
    process.env.MY_PROVIDER_API_KEY || '',
    process.env.MY_PROVIDER_BASE_URL
  );
```

**Verify**: Provider is exported. Factory case handles env var reads. No bare API keys in code.

### Step 3: Add configuration in config.ts
**File**: `src/llm/config.ts`

Add to the config object returned by `getConfig()`:

```typescript
my_provider: {
  apiKey: process.env.MY_PROVIDER_API_KEY,
  baseUrl: process.env.MY_PROVIDER_BASE_URL,
  isSeatBased: false, // or true if user-seat-licensed (e.g., Cursor)
  requiresAuth: true,
}
```

If the provider has a fast-model option, add to `getFastModel()`:

```typescript
if (provider === 'my-provider') {
  return process.env.MY_PROVIDER_FAST_MODEL || 'my-provider-fast-default';
}
```

**Verify**: Config key matches case in factory. Env vars are consistent across both files. `isSeatBased` is set correctly.

### Step 4: Implement error handling
**File**: `src/llm/my-provider.ts`

Wrap API calls in try/catch. Map errors to `LLMError`:

```typescript
try {
  const response = await fetch(`${this.baseUrl}/chat`, {
    method: 'POST',
    headers: { 'Authorization': `Bearer ${this.apiKey}`, 'Content-Type': 'application/json' },
    body: JSON.stringify(payload),
  });

  if (!response.ok) {
    if (response.status === 401) {
      throw new LLMError('Invalid API key', 'auth');
    }
    if (response.status === 429) {
      throw new LLMError('Rate limit exceeded', 'rate_limit');
    }
    if (response.status === 404) {
      throw new LLMError(`Model ${params.model} not found`, 'model_not_found');
    }
    throw new LLMError(`HTTP ${response.status}: ${await response.text()}`, 'unknown');
  }
  // Parse and return
} catch (err) {
  if (err instanceof LLMError) throw err;
  throw new LLMError(`Provider error: ${err.message}`, 'unknown');
}
```

**Verify**: All HTTP error codes (401, 429, 404, 5xx) are handled. Network errors are caught. Messages are clear and actionable.

### Step 5: Implement stream() with correct format
**File**: `src/llm/my-provider.ts`

Yield chunks in the exact format expected by `stream-parser.ts`:

```typescript
async *stream(params: {...}): AsyncIterable<StreamChunk> {
  const response = await fetch(`${this.baseUrl}/chat/stream`, { /* ... */ });
  const reader = response.body?.getReader();
  if (!reader) throw new LLMError('Stream not available', 'unknown');

  let buffer = '';
  const decoder = new TextDecoder();

  try {
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      buffer += decoder.decode(value, { stream: true });
      const lines = buffer.split('\n');
      buffer = lines.pop() || '';

      for (const line of lines) {
        if (!line || line.startsWith(':')) continue; // SSE comments
        if (line.startsWith('data: ')) {
          const data = JSON.parse(line.slice(6));
          if (data.choices?.[0]?.delta?.content) {
            yield { type: 'text', text: data.choices[0].delta.content };
          }
          if (data.choices?.[0]?.finish_reason) {
            yield { type: 'stop', stop_reason: data.choices[0].finish_reason };
          }
        }
      }
    }
  } finally {
    reader.releaseLock();
  }
}
```

**Verify**: Stream yields chunks with correct type field. Stop reason is included on finish. Buffer handles partial lines correctly.

### Step 6: Write tests
**File**: `src/llm/__tests__/my-provider.test.ts`

Add minimal tests covering both methods:

```typescript
import { describe, it, expect, vi } from 'vitest';
import { MyProvider } from '../my-provider.js';

describe('MyProvider', () => {
  it('throws if API key is missing', () => {
    expect(() => new MyProvider('')).toThrow('API key required');
  });

  it('call() returns LLMResponse with text', async () => {
    vi.stubGlobal('fetch', vi.fn(() =>
      Promise.resolve(new Response(JSON.stringify({
        choices: [{ message: { content: 'Hello' } }],
      })))
    ));

    const provider = new MyProvider('test-key');
    const response = await provider.call({
      model: 'gpt-5',
      messages: [{ role: 'user', content: 'Hi' }],
    });

    expect(response.text).toBe('Hello');
  });

  it('stream() yields text chunks', async () => {
    // Mock streaming response
    vi.stubGlobal('fetch', vi.fn(() =>
      Promise.resolve(new Response(new ReadableStream({ /* ... */ })))
    ));

    const provider = new MyProvider('test-key');
    const chunks = [];
    for await (const chunk of provider.stream({
      model: 'gpt-5',
      messages: [{ role: 'user', content: 'Hi' }],
    })) {
      chunks.push(chunk);
    }

    expect(chunks.some(c => c.type === 'text')).toBe(true);
  });
});
```

**Verify**: Tests run without errors. `npm run test -- src/llm/__tests__/my-provider.test.ts` passes.

### Step 7: Update types if needed
**File**: `src/llm/types.ts`

If your provider needs additional config fields, extend `ProviderConfig`:

```typescript
export interface ProviderConfig {
  // ... existing fields
  myProvider?: {
    apiKey: string;
    baseUrl?: string;
    customField?: string;
  };
}
```

**Verify**: Type is used consistently in `config.ts` and `index.ts`.

## Examples

### User request:
"Add support for Anthropic's Vertex AI provider."

### Actions taken:

1. Create `src/llm/vertex.ts` implementing `LLMProvider` with `call()` and `stream()` using Vertex SDK.
2. Export from `src/llm/index.ts`: `export { VertexProvider }`.
3. Add factory case in `getLLMProvider()`: `case 'vertex': return new VertexProvider(projectId, location, ...)`
4. Update `src/llm/config.ts` with Vertex env vars and `isSeatBased: false`.
5. Implement error handling for Vertex-specific errors (auth, model not found, quota).
6. Implement `stream()` yielding chunks from Vertex streaming API.
7. Write tests in `src/llm/__tests__/vertex.test.ts` for both methods.
8. Run `npm run test -- src/llm/__tests__/vertex.test.ts` to verify.

### Result:
Caliber now supports Vertex AI. Users can set `VERTEX_PROJECT_ID` and `VERTEX_LOCATION`, then select Vertex as their LLM provider during `caliber init`.

## Common Issues

**Error: "Cannot find module './my-provider.js'"**
- ESM imports require `.js` extensions. Add `.js` to all relative imports in `index.ts`.
- Verify file exists at exact path.

**Error: "LLMError is not exported"**
- Import from `src/llm/types.js` not `src/llm.js`.
- Correct: `import { LLMError } from './types.js';`

**Stream yields empty text or stops early**
- Verify buffer handling in stream loop. Lines must be split on `\n` and incomplete lines stored in buffer.
- Check API response format. Some APIs use Server-Sent Events (`data: `), others use line-delimited JSON.
- Test with `curl` first: `curl -N https://api/stream` to see raw format.

**"Rate limit exceeded" error on every call**
- Verify API key is valid and has quota remaining. Check provider dashboard.
- If seat-based, ensure you have a valid license (Cursor, Claude CLI).
- Check if `isSeatBased: true` is set correctly — seat-based providers bypass rate limit checks in some contexts.

**Provider not appearing in `caliber init` flow**
- Verify provider is exported from `src/llm/index.ts`.
- Check `getLLMProvider()` case statement matches provider name.
- Verify config key exists in `src/llm/config.ts`.
- Run `caliber status` to see if provider is detected.

**Tests fail with "fetch is not defined"**
- Use `vi.stubGlobal('fetch', ...)` in Vitest to mock fetch.
- Or import a fetch polyfill like `node-fetch` if targeting Node < 18.
- Verify `src/test/setup.ts` doesn't stub fetch globally (it shouldn't).