online-evals · v0.2.0 · 2026-10-26 · sha256 88c45d07bd021aae
online-evals v0.2.0A
Immutable. This exact content never changes and is served at /api/v1/blob/88c45d07bd021aae.
---
name: online-evals
description: Attach judges to config variations for automatic LLM-as-a-judge evaluation. Create custom judges, configure sampling rates, and monitor quality scores.
compatibility: Requires the remotely hosted LaunchDarkly MCP server. Scripted setup without MCP needs an API access token with ai-configs:write permission. SDK examples use `launchdarkly-ai-server` 0.3.0+ (Python) or `@launchdarkly/ai-node` 0.3.0+ (Node.js).
metadata:
author: launchdarkly
version: "0.2.0"
---
# Config Online Evaluations
Attach judges to config variations for automatic quality scoring using LLM-as-a-judge methodology. Judges evaluate responses and return scores between 0.0 and 1.0.
## Prerequisites
- LaunchDarkly account with AgentControl enabled
- The remotely hosted LaunchDarkly MCP server configured in your environment
- Existing config with variations (use `configs-create` skill)
- To run judges from your application: `launchdarkly-ai-server` 0.3.0+ (Python) or `@launchdarkly/ai-node` 0.3.0+ (Node.js), plus a provider handler package for each provider your configs and judges use
## API Access
Use the LaunchDarkly MCP server tools for every step in the workflow: they authenticate via OAuth and need no token. The REST API is only for scripts and CI that can't use MCP (see [Scripting without MCP](#scripting-without-mcp)). In that case, ask the user to paste an API access token for this session and substitute it for `{api_token}` in the examples below. Do not search environment variables, `.env` files, or agent/MCP config files for credentials.
## Core Concepts
### What Are Judges?
Judges are specialized configs in **judge mode** that evaluate responses from other configs. They use an LLM to score outputs and return structured results:
```json
{
"score": 0.85,
"reasoning": "Answered correctly with one minor omission"
}
```
### Built-in Judges
LaunchDarkly provides three pre-configured judges:
| Judge | Metric Key | Measures |
|-------|-----------|----------|
| Accuracy | `$ld:ai:judge:accuracy` | How correct and grounded the response is |
| Relevance | `$ld:ai:judge:relevance` | How well it addresses the user request |
| Toxicity | `$ld:ai:judge:toxicity` | Harmful or unsafe phrasing (lower = safer) |
### Completion Mode Only
Judges can only be attached to **completion mode** configs in the UI. Attaching them through the API (Step 2) works for any config, and the SDK runs whatever judges the served variation has.
### Restrictions
- Cannot attach judges to judges (no recursion)
- Cannot attach multiple judges with the same metric key to a single variation
- Cannot view/edit model parameters or tools on judge variations
## Workflow
### Step 1: Create Custom Judges (Optional)
For domain-specific evaluation, create a judge config with `create-agentcontrol-config`:
```json
{
"projectKey": "{projectKey}",
"key": "security-judge",
"name": "Security Judge",
"mode": "judge",
"evaluationMetricKey": "$ld:ai:judge:security",
"isInverted": false
}
```
> **Note:** `evaluationMetricKey` must start with `$ld:ai:judge:`. Set `isInverted: true` for metrics like toxicity where 0.0 is better.
Then add a variation with the evaluation prompt using `create-agentcontrol-config-variation`. Get `modelConfigKey` from `list-agentcontrol-model-configs`:
```json
{
"projectKey": "{projectKey}",
"configKey": "security-judge",
"key": "default",
"name": "Default",
"modelConfigKey": "OpenAI.gpt-4o-mini",
"modelName": "gpt-4o-mini",
"parameters": {"temperature": 0.3},
"messages": [
{
"role": "system",
"content": "You are a security auditor. Score from 0.0 to 1.0:\n- 1.0: No security issues\n- 0.7-0.9: Minor issues\n- 0.4-0.6: Moderate issues\n- 0.1-0.3: Serious vulnerabilities\n- 0.0: Critical vulnerabilities\n\nCheck for: SQL injection, XSS, hardcoded secrets, command injection.\n\n{{message_history}}"
}
]
}
```
> **Important:** End the judge prompt with `{{message_history}}`. The SDK fills that variable with the conversation being judged, so a prompt without it never sees the response it's scoring. The SDK adds the score-and-reasoning output format itself.
### Step 2: Attach Judges to Variations
Use `update-agentcontrol-config-variation` with `judgeConfiguration`. Get `variationKey` from `get-agentcontrol-config`:
```json
{
"projectKey": "{projectKey}",
"configKey": "{configKey}",
"variationKey": "{variationKey}",
"judgeConfiguration": {
"judges": [
{"judgeConfigKey": "security-judge", "samplingRate": 1.0},
{"judgeConfigKey": "api-contract-judge", "samplingRate": 0.5}
]
}
}
```
> **Important:** The `judges` array **replaces all existing** judge attachments. An empty array removes all judges.
### Step 3: Set Fallthrough on Judges
Each judge config needs its fallthrough set to the enabled variation in every environment that runs it. Configs default to the "disabled" variation. Use `update-agentcontrol-config-rollout`:
```json
{
"projectKey": "{projectKey}",
"configKey": "security-judge",
"env": "production",
"rolloutType": "variation",
"variationKey": "default"
}
```
Then run `get-agentcontrol-config-health` on the judge to confirm it's targetable.
## Scripting without MCP
For CI or scripts that can't use the MCP server, the same three steps through the REST API. `turnTargetingOn` does not work for configs, so `set_fallthrough` sends `updateFallthroughVariationOrRollout`:
```python
import requests
import os
from typing import Optional
class AIConfigJudges:
"""Manager for config judge attachments"""
def __init__(self, api_token: str, project_key: str):
self.api_token = api_token
self.project_key = project_key
self.base_url = "https://app.launchdarkly.com/api/v2"
self.headers = {
"Authorization": api_token,
"Content-Type": "application/json",
"LD-API-Version": "beta"
}
def attach_judges(self, config_key: str, variation_key: str,
judges: list[dict]) -> dict:
"""
Attach judges to a variation.
Args:
config_key: config key
variation_key: Variation key
judges: List of {"judgeConfigKey": str, "samplingRate": float}
"""
url = f"{self.base_url}/projects/{self.project_key}/ai-configs/{config_key}/variations/{variation_key}"
response = requests.patch(url, headers=self.headers, json={
"judgeConfiguration": {"judges": judges}
})
if response.status_code == 200:
print(f"[OK] Attached {len(judges)} judges to {config_key}/{variation_key}")
return response.json()
print(f"[ERROR] {response.status_code}: {response.text}")
return {}
def create_judge(self, key: str, name: str, metric_key: str,
system_prompt: str, model: str = "OpenAI.gpt-4o-mini",
is_inverted: bool = False) -> dict:
"""
Create a judge config.
Args:
key: Judge config key
name: Display name
metric_key: Metric key for scoring; "security" is sent as "$ld:ai:judge:security"
system_prompt: Evaluation instructions; {{message_history}} is appended if missing
is_inverted: True if lower scores are better (e.g., toxicity)
"""
# The API rejects a bare metric key
if not metric_key.startswith("$ld:ai:judge:"):
metric_key = f"$ld:ai:judge:{metric_key}"
# Create config
config_url = f"{self.base_url}/projects/{self.project_key}/ai-configs"
response = requests.post(config_url, headers=self.headers, json={
"key": key,
"name": name,
"mode": "judge",
"evaluationMetricKey": metric_key,
"isInverted": is_inverted
})
if response.status_code not in [200, 201]:
print(f"[ERROR] Creating config: {response.text}")
return {}
# Without {{message_history}} the judge never sees the conversation it scores
if "{{message_history}}" not in system_prompt:
system_prompt = f"{system_prompt}\n\n{{{{message_history}}}}"
# Create variation
var_url = f"{self.base_url}/projects/{self.project_key}/ai-configs/{key}/variations"
response = requests.post(var_url, headers=self.headers, json={
"key": "default",
"name": "Default",
"messages": [{"role": "system", "content": system_prompt}],
"modelConfigKey": model,
"model": {"parameters": {"temperature": 0.3}}
})
if response.status_code in [200, 201]:
print(f"[OK] Created judge: {key}")
return response.json()
print(f"[ERROR] Creating variation: {response.text}")
return {}
def set_fallthrough(self, config_key: str, environment: str,
variation_key: str = "default") -> bool:
"""
Set fallthrough to enable a judge config.
Note: turnTargetingOn doesn't work for configs. Instead, set the
fallthrough from disabled (index 0) to the enabled variation.
"""
# Get variation ID
url = f"{self.base_url}/projects/{self.project_key}/ai-configs/{config_key}/targeting"
response = requests.get(url, headers=self.headers)
if response.status_code != 200:
print(f"[ERROR] {response.status_code}: {response.text}")
return False
targeting = response.json()
variation_id = None
for var in targeting.get("variations", []):
# The variation key is in value._ldMeta.variationKey, not on the item itself
meta_key = ((var.get("value") or {}).get("_ldMeta") or {}).get("variationKey")
if meta_key == variation_key or var.get("name") == variation_key:
variation_id = var.get("_id")
break
if not variation_id:
print(f"[ERROR] Variation '{variation_key}' not found")
return False
# Set fallthrough
response = requests.patch(url, headers={
**self.headers,
"Content-Type": "application/json; domain-model=launchdarkly.semanticpatch"
}, json={
"environmentKey": environment,
"instructions": [{
"kind": "updateFallthroughVariationOrRollout",
"variationId": variation_id
}]
})
if response.status_code == 200:
print(f"[OK] Fallthrough set for {config_key}")
return True
print(f"[ERROR] {response.status_code}: {response.text}")
return False
```
## SDK: Inline Evaluation
`invoke()` runs every judge attached to the served variation, samples each at its rate, and records the scores as metrics. Pass a handler for each provider your config and its judges use. The config must be completion mode: a messages handler does not match an agent-mode config.
```python
import asyncio
from launchdarkly_ai_server import config, shutdown
from launchdarkly_ai_openai_messages import create_openai_messages_handler
config_key = "support-agent" # completion mode
async def main():
# Reads LD_SDK_KEY and OPENAI_API_KEY from the environment
result = await config(
key=config_key,
handler=create_openai_messages_handler(),
).invoke("How can LaunchDarkly help me?", {"kind": "user", "key": "user-123"})
print(result.response)
for judge_key, judge in (result.judge_results or {}).items():
print(judge_key, judge.score)
# Flushes trailing events, including judge scores
await shutdown()
asyncio.run(main())
```
```typescript
import { config, shutdown } from '@launchdarkly/ai-node';
import { createOpenAIHandler } from '@launchdarkly/ai-openai-messages';
const result = await config({
key: 'support-agent',
handler: createOpenAIHandler(),
}).invoke('How can LaunchDarkly help me?', { kind: 'user', key: 'user-123' });
console.log(result.response);
for (const [judgeKey, judge] of Object.entries(result.judgeResults ?? {})) {
console.log(judgeKey, judge.score);
}
await shutdown();
```
Inline judges add their latency to the request. To keep them off the request path, run them in the background.
## SDK: Background Evaluation
Pass `skip_judges=True` (`skipJudges: true`) and `invoke()` returns the sampled judges as `judge_tasks` (`judgeTasks`) instead of running them. Hand each task to a worker, run it with `run_judge`, and record the score yourself, because `run_judge` doesn't:
```python
from ldclient import Context
from launchdarkly_ai_server import config, get_client, run_judge
from launchdarkly_ai_openai_messages import create_openai_messages_handler
handler = create_openai_messages_handler()
config_key = "support-agent" # completion mode
result = await config(key=config_key, handler=handler, skip_judges=True).invoke(
"How can LaunchDarkly help me?", {"kind": "user", "key": "user-123"}
)
# In your worker:
for task in result.judge_tasks or []:
judged = await run_judge(task, [handler])
if judged and task.evaluation_metric_key:
get_client().track(
task.evaluation_metric_key,
Context.from_dict(task.user_context),
judged.track_data,
judged.score,
)
```
```typescript
import { config, getClient, runJudge } from '@launchdarkly/ai-node';
import { createOpenAIHandler } from '@launchdarkly/ai-openai-messages';
const handler = createOpenAIHandler();
const result = await config({ key: 'support-agent', handler, skipJudges: true }).invoke(
'How can LaunchDarkly help me?',
{ kind: 'user', key: 'user-123' },
);
// In your worker:
for (const task of result.judgeTasks ?? []) {
const judged = await runJudge(task, [handler]);
if (judged && task.evaluationMetricKey) {
getClient().track(task.evaluationMetricKey, task.userContext, judged.trackData, judged.score);
}
}
```
> **Note:** In Python, `task.user_context` is a plain dict, so convert it with `Context.from_dict()` before calling `track()`. `track_data` (`trackData`) already includes `judgeConfigKey`.
## SDK: Custom Pipelines
If your application calls the model itself, wrap that call in a handler with `create_handler` (`createHandler`). `invoke()` then runs your code and the attached judges, the same as with a built-in handler. The judges still need a handler for their own provider:
```python
from launchdarkly_ai_server import config, create_handler, parse_template
from launchdarkly_ai_openai_messages import create_openai_messages_handler
config_key = "support-agent" # completion mode
# run_my_agent is your existing pipeline; it returns the text and the provider's usage object
async def my_pipeline(ld_config, user_input, tool_handlers, variables, history):
# Custom handlers fill {{name}} placeholders themselves
messages = [{**m, "content": parse_template(m["content"], variables)} for m in ld_config.get("messages", [])]
output, usage = await run_my_agent(messages, user_input)
return {"output": output, "usage": {"input": usage.input_tokens, "output": usage.output_tokens}}
handlers = [
create_handler(("Anthropic", "messages"), my_pipeline), # the config's provider and mode
create_openai_messages_handler(), # the judge's provider
]
result = await config(key=config_key, handler=handlers).invoke(user_input, context)
```
The provider name in `create_handler` must match the config's model provider exactly, including case, and the first argument must be a tuple. For the full handler contract, see the `tools` skill.
## Sampling Rates
Each evaluated response sends an additional request to your model provider, increasing token usage and costs. Start with a lower sampling percentage and increase only if you need more evaluation coverage.
You can adjust sampling rates at any time from the Judges section of a variation, or disable a judge by setting its sampling to 0%.
## Viewing Results
1. Navigate to **configs** > select your config
2. Click **Monitoring** tab
3. Select **Evaluator metrics** from dropdown
4. View scores by variation and time range
Results appear within 1-2 minutes of evaluation.
## Use in Guardrails and Experiments
Evaluation metrics integrate with:
- **Guarded rollouts**: Pause/revert when scores fall below threshold
- **Experiments**: Compare variations using evaluation metrics as goals
## Error Handling
| Status | Cause | Solution |
|--------|-------|----------|
| 404 | Config/variation not found | Verify keys exist |
| 400 | Invalid judge config | Check judgeConfigKey exists |
| 403 | Insufficient permissions | Reconnect the MCP server, or check the API token's permissions |
| 422 | Duplicate metric key | Cannot attach multiple judges with same metric key |
## Next Steps
After attaching judges:
1. **Set fallthrough** on judge configs to an enabled variation (required)
2. **Monitor results** in Monitoring tab
3. **Adjust sampling** based on cost/coverage needs
4. **Set up guarded rollouts** for automatic regression detection
## Related Skills
- `configs-create` - Create configs and judges
- `configs-targeting` - Configure targeting rules
- `configs-variations` - Manage variations
## References
- [Online Evaluations](https://docs.launchdarkly.com/home/ai-configs/online-evaluations.md)
- [Custom Judges](https://docs.launchdarkly.com/home/ai-configs/custom-judges.md)
- [AI SDK reference for Python](https://launchdarkly.com/docs/sdk/ai/python)
- [AI SDK reference for Node.js](https://launchdarkly.com/docs/sdk/ai/node-js)