---
name: error-troubleshooting
description: Diagnose and fix common errors in AIRT attack generation, execution, and results collection
allowed-tools: generate_attack generate_agentic_attack execute_workflow inspect_results list_workflows
---

# Error Troubleshooting

Common errors and fixes for AIRT attack workflows.

## Model Errors

### "Model not found" / "Invalid model"
- **Cause**: Bare model name without provider prefix
- **Fix**: Always use `provider/model` format: `openai/gpt-4o`, `anthropic/claude-sonnet-4-20250514`
- **Aliases**: Use short aliases like `gpt-4o`, `claude`, `groq` — they auto-resolve

### "Rate limit exceeded" / 429
- **Cause**: Too many concurrent requests to the provider
- **Fix**: Reduce `n_iterations` or switch to a less rate-limited provider
- **For Groq**: Groq has strict rate limits. Use `groq/llama-3.3-70b-versatile` for lower limits

### "Authentication failed" / 401
- **Cause**: Missing or invalid API key for the provider
- **Fix**: Ensure the correct API key is set as a platform secret (OPENAI_API_KEY, ANTHROPIC_API_KEY, GROQ_API_KEY, etc.)
- **Check**: API keys are pre-configured in the environment. If missing, ask the user to add them via Settings > Secrets

### "Context length exceeded"
- **Cause**: Conversation history too long for model's context window
- **Fix**: Reduce `context_depth` or `n_iterations`, or switch to a model with larger context

## Transform Errors

### "Transform not found: <name>"
- **Cause**: Wrong transform name. Names must match TRANSFORM_DEFS exactly.
- **Fix**: Check the transform-reference skill for exact names
- **Common mistakes**: `base64_encode` → `base64`, `rot13_cipher` → `rot13`, `caesar_cipher` → `caesar`

### "Transform requires LLM but no transform_model"
- **Cause**: LLM-powered transforms (persuasion, language) need a model
- **Fix**: Set `transform_model` parameter, or it defaults to `attacker_model`
- **LLM transforms**: `adapt_language`, `code_switch`, `dialectal_variation`, `transliterate`, all persuasion, `role_play_wrapper`, `cognitive_hacking`, `skeleton_key_framing`, `many_shot_examples`

### "Invalid parameter for transform"
- **Cause**: Wrong parameter syntax
- **Fix**: Use parentheses: `caesar(5)`, `adapt_language(Zulu)`, `vigenere(SECRET)`, `affine(5,8)`

## Category Attack Errors

### "Unknown attack: 't'" / "Unknown attack: '['" (single characters)
- **Cause**: The `attacks` argument to `generate_category_attack` was iterated
  character-by-character. This happened when a bare string was passed and the
  runner looped over it directly (e.g. `"tap"` -> `'t'`, `'a'`, `'p'`).
- **Fix**: The runner now normalizes `attacks` via `_normalize_attack_names`,
  accepting a list (`["tap", "goat"]`), a comma-separated string
  (`"tap,goat"`), or a single name (`"tap"`). If you still see single-character
  attack errors, you are on an old build — update the capability.
- **Workaround (older builds)**: Run the category via per-goal `generate_attack`
  calls with `goal_category=<slug>` instead of `generate_category_attack`.
- **Signature to recognize**: the error lists all valid attacks but complains
  about a one-character name. That always means an iterable-splitting bug, not a
  genuinely unknown attack.

## Scorer Errors

### "Scorer not found: <name>"
- **Cause**: Wrong scorer name
- **Fix**: Check the scorer-reference skill for exact names
- **Common mistakes**: `memory_injection_detected` → `memory_injection`, `goal_hijack_detected` → `goal_hijacking`

## Execution Errors

### "Workflow execution timed out"
- **Cause**: Attack took longer than timeout (default 300s, max 600s)
- **Fix**: Increase timeout with `execute_workflow(filename, timeout=600)`, or reduce `n_iterations`
- **For long attacks**: `rainbow`, `autodan`, `beast` can take 10+ minutes. Use max timeout.

### "Syntax error in generated script"
- **Cause**: Code generation bug
- **Fix**: Regenerate with slightly different parameters. If persistent, try a different attack type.

### "ImportError: cannot import"
- **Cause**: SDK version mismatch or missing module
- **Fix**: This is an environment issue. Try a different attack that uses different SDK modules.

### "Connection refused" / "Network error"
- **Cause**: Sandbox can't reach the API or LLM provider
- **Fix**: Check SANDBOX_SERVER_URL configuration. For local dev, use `http://host.docker.internal:8000`

## Agentic Attack Errors

### "Agent endpoint returned 4xx/5xx"
- **Cause**: Wrong agent URL, auth, or request format
- **Fix**: Verify `agent_url`, `agent_auth_type`, and `agent_preset`. Test the endpoint manually first.

### "Could not extract response text"
- **Cause**: `agent_response_text_path` doesn't match the actual response structure
- **Fix**: Use `agent_preset` if the agent follows OpenAI/Anthropic format. For custom APIs, provide the correct JSONPath.

### "No tool calls in response"
- **Cause**: Agent didn't invoke tools (may not be a tool-using agent)
- **Fix**: Ensure the agent is tool-enabled. Use more aggressive transforms like `tool_restriction_bypass`.

## Results Issues

### "No analytics files found"
- **Cause**: Attack didn't complete or output directory is wrong
- **Fix**: Check `inspect_results(file_type="all")` to see what files exist

### "ASR is 0.0 but trials ran"
- **Cause**: No trials scored above threshold (0.5 default)
- **Fix**: This means the target defended well. Try different attacks or transforms.

### "0 trials completed"
- **Cause**: All trials errored or timed out
- **Fix**: Check for model/network errors. Reduce complexity (fewer transforms, simpler attack).

## Provisioning & Sandbox Errors

For the full lifecycle, load the `provisioning-and-lifecycle` skill.

### "Task not found" / 404 provisioning a bundled target
- **Cause**: A bare task name resolves only within the caller's org, but bundled targets (`ml-extraction-*`, `*-mesh`) live in the public `dreadnode/` catalog.
- **Fix**: Use `provision_environment` with the BARE name - it catches the 404 and retries as `<org>/<name>` automatically. Never shell out to `dreadnode env`.
- **Do NOT**: qualify a bundled task with your own org (e.g. `aisf-learner-aug-2026/ml-extraction-imdb-text`) - it is not there and there is no fallback. Use the bare name or `dreadnode/<name>`.

### "404" on GET environments/<id>/status
- **Cause**: The sandbox is already torn down or expired - you are polling a dead environment.
- **Fix**: Treat it as terminal ("terminated"), stop polling, and do not surface it as an error.

### Endpoint mismatch 404 (/attack vs /predict)
- **Cause**: Probing the wrong endpoint for the target type - e.g. fetching `/attack` on an ML classifier that only serves `/predict`.
- **Fix**: Read the `>>> NEXT STEP` line from `provision_environment`. Classifier -> `/predict`; mesh -> `/attack`. Never probe both.

### Sandbox lifecycle & billing (avoid leaks)
- **Cause**: A hosted sandbox bills for its whole lifetime; forgetting teardown leaks cost until TTL.
- **Fix**: Finish with `update_assessment_status` (auto-teardown fires on `completed` and `failed`), or call `teardown_environment()` on early abort. Set `AIRT_ENV_TEARDOWN_GRACE_SEC` >= your longest attack timeout so teardown does not kill an in-flight attack.

### Transient vs fatal (Note: vs Error:)
- **`Note:` prefix** = transient network fault (TLS/timeout/conn-reset/502-504) already auto-retried; it did not affect running or recorded work - just re-run the step, do not report failure.
- **`Error:` prefix** = a non-fatal input/tool issue - adjust params, do not blind-retry.

## Retry Strategy

1. **First failure**: Read the error message, adjust the specific parameter that failed
2. **Second failure**: Try a different model alias (e.g., full path instead of short alias)
3. **Third failure**: Simplify — fewer transforms, different attack type, lower iterations
4. **Still failing**: Try the simplest possible config: `generate_attack(attack_type="prompt", goal=..., target_model="openai/gpt-4o")` to verify basic connectivity