AGENTS.md git:20260907.62c0f3cA

Immutable. This exact content is served forever at /api/v1/blob/b87587cccd8aa158.

# AGENTS.md

`hermes-jailbench` is a deterministic jailbreak regression benchmark for LLM endpoints.

## Use it for

- rerunning known jailbreak patterns after a model or prompt change
- producing a structured refusal, partial, and complied report
- smoke-testing the CLI with `--demo` or `--dry-run` before using real credentials
- exercising the full client/retry/scorer path offline against `python -m hermes_jailbench.mock_target` (loopback only, no key)

## Do not use it for

- claiming a model is safe against novel attacks
- multi-turn adversarial testing
- semantic judgment of ambiguous responses without human review

## Minimal commands

```bash
pip install -e ".[dev]"
hermes-jailbench --demo
hermes-jailbench --dry-run
pytest -q
ruff check hermes_jailbench tests
```

## Output shape

- terminal summary for demo and standard runs
- markdown or JSON report for saved output
- per-attack verdicts: `REFUSED`, `PARTIAL`, `COMPLIED`, or `ERROR`

## Success means

- attacks run without mutating the library state
- demo and dry-run work without an API key
- scorer output stays deterministic for the same response text
- tests stay offline and green

## Common failure cases

- users expect this tool to generate novel jailbreaks instead of replaying known ones
- API credentials or model IDs are invalid
- a response is ambiguous enough that the deterministic scorer needs manual review

## Maintainer notes

- keep the scorer deterministic and offline
- keep `run_bench()` and CLI behavior aligned with README examples
- do not add live-network behavior to tests