# Instructions for Claude — detection policies

These instructions apply to any work on the detection rule packs. Production
rules live in the external `agent-reliability-rules` repository
(`https://github.com/trustabl/agent-reliability-rules`); `testdata/rules-fixture/`
(this directory) is an in-engine **test mirror** of those packs, and **must be
kept in sync with the live repo** — see the "Two-repo rule model" section in
the top-level [`CLAUDE.md`](../../CLAUDE.md) for the full change workflow. The
`../` relative links below refer to the engine's `internal/rules/` package —
from this fixture location that is `../../internal/rules/`.

## Required reading order before editing

1. [`../schema.yaml`](../schema.yaml) — authoritative field reference.
2. [`README.md`](README.md) — author conventions in this directory.
3. The closest existing rule to what's being asked for — pattern example.

Do not skip step 1.

## Hard rules

- **Never invent YAML keys.** The schema is closed
  (`KnownFields(true)`). If a field you want does not exist in
  [`../schema.go`](../schema.go), extending the schema is a four-file
  change (schema.go + predicates.go + evaluator.go + schema.yaml). Make
  all four changes in one commit.
- **Never change a rule's `id` after it has shipped.** IDs are external
  identifiers; downstream consumers cite them.
- **Never duplicate a rule ID across files.** The loader rejects this at
  startup; per-rule tests catch it faster — run `go test ./...`.
- **Never widen `applies_to` across SDKs casually.** A rule's
  `explanation` / `fix` text is usually SDK-specific. Adding
  `openai_tool` to a Claude-SDK rule (or vice versa) makes the user-facing
  text lie. If a cross-SDK pattern is genuinely needed, author a separate
  rule under that SDK's category (`policies/<sdk>_sdk/<topic>.yaml`) with
  framing that matches the target SDK.
- **Never silence the per-rule test suite** in
  [`policies_test.go`](../policies_test.go). The
  `TestPolicyRules_AllRulesCovered` test fails when a shipped rule has no
  fire/silent case in `policyRuleCases`; that's the contract that keeps
  every shipped rule honest. Add the test cases, don't remove the check.
- **Never write rules at `info` severity.** Reserved.

## Required fields per rule

Every rule MUST set: `id`, `title`, `scope`, `severity`, `confidence`,
`applies_to`, `match`, `explanation`, `fix`. The loader refuses to start
the scanner if any are missing (`scope` empty is rejected with "scope is
required") — this surfaces as a `scan: ...` error in the CLI.

`language:` is OPTIONAL but state it explicitly — defaults to `python`
when omitted. For TypeScript / JavaScript / Go rules, set it explicitly:
`language: typescript`. Python and TypeScript discovery are both plumbed in
today (Claude SDK, OpenAI Agents, Google ADK, and MCP all have TS discovery
and shipped TS rules). JavaScript (`.js`/`.jsx`/`.mjs`/`.cjs`) now shares the
TypeScript grammar and discovery (the tsx parser parses plain JS) and is
audited by the `language: typescript` rule packs via the TS/JS family gate
(`models.IsTSOrJS`) — so a `typescript` rule fires on `.js` too (and a
`javascript` rule, were one shipped, fires on both). Go, C#, PHP, and Rust have
MCP tool discovery with field-based rules (MCP-015..022), so rules in those
languages fire against discovered MCP tools.

## Per-scope `applies_to` values

The `applies_to` list constrains which discovered entities a rule fires
against. Pick values from the table for the scope you're targeting.

**`scope: tool`** — receives a `ToolDef`; `applies_to` is matched against
`ToolDef.Kind`:

| `applies_to` value   | Matches                                       |
| -------------------- | --------------------------------------------- |
| `openai_tool`        | `@function_tool`-decorated Python function    |
| `claude_sdk_tool`    | `@tool` / `@claude_tool` / `claude_agent_sdk` |
| `mcp_tool`           | `@server.tool`, `@mcp.tool`, `.register_tool` |
| `shell_invocation`   | Bare function that calls `subprocess.*` etc. (no rules currently target this — OSH-* moved to a closed-source project) |
| `adk_function_tool`  | `FunctionTool(fn)` wrapping a Python function (Google ADK) |
| `langgraph_node`     | Plain function registered via `<builder>.add_node(...)` in a LangGraph graph (not a `@tool`) |

**`scope: agent`** — receives an `AgentDef`; `applies_to` is matched against
`AgentDef.Class` + `AgentDef.SDK`:

| `applies_to` value        | Matches                                                       |
| ------------------------- | ------------------------------------------------------------- |
| `openai_agent`            | `Agent(...)` from `openai-agents` SDK                         |
| `openai_sandbox_agent`    | `SandboxAgent(...)` from `openai-agents` SDK                  |
| `claude_agent_definition` | `AgentDefinition(...)` from `claude-agent-sdk`                |
| `claude_query_main`       | `query(...)` main-thread agent (`QueryMainAgent`) from the Claude TS SDK |
| `adk_llm_agent`           | `LlmAgent(...)` / `Agent(...)` alias from `google-adk`        |
| `adk_sequential_agent`    | `SequentialAgent(...)` from `google-adk`                      |
| `adk_parallel_agent`      | `ParallelAgent(...)` from `google-adk`                        |
| `adk_loop_agent`          | `LoopAgent(...)` from `google-adk`                            |
| `adk_langgraph_agent`     | `LanggraphAgent(...)` from `google-adk`                       |

**`scope: repo`** — receives `RepoProfile` + `RepoInventory`. `applies_to`
at this scope is matched against a fixed token list (the loader's
`validAppliesToForScope`); these tokens are *category-like* labels, not the
SDK enum values used by the `repo_has_sdk_in_code` predicate:

| `applies_to` value | Matches repos that use               |
| ------------------ | ------------------------------------ |
| `claude_sdk`       | Claude Agent SDK                     |
| `openai_agents`    | OpenAI Agents SDK                    |
| `openshell`        | NVIDIA OpenShell SDK (no rules currently target this — OSH-* moved to a closed-source project) |
| `mcp`              | Model Context Protocol               |
| `google_adk`       | Google ADK (Python)                  |
| `langchain`        | LangChain / LangGraph                |
| `crewai`           | CrewAI                               |
| `pydantic_ai`      | Pydantic AI                          |
| `vercel_ai`        | Vercel AI SDK (TypeScript)           |
| `autogen`          | AutoGen / AG2                        |
| `raw_llm_sdk`      | Bare `anthropic` / `openai` client usage (no agent framework); category `raw_llm_sdk` loads unconditionally, the rule's own predicate gates |
| `agent_config`     | Pre-trust harness config committed to the repo (`.claude/settings.json`, `.mcp.json`, `.gemini/.env`, `.vscode/settings.json`, `.codex/config.toml`); category `agent_config` loads unconditionally, the rule's own `repo_*` predicate gates. Findings carry the offending file and line |

Repo-scope rules typically combine `applies_to` with a `repo_has_sdk_in_code`
predicate to narrow firing to repos that actually use the SDK in code (e.g.
OAI-201's `match: { all: [ repo_has_sdk_in_code: [openai_agents], ... ] }`).
The two tokens look similar but live in different namespaces — `applies_to`
accepts `openai_agents`, and `repo_has_sdk_in_code` also accepts `openai_agents`
(the SDK enum, `models.SDKOpenAIAgents`); for Claude, `applies_to` uses
`claude_sdk` (the category) while `repo_has_sdk_in_code` uses `claude_agent_sdk`
(the SDK enum). Mismatching the two will silently fail the loader's scope
check.

Always set `applies_to` explicitly — the loader does not infer scope from
the category. Omitting it would make a rule fire against every entity of
that scope regardless of SDK, which is almost never correct.

**`scope: subagent`** — receives a `SubagentDef` (one `.claude/agents/*.md`
frontmatter declaration, matched at any path depth). `applies_to` is matched
against a fixed token:

| `applies_to` value | Matches                                                         |
| ------------------ | --------------------------------------------------------------- |
| `claude_subagent`  | A `.claude/agents/*.md` subagent declaration (any path depth)   |

Subagent rules use the `subagent_grants_tool` predicate (true when
`SubagentDef.Tools` contains a listed tool name). They carry **no `language:`
field** — subagents are markdown frontmatter, not code, and the
`subagentRuleDetector.Applies` method does not gate on language. The shipped
rules are CSDK-110 ("Subagent granted the built-in Bash tool") and CSDK-111
("Subagent granted filesystem-write or web-fetch built-ins"), both in
`claude_sdk/subagent_safety.yaml`.

## "Add a rule for X"

Default sequence:

1. Read the structurally-closest existing rule (e.g. for a new network
   rule, read `claude_sdk/network.yaml`).
2. Decide whether existing predicates cover the case. Existing first; new
   primitives are a real cost.
3. Pick the category subdirectory and topic file. If neither matches, ASK
   the user before creating new ones.
4. Write the rule. Match the explanation/fix tone of nearby rules — a
   paragraph, plain language, names the consequence and the fix concretely.
5. Add at minimum one fire case AND one silent case to `policyRuleCases`
   in [`../policies_test.go`](../policies_test.go). The
   `TestPolicyRules_AllRulesCovered` guard requires every shipped rule to
   appear in this table.
6. Run `go test ./...`.

The `testdata/corpus/` directory is a real-agent corpus, not a controlled
fixture. The scanner sweep over `testdata/corpus/*` only asserts no crash — it
does NOT assert that any specific rule fires there. Per-rule fire/silent
correctness is the job of `policies_test.go`.

## "Remove a rule"

1. Delete the rule entry from its YAML file. If the file is now empty,
   delete the file.
2. Remove all matching cases from `policyRuleCases` in
   [`../policies_test.go`](../policies_test.go).
3. Run `go test ./...`.

## "Change a rule's severity / confidence / explanation"

In-place edit. No fixture changes required. Run `go test ./...` — the
smoke test only asserts that rules fire, not what severity they fire at.

## When the per-rule test fails after you add a rule

[`../policies_test.go`](../policies_test.go) drives a fire/silent case
table for every shipped rule. Two failure modes and what they mean:

1. **`TestPolicyRules / "RULE-ID fires on ..."` failed** — your rule
   matched a snippet that should have triggered it but didn't. Walk the
   `match:` expression against the snippet by hand; usually a predicate
   set to `true`/`false` is inverted, or `applies_to` is missing the
   ToolKind your test uses.
2. **`TestPolicyRules_AllRulesCovered` failed** with your rule ID listed
   as missing — you added a rule but no entry in `policyRuleCases`. Fix
   by adding at least one fire case and one silent case for that rule ID.

The corpus sweep in
[`../../scanner/scanner_test.go`](../../scanner/scanner_test.go) only
checks the scanner doesn't crash; it does not assert findings. Don't
expect rule failures to surface there.

## Path to production-grade (known gaps)

The shipped pack is calibrated as "signal to investigate," not an
authoritative gate. The highest-leverage work to move it toward
production-grade, in priority order:

- **(a) Value-aware `timeout` check — DONE.** `call_without_kwarg` now treats a
  kwarg present with literal `None` as missing, so `requests.get(url,
  timeout=None)` fires. Same for `agent_kwarg_missing` (e.g.
  `before_tool_callback=None`).
- **(b) Local client-alias matching — DONE (same-function scope).**
  `call_without_kwarg` and `has_dynamic_url_call` resolve local-variable client
  aliases (`s = requests.Session(); s.get(...)`, and the `with ... as c:` form)
  via `analysis.ResolveClientAliases` + `IsHTTPCallNode`. **Residual
  limitation:** instance attributes (`self.client.get(...)`) and
  cross-function / cross-module aliases are still unresolved. aiohttp session
  aliases canonicalize to `aiohttp.ClientSession.<method>` (see
  `analysis.clientConstructorModule` and its guard test), which the timeout
  rules' callee lists carry — aliased aiohttp works end-to-end, same as
  `requests` and `httpx`.
- **(c) Hosted + decorated shell tools in the agent shell-tool rules — DONE.**
  `agent_uses_tool_kind: [shell_invocation]` (OAI-101, OAI-104) now matches
  three shapes, not just a bare undecorated function: (1) a bare
  `KindShellInvocation` function; (2) the SDK's hosted shell tools
  (`ShellTool`, `LocalShellTool`, `CodeInterpreterTool`, `ApplyPatchTool`,
  ADK `ExecuteBashTool`/`BashTool`) via the `hostedClassToKind` map over
  `HostedToolRefs`; and
  (3) a *decorated* tool (`@function_tool` / `tool()`) that shells out —
  discovery stamps a structural `shells_out` fact on the `ToolDef` (Python in
  `buildTool`, TS in `tsHandlerFacts`) and `PredAgentUsesToolKind` honors it
  for `shell_invocation`. Residual: the agent rules still only key on shell
  reach, not the broader "filesystem-touching" half their titles also name.
- **(d) Source-level fire/silent fixtures — started, not general.** Per-rule
  cases in `policies_test.go` feed hand-constructed typed inputs, so they
  prove predicate logic but not discovery → detection end-to-end. OAI-201
  (`internal/scanner/scanner_test.go`,
  `TestScan_OAI201_TracingDisabled`) now has a fire/silent pair that runs
  real `.py` source through the full scanner rather than a hand-built
  `RepoInventory` — it fires on a plain `Agent(...)` repo and goes silent on
  `Runner.run(agent, run_config=RunConfig(tracing_disabled=True))`. The rest
  of the pack still uses hand-constructed inputs; add more fixtures of this
  shape as rules whose detection lives inside computed-fact logic (like
  `disablesDefaultTracing`) are touched, so a discovery change that stops
  producing a shape a rule depends on fails a test instead of silently
  killing the rule.

## Output discipline for explanation/fix text

These strings are user-facing — they appear in the CLI's scan summary (and
JSON output) and guide whether a user acts on the finding. Vague text
undermines the product.

- **explanation**: name the consequence, not just the pattern. "Returns
  exceptions to the model as opaque strings, so it can't recover" beats
  "raises exceptions".
- **fix**: prescribe the concrete change. "Pass `timeout=` (5–30s)" beats
  "add error handling".
- Match the tone of the surrounding rules — paragraph, no headers, no
  bullets.

## What this directory is NOT

- Not a place for ad-hoc YAML files. Every `.yaml` file here is loaded as
  a policy. Don't drop notes, examples, or work-in-progress files inline.
- Not a doc directory. The README.md and this CLAUDE.md are exceptions;
  they are filtered out by the loader's `.yaml`-suffix check, but the
  convention is "data only, two markdown docs only".