git:20260828.d5d9d56 to git:20260829.1a1c68e

# CLAUDE.md - apps/worker

The concurrency kernel plus the Agent Sandbox substrate, and the F3 eval-stream

runner (`curie_worker.eval`), which runs eval suites off the `curie:evals`

stream against the frozen eval-case format (#8, ADR-0019). Full behavior spec

lives in `apps/worker/README.md`; this file is the enforceable-rule summary. Read

`../../ARCHITECTURE.md`'s message-flow diagram before changing `kernel.py`.

## The kernel is sacred: single owner, adversarial review

`kernel.py`/`consumer.py`/`threadlock.py`/`markers.py` are correctness logic

with races that only show up under concurrent load. **One change owns this

module at a time, it is never split across parallel work, and any change needs

an adversarial review (spec-vs-impl + side-effects-detective, minimum) before

merge.** If you are touching `kernel.py` as a side effect of another change,

stop -- that is scope creep on the sacred module.

## The four rules the kernel enforces (each has a provoking integration test)

1. **One live session per thread.** A follow-up to a thread with a live turn

is a *steer*, never a new turn. The per-thread lock (Valkey `SET NX PX`

plus an in-process FIFO lock) wraps the route decision; the new turn

opens *before* the lock releases. The lock is never held during

streaming -- if you find yourself holding it longer, you broke "steering

is never blocked."

2. **The finish race.** A steer arriving as the turn ends returns 409; the

kernel opens a fresh turn on the same idle sandbox. Do not "fix" a 409 by

retrying the steer instead of falling back to a new turn.

3. **Steer vs. interrupt.** Default is steer; `Kernel.interrupt_thread` is

the explicit hard stop. **The kernel never keyword-guesses intent** -- do

not add text-sniffing to decide between the two; that is an explicit

caller signal.

4. **No auto-retry after side effects.** A failed run that emitted

`side_effect_flag` escalates to a human instead of retrying; the flag is

persisted to Valkey the instant it is seen so a crash mid-side-effect

still escalates on reclaim. Flag-clean failures retry by classification

## Delivery is bounded (ADR-0039, #505)

The reclaim loop is **not** infinite. An entry already delivered `max_delivery`

times (`CURIE_MAX_DELIVERY`, default 5, floor 2) is dead-lettered to

`<stream>:dead` and acked off the group instead of re-dispatched. Unbounded

reclaim let one permanently-failing entry starve the whole consumer group -- a

silent total stall. Removing or bypassing the cap is that regression, not a

simplification.

- **The delivery count is read from the PEL every pass, never tracked in

process memory.** A process-local counter resets on restart, so a

crash-looping worker would retry poison forever -- the exact stall shape.

- **Read the count with `XPENDING` *before* `XAUTOCLAIM`**, which bumps it on

claim; the pre-claim value is the budget already spent. Cap at `>=`. Keep the

`IDLE` filter equal to `XAUTOCLAIM`'s `min_idle_time`, and keep the

`_inflight_ids` skip ahead of the cap check.

- **`XADD` before `XACK`, never the reverse.** A crash between them costs a

duplicate graveyard row; the reverse costs the entry.

- **`max_delivery` is not `max_attempts`.** The latter is the kernel's

flag-clean retry *classification* inside one delivery. Do not conflate them.

- **Transient failure is unchanged**: a mid-turn crash still reclaims, retries,

and acks. Only an exhausted budget dead-letters.

- **Exception (#708): approval-resume reply is best-effort in the pure-offline

loop.** When an approval-resume turn's reply endpoint is unreachable AND no

default transport is configured, the adapter swallows the unreachable error

instead of raising -- the turn ACKs without delivering the reply instead of

dead-lettering. Both adapters honor it: `SlackReplyAdapter` when no default

Slack transport is configured, `HttpReplyAdapter` by logging and returning

with no fallback target at all (a fallback there would post an email reply

into a Slack workspace). A configured default transport, or any non-resume

turn, still raises and dead-letters normally. A MISSING egress credential is

never swallowed: fail-closed outranks best-effort.

- **The graveyard has no consumer group, but it IS bounded** by an approximate

`MAXLEN` (`CURIE_DEAD_LETTER_MAXLEN`, default 10000) on every `XADD`. The

unparseable path dead-letters per inbound entry, so an unbounded graveyard hands

a wire-DTO drift a full-ingest-rate OOM against the same Valkey that holds the

kernel's locks and markers. Rows are therefore **best-effort**: under a flood the

oldest are evicted. Do not treat the graveyard as a complete audit log, and do

not add a consumer group, a retention policy, or replay tooling as a passenger

on another change.

- **A dead-letter stream equal to the source stream is rejected at config

validation.** `XADD` precedes `XACK`, so a self-targeting graveyard re-queues

failures onto the stream they came from and hot-loops. Do not soften that

validator into a warning.

## The sandbox substrate (`curie_worker.sandbox`)

- **The kernel talks in `thread_key` and `SandboxHandle` only.** Everything

Kubernetes-shaped stays behind the `SandboxSubstrate`/`SandboxClient` seam

-- never leak a K8s type across it into the kernel.

- **`claim()` is claim-or-adopt.** A lost creation race deletes the loser's

claim; the route lives in Valkey (`curie:sandbox:route:<thread_key>`)

with a TTL that `claim()`/`touch()` refresh.

- **Claims carrying per-claim env (the resume path) cold-create** instead of

binding the warm pool, because env cannot be injected into an

already-running pod. Expected latency, not a bug.

- **Suspend deletes the pod.** Never assume a suspended sandbox's process or

prompt cache survives; a resumed or restarted sandbox rehydrates from external

state via `CURIE_HISTORY_REF`.

- **The client seam is sync; unit tests fake only `SandboxClient`** (the K8s

control plane) -- Valkey is never mocked.

- **The history ref is a deterministic state-store URL, not an SDK session id**

(ADR-0029). `binding.boot_env` sets `CURIE_HISTORY_REF` to this thread's

transcript key on the durable state store (`.../state/transcript/<thread_key>`)

on **every** claim, so a fresh, restarted, or resumed sandbox all boot with the

same ref and the runner rehydrates the conversation as a preamble. There is no

SDK session id to capture off the ACI `final` frame and no frozen-contract

change; do not reintroduce a frame-captured resume id.

## Deployment-to-runtime binding (implemented)

`binding.py` resolves a thread's Slack channel to its agent, that agent's active

deployment (prod outranks dev, then most recent), and the resolved

`CURIE_BUNDLE_REF`, and injects it into the sandbox claim so the boot picks up

the bundle version the API's git-flow engine produced. It is a read-only query

layer over the shared Postgres tables (agents -> deployments -> agent_versions),

deliberately not an import of the API package. The seam to remember: a thread

keeps the sandbox and bundle it first booted with; only a fresh mention (a new

claim) picks up a newer deployment. Any change that touches the kernel's claim

path here is still bound by the sacred-module rule above.

## Verify

```bash

uv run pytest apps/worker/tests/kernel -q # real Valkey, fake K8s client, in-process fake runner, recording Slack sink

uv run pytest apps/worker/tests/sandbox -q # unit; CURIE_SANDBOX_E2E=1 additionally drives a real cluster

```

Only Slack and the model behind the runner are faked in the kernel suite.