recover-tier git:20260805.fd5c097A

Immutable. This exact content never changes and is served at /api/v1/blob/b2982e9c53006319.

---
name: recover-tier
description: Recover a LethAL tier that is quarantined or holding a stranded operation marker — recycle the environment, force-reset the lease, clear the quarantine. Use when a run refuses with "tier ... is quarantined", "container-needs-recycle", or "operation-orphaned". Publishes to and force-clears state on a live environment, so it is user-invoked only.
disable-model-invocation: true
---

# Recover a quarantined tier

A stranded mutant leaves TWO pieces of state behind, in different places, and clearing either one
alone leaves the tier refusing work:

| state | lives in | cleared by |
|---|---|---|
| durable quarantine record | `~/.lethal/quarantine/<hash>.json` (local) | `lethal clear-quarantine` |
| op marker on the `LC Lease` row | the environment's DATABASE | `lethal force-reset-lease` |

**An environment restart clears neither.** It kills sessions; the lease row is a table. That is
exactly why design §8 makes the restart and `ForceResetLease` separate steps, and it is the
mistake to expect: a restart looks like it should have been enough.

## When this applies

A run refuses with one of:

- `tier <server>|<instance> is quarantined (test-run: test in-flight-unknown running ...)`
- `LeaseUnavailableError: container-needs-recycle: AcquireLease reported operation-orphaned twice
  with an UNCHANGED marker`

Both usually mean a mutant did not terminate (R53's class — a negated loop-exit condition
reproduces this every time) and its attempt was abandoned mid-flight.

## The safety precondition, which is the whole reason this is user-invoked

`force-reset-lease` bumps the epoch, mints a new server generation and clears the op marker. That
is only SAFE once the stranded AL is actually dead. If the operation is still executing, a fresh
session can now take the lease while mutated code runs underneath it — a false verdict, which is
the failure class this whole layer exists to prevent.

So the restart is not optional and it is not a formality. Do it, confirm the environment reports
`Running` again, and only then reset. Do not reason from `serverGeneration` in the error message:
`operation-orphaned` means "op marker set, lease lapsed past grace" (`ControlState.Codeunit.al`)
and says nothing at all about whether a restart happened.

## Steps

### 1. Identify the tier

The refusal names it as `<server>|<instance>`, e.g.
`https://demoportaldev.continiaonline.com|a8f54c93-641a-4eb5-ac16-461e21a7bada`. For a container
the instance is the BC instance name; for an environment-tool environment it is the environment id.

### 2. Recycle

Container:

```bash
docker context use desktop-windows
Restart-BcContainer -containerName Cronus281       # PowerShell tool
```

Environment tool:

```bash
cd U:/Git
./CLI/continia.exe env stop <envId>
./CLI/continia.exe env start <envId>
# then WAIT for Running — starting is not started
until ./CLI/continia.exe env get <envId> | grep -qE "Status:\s+Running"; do sleep 20; done
```

### 3. Force-reset the lease

`force-reset-lease` resolves an env-tool config the same read-only way `lethal run`/`lethal doctor`
do — pass your real config straight through, no materialised copy needed:

```bash
bun packages/runner/src/cli.ts force-reset-lease --server <server> --instance <instance> --config <base-config.json>
```

Credentials never touch disk or a log: resolution happens in-process against the config's own
`envTool.resolve` blocks and is never printed or written out (`resolveForceResetLeaseConfig`,
`packages/runner/src/cli.ts`). For a directly-configured container (no `envTool` section), this is
unchanged from before — `server`/`serverInstance`/`username`/`password` are already in the file.

### 4. Clear the quarantine

```bash
bun packages/runner/src/cli.ts clear-quarantine --server <server> --instance <instance>
```

Expect `cleared`. `not-quarantined` means the tier key did not match — check the server/instance
spelling against the refusal message verbatim.

### 5. Re-run, and do not let the same mutant strand again

A non-terminating mutant reproduces every time. `--resume` skips it by design:

```
1 stranded the tier on a prior run and will be SKIPPED rather than re-executed
```

Raising `--mutant-timeout-ms` does NOT help a mutant that never terminates — measured at both 30 s
and 120 s on the same mutant.

**`--resume` is fine for finishing a run. It is NOT a valid input to a differential gate:** it
carries prior verdicts while recomputing attribution, so each row mixes two runs. See the
`coverage-differential` skill.

## Report honestly

Say which steps ran, that the restart preceded the reset, and the epoch/generation transition
`force-reset-lease` printed. A recovery whose precondition was skipped is worth knowing about
later, when a verdict looks strange.