awa-operations git:20261026.c9387adA

Immutable. This exact content never changes and is served at /api/v1/blob/1e6a63f07132e0de.

---
name: awa-operations
description: Deploy, upgrade, and operate the Awa Postgres-native job queue and its worker fleet. Use when running database migrations, planning a rolling upgrade, sizing connection pools on managed Postgres, operating the dead-letter queue or cron schedules, driving a storage-engine transition, configuring the awa CLI, or serving the web admin UI.
license: MIT
compatibility: Requires an awa CLI matching the deployed runtime version and PostgreSQL access. Written for awa 0.7; pin to the release tag matching your deployment. The CLI, `awa health`, and `awa storage status` work against the database alone; the runtime `/readyz` probe and the fleet-liveness fields require a live worker fleet.
---

# Awa Operations

Use the awa CLI version that matches the deployed runtime and schema. A binary
refuses a database schema newer than it understands, so operate every database
from a CLI at least as new as the fleet. Read the upgrade note for the exact
version pair before changing anything (`docs/upgrade-0.6-to-0.7.md`,
`docs/upgrade-0.5-to-0.6.md`).

## Connect Safely

`awa` resolves a target in this order: `--database-url` > `--context`
(`AWA_CONTEXT`) > `DATABASE_URL` env > `default_context` in
`~/.config/awa/config.toml`. Every command echoes the resolved target with the
password stripped before acting; confirm it.

Read-only commands may fall back to `DATABASE_URL` or the default context.
Mutating commands refuse to run against an implicit target whenever more than
one context is defined — pass an explicit `--context` or `--database-url`. A
context marked `production = true` prompts for confirmation on a mutating
command; `--yes` skips the prompt, and a non-interactive shell without `--yes`
refuses rather than hanging. An explicit `--database-url` is never treated as
production.

Prefer `url_env` over an inline `url` in config so credentials never sit in the
file. Never print database URLs or passwords in logs or issues.

## Migrations

Migrations are forward-only; there are no down migrations. Roll back with a
database snapshot or reverse SQL, not with the CLI.

Use an updated binary to generate pending upgrade SQL. The runner records
versions, not checksums; applied steps are not rerun. Preserve already-applied
files in checksum-based external runners rather than replacing their history.

Migration discovery skips unregistered custom schemas with foreign-owned objects,
untrusted `CREATE`/`TRIGGER` grants, or any trigger awa did not install. Unsafe
registered/prepared substrates block the bundle before version advancement.
The error names up to five offending objects; the upgrade guide gives fixes for
`GRANT ALL` table grants, a 0.6 runtime's schema `CREATE` grant, and schemas
prepared by another role. Sequences owned by another role are accepted. The same condition in `awa` aborts before
installation, including for superuser migrators. Use a consistent dedicated owner
role and review ownership and grants before retrying; see the upgrade guide.

One `awa migrate` applies the whole pending range in a single transaction under
an advisory lock, so a failed or interrupted run commits nothing and concurrent
runners converge on one application. Every migration is re-runnable, so a retry
after any failure is safe.

```bash
awa migrate --pending                 # apply from current DB version to latest
awa migrate --sql --pending           # print the SQL instead of applying
awa migrate --extract-to ./out        # write SQL files for an external runner
```

`--sql` output is wrapped in one transaction taking the runner's advisory lock,
so `awa migrate --sql | psql -v ON_ERROR_STOP=1 "$DATABASE_URL"` is atomic.
`ON_ERROR_STOP=1` is what makes a failure *detectable*: plain psql rolls the
transaction back but still exits 0. Pass `--no-transaction`
when the consuming runner opens its own transaction per migration (Flyway,
Liquibase, dbmate); `--extract-to` files are never wrapped, for that reason.
`--from` / `--to` / `--version` select a range to render and are rejected on the
apply path — applying always targets the current version.

The migrator runs pre-flight gates before touching the schema: it refuses a
schema newer than the binary understands, refuses to apply while the storage
transition is unfinalized where a gate requires it, enforces per-migration
minimum runtime versions, and enforces exclusive (no-live-runtime) windows for
non-rolling migrations. `--allow-live-runtimes` overrides the live-runtime
checks — use it only after independently verifying compatibility.

Extracted or printed SQL does **not** run the Rust pre-flights. An external
runner must enforce version floors and exclusive windows itself. Do not append
`storage prepare`/`enter-mixed-transition`/`finalize` to extracted migration
files; those are not migration DDL.

For a pending range crossing v045, acquire `LOCK TABLE awa.cron_jobs IN ACCESS
EXCLUSIVE MODE` before the first pending DDL in that transaction if the table
already exists. The Rust migrator does this automatically. Cron evaluation
pauses until commit; the lock prevents a released enqueue from deadlocking with
earlier storage DDL. See the 0.6→0.7 upgrade guide for the rollout boundary.

Do not skip a major version. 0.5 → 0.7 is unsupported; step through 0.6.

## Restricted Database Roles

When deploying a runtime without schema CREATE, apply v049 and provision every
queue with `awa storage prepare-queue` as the migrator before starting producers
or workers. Match shard/stripe counts; provision additional lanes before changing
routing. Use the sequence grants and owner defaults in
[Database roles](../../docs/security/database-roles.md#provision-queues-before-starting-restricted-runtimes).

## Rolling Upgrades

Awa runs mixed-version fleets during a rollout; the constraints are version
specific, so follow the matching upgrade doc. The 0.6 → 0.7 upgrade is the
current worked example and its shape generalizes:

1. **Stepping stone first.** Roll the entire fleet to 0.6.2+ before applying the
   0.7 (v043) ring-rotation migration. 0.6.0/0.6.1 misclassify a newer schema
   destructively (#392).
2. **Finalize storage.** A 0.7 binary refuses `awa migrate` unless the storage
   transition is finalized (`storage status` shows `state=active`,
   `current_engine=queue_storage`) or the install is fresh.
3. **Roll binaries and migrate in either order.** Mixed 0.6.2/0.7 is safe in
   compatibility mode. If binaries roll first, a 0.7 worker refuses to start on
   the old schema and crash-loops harmlessly until the migration commits.
4. **Promote ring authority only after the fleet is fully 0.7.** Use
   `awa storage flip-ring-authority`, or let maintenance auto-flip once every
   worker has been 0.7 for a stable period.

Crash-safety facts to respect:

- After the flip, roll back only to another 0.7 build. The flip poisons the
  compatibility cursor fields and a database trigger rejects old-style cursor
  advances, so a returning pre-flip binary fails loudly.
- While a v043 fleet is still all-0.6.2, deadline-based rescue resumes only once
  a 0.7 runtime takes maintenance leadership. Roll the maintenance leader
  promptly.

The `flip-ring-authority` step itself has a hard guard — see Storage
Transitions below.

## Health And The Fleet

Workers are stateless; all state is in Postgres, and scaling means running more
processes. Exactly one worker per database is the elected maintenance leader
(session-scoped advisory lock — if it dies, another is elected on the next
tick). The leader alone runs cluster-wide scans: deferred→ready promotion, the
three rescue paths (stale-heartbeat, hard-deadline, callback-timeout), ring
rotation and prune, DLQ and descriptor cleanup, and cron evaluation. During an
unfinalized transition it promotes both planes' deferred backlogs, so drain and
post-flip execution do not depend on which runtime holds leadership. Every
worker dispatches and heartbeats its own attempts. There is no `pg_cron` or
external scheduler.

Late completions from a rescued attempt lose against the newer attempt via the
`run_lease` guard, so rescue never double-commits. Set `heartbeat_staleness`
(default 90s) to at least 3× the heartbeat interval; detection latency is
roughly `heartbeat_staleness + heartbeat_rescue_interval`.

Finished handlers retry durable finalization for up to 120s from the first
finalize attempt when database writes are unavailable, including connection,
pool-acquire and read-only-session errors. Recovery requires a writable endpoint
and usable pooled connections; a DNS change alone does not redirect existing
read-only sessions. Monitor
`awa.completion.reconnect_retry` for outages or pool pressure. Pending durable
finalizers are bounded by the larger of worker capacity and twice the configured
completion batch size times completion shards (one flushing and one queued batch
per shard). A full budget backpressures returning handlers, retaining at most one
additional saved result per worker. Shutdown drains finalizers with handlers until the drain
deadline. Rescue can still win first, and expired/cancelled finalization falls
back to rescue, so keep handlers idempotent. A stale reply after an availability
error may mean the first finalize committed or another actor closed the lease;
completion metrics and lifecycle hooks are best effort in that case.

Health signals:

- Runtime listener (opt in via `AWA_HEALTH_ADDR`): `GET /healthz` liveness (200
  even while draining) and `GET /readyz` (503 with failing checks:
  postgres_connected, schema_version, schema_compatible, poll/heartbeat/
  maintenance loops, shutting_down). `leader` is informational and never gates.
  The listener has no TLS or auth — keep it off public ingress.
- `awa health [--json]`: database-only readiness (reachable, schema migrated for
  this binary, fleet heartbeat counts). Exit 0 ready, 1 unreachable or
  unmigrated.

On Kubernetes, set `terminationGracePeriodSeconds` above the shutdown drain
timeout (e.g. 40s for a 30s drain) so a leader finishes draining before SIGKILL.

## Dead-Letter Queue

The DLQ is a separate physical table, not a job state; a DLQ'd job cannot be
claimed until retried out. It is per-queue opt-in (`dlq_enabled`) with a
default 30-day retention pruned by the maintenance leader. `dlq_reason` is
`max_attempts_exhausted`, `terminal_error`, `heartbeat_stale`,
`deadline_exceeded`, `callback_timeout`, or the reason passed to `awa dlq move`.

A poison job (one that crashes, hangs or panics on every attempt) stops after
`max_attempts`: crash and deadline rescues spend attempts like handler errors.
To contain one, `awa job cancel <id>` it or `awa queue pause` its queue, ship
the fix, then `awa dlq retry-bulk --queue <q> --kind <kind>`.

```bash
awa dlq depth --queue emails
awa dlq list --queue emails --limit 20
awa dlq retry <id>                       # resets attempt/lease, re-runs
awa dlq retry-bulk --queue emails        # bulk; --all required with no filter
awa dlq purge --kind SendEmail           # destructive; --all required if unfiltered
```

- Retry resets the job to `attempt=0`, `run_lease=0` on the runnable path with
  its stored queue/priority and immediate `run_at`. Use the programmatic retry
  APIs to override those.
- `retry-bulk`, `purge`, and `move` require `--all` when no filter is given —
  the guard against reviving or wiping the whole DLQ. Purge is unrecoverable;
  prefer `retry-bulk` if you might want the data back.
- Enabling `dlq_enabled` does not retroactively move existing `failed` rows.
  Move them explicitly with `awa dlq move --reason ... [--all]`.

## Cron

The default CLI cron surface includes `list` and `remove <name>`. Pause, resume, and
trigger are admin operations (the web UI and admin API, and the Python client),
not the CLI. Only the maintenance leader evaluates due schedules; enqueue is
atomic and re-checks `paused_at` inside the same statement, so a pause races
cleanly against a fire — a pause takes effect even if it lands between the
evaluator's read and the enqueue. Schedules are declared in application code;
for timezone and missed-fire (`coalesce`/`catch_up`) semantics, see the
`awa-jobs` skill.

### Owned schedules (v045+)

Use current operator clients. `awa cron plan OWNER` explains fleet capability,
manifest disagreement, zero-live refusal, conflicts, grace, and retirement
candidates. `--manifest PATH` previews a JSON array of proposed definitions
without publishing it. `cron adopt NAME OWNER` (optionally
`--expected-owner OLD` for transfer), `cron retire NAME`,
`cron retire-owner OWNER`, `cron restore NAME`, and `cron restore-owner OWNER`
default to dry-run; `--apply`
commits and `--actor LABEL` identifies the operator. The web UI provides previews
and Apply too. Restore-owner restores only retired rows, preserving pause and
starting their evaluation from now. Owned/retired rows cannot be physically removed.
Reintroducing a retired desired name does not fail startup, but it remains inert
until restoration. Plan/preview reads do not hold the evidence writer lock;
apply rechecks the decision. Actions reset only affected owners, including both
sides of an ownership transfer.

A retired schedule differs from pause: no current automatic/manual firing,
no resurrection by registration, and explicit restore starts from now without
replaying retired time. Legacy automatic enqueue and UPSERT honor the database
fence; legacy manual-trigger clients are outside this operator contract. Zero
live declarations never auto-retire an owner. Use retire-owner explicitly to
decommission a fleet. Keep the v045 snapshot trigger installed; it serializes
legacy evidence publication with retirement without locking job heartbeats.

## Storage Transitions

`awa storage status` is the source of truth: it reports the transition state
(`canonical` → `prepared` → `mixed_transition` → `active`), current/active/
prepared engine, backlog, blockers, and `can_finalize`. Drive transitions with
the storage commands so routing, drain, readiness, and interlocks move
together — never by swapping which workers you run.

```bash
awa storage status
awa storage prepare --engine queue_storage
awa storage enter-mixed-transition        # needs a live queue_storage_target runtime
awa storage finalize --check              # dry run: exit 0 ready, exit 2 blocked
awa storage finalize --wait               # poll until gates stay clear
```

On builds carrying #457, `storage enter-mixed-transition --quiesced` permits a
stopped fleet to flip without a target runtime. Stop all workers and keep them
stopped until it returns; any fresh heartbeat refuses the command. A stale
heartbeat is not a fence against a paused worker returning. Canonical backlog
is preserved and still blocks finalize: drain it before stopping for a role-free
cutover, or retain explicit canonical drain workers afterward. Follow the
[quiesced upgrade procedure](../../docs/upgrade-0.5-to-0.6.md#quiesced-alternative-to-the-witness-runtime).
On builds carrying #462, callback resolution falls back to canonical jobs only
during mixed transition, preserving lease fencing and transaction atomicity.
Both fixes need a 0.6 backport for unfinalized clusters; installing 0.7 first is
not a supported workaround.

Dangerous operations:

- `prepare-queue-storage-schema --reset --schema awa` is refused — it would drop
  the migration-owned `awa` schema (schema_version, runtime_instances, …) and
  leave the database unrecoverable. `DROP SCHEMA awa CASCADE` is not a supported
  operator action.
- The transition is a one-way door once queue-storage work is accepted:
  `storage abort` is rejected once rows exist in the new tables. Recovery is
  `finalize` or database restore.
- `flip-ring-authority` is one-way; it refuses without `--force` while any
  fresh-heartbeat runtime is not known to be flip-aware (an unknown or
  unparseable runtime version counts as not flip-aware).
- `rebuild-terminal-counters` is for counter-drift incidents and is best run on
  a quiesced fleet.

While the transition is unfinalized, a canonical attempt that re-schedules itself (snooze, retry backoff, `RetryAfter`) leaves the canonical plane rather than returning to canonical `scheduled_jobs`, so the drain converges even for handlers that snooze on every run instead of completing (#456). The ordinary outcome is a fresh-id row in the prepared queue-storage schema's `deferred_jobs`. If a newer duplicate owns a unique claim required by the destination state, the old attempt instead becomes cancelled terminal evidence with `rescheduled as duplicate`; it is not executable and the newer holder wins. If `finalize --wait` sits at a flat non-zero backlog while jobs are executing, you are on a build without that fix.

Rings rotate only when active — a frozen generation on an idle queue is healthy.
The condition to alert on is a frozen generation *with* accumulating rows
(`skipped_busy`): a pinned ring, almost always caused by a long-held snapshot.

## Ring Rotation Cadence

For queue storage, `ClientBuilder::ring_rotation_cadence` seals a slot after a
five-second observed fill window once it holds 256 rows, at a 2 MiB total
physical-size target, or after 60 seconds regardless of rows, by default. Python
exposes `queue_storage_rotate_min_slot_age_ms` and
`queue_storage_rotate_target_slot_bytes`, plus `queue_storage_rotate_min_slot_rows`
and `queue_storage_rotate_max_slot_age_ms` (row floor before the age window seals,
and the ceiling that seals regardless). A zero byte target disables the size
trigger, a zero row floor disables the floor, and a zero maximum age disables the
ceiling. A row floor needs a size target or a ceiling, and a queue-storage client
refuses to build without one: lease rows are deleted as attempts finish, so a lease slot may
never reach the floor. A zero minimum age, or Rust `RingRotationCadence::EAGER`,
permits sealing each tick whatever the other settings.
Rust cadence and claim-interval overrides survive `queue_storage` calls in either order.
Empty slots bypass the fill window so reclamation can continue after drain.
Leader changes reset the local monotonic fill timers; the 60-second ceiling counts from when the slot opened on the database clock, so it still applies. The size is sampled and
can be exceeded by bursts or blocked reuse; use actual retained storage and
blocked-prune metrics for capacity planning. It is not a retention policy or
storage limit. Queue prune lock contention retries on the next decision tick;
active-data gates retain exponential backoff. Persistent contention can still
stall reclamation. See `docs/configuration.md` for the scheduling contract.

## Managed Postgres

Prefer PostgreSQL 18 where available; PG17 on some managed platforms caps
throughput because readers queue behind the ring-rotation `ACCESS EXCLUSIVE`
lock. Size for steady-state completion rate, not burst.

- **MVCC discipline is the biggest footgun.** Any long-held snapshot — `idle in
  transaction`, a BI tool, `pg_dump`, a stuck migration — pins the database-wide
  horizon and blocks vacuum *and* Awa ring prune across every table. Keep
  analytical readers on a replica; bound roles with
  `idle_in_transaction_session_timeout` and `statement_timeout`; alert on
  `pg_stat_activity.xact_start` age.
- **Pool sizing.** `max_connections >= sum(queue claimers) + 5`, plus headroom
  for handler SQL, producers, and admin. Each claimer holds a LISTEN connection;
  the leader holds one advisory-lock connection.
- **DDL creds.** Migrations and custom-schema prep need DDL-capable credentials.
  Cloud SQL IAM service accounts need a one-shot manual `cloudsqlsuperuser`
  grant; AlloyDB grants `alloydbsuperuser` automatically.
- **Role model.** `awa_owner` owns the schema (NOLOGIN); `awa_migrator` runs
  `awa migrate`; `awa_runtime` (least privilege, cannot alter schema) runs
  workers, producers, `awa serve`, and CLI admin.
- **v047+ migration bundles.** Use `awa migrate` or its generated `--sql`
  bundle rather than an isolated migration tail. Installer replay replaces
  triggers on ring state and queue heads and takes table locks through commit;
  schedule a quiet window, bound `lock_timeout`, and retry the whole transaction
  after a timeout. DDL credentials need ownership of all installed substrates.
- **CDC / logical replication.** Keep the awa schema (and any custom
  queue-storage schema) out of publications: list application tables, or with
  Debezium use `publication.autocreate.mode=filtered` plus
  `schema.exclude.list=awa,custom_queue_schema`, adding every installed custom
  queue-storage schema (replace the example name). Debezium matches each schema
  regex against the complete name; escape regex metacharacters in literal names.
  Alternatively, use `schema.include.list` for application schemas instead of an
  exclusion list; do not set both. See the [Debezium PostgreSQL filter reference](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-schema-exclude-list).
  If Awa is published anyway, schema v047+ works (every table has a replica
  identity); older schemas fail maintenance with
  `cannot delete from table ... does not have a replica identity`. Ring prune
  is a `TRUNCATE`, which publications publish by default. Slot lag can retain
  required WAL; lag alone does not establish an ordinary-table MVCC horizon pin.
  Inspect [`pg_replication_slots`](https://www.postgresql.org/docs/17/view-pg-replication-slots.html):
  `xmin` prevents vacuum from removing affected ordinary-table tuples, while
  `catalog_xmin` protects catalog tuples. Diagnose those horizons separately
  from WAL retention (`restart_lsn`) and consumer progress (`confirmed_flush_lsn`).

## Web Admin UI

`awa serve` is a read/write admin console, not the worker runtime. It has **no
built-in authentication** — treat it like a database admin console and restrict
access with ingress policy, firewall, or private networking. It binds
`127.0.0.1:3000` by default; set `--host 0.0.0.0` in containers.

Run one `awa serve` per database (single-backend by design; `--peer` links are
navigation only, no shared data plane). Use `--read-only` / `AWA_READ_ONLY=1`
for incident read-outs or against a replica — mutation endpoints return 503 and
the UI hides its buttons; read-only is auto-detected against a read-only
connection. Set `--callback-hmac-secret` to verify callback signatures; for
externally reachable callbacks with a private admin surface, run
`awa callbacks serve` instead, whose router has no admin routes.

## Before You Finish A Change

1. Confirm the resolved target and that you are on the intended context.
2. For a schema change, run `awa migrate --sql --pending` and read the plan.
3. For a storage or upgrade step, run `awa storage status` (and `finalize
   --check`) before and after.
4. Verify `awa health` and `/readyz` across the fleet, not just one worker.
5. Read the matching-version upgrade doc for gates, rollback limits, and
   ordering specific to the pair you are moving between.