deploy-runbook git:20261104.03a9677A

Immutable. This exact content never changes and is served at /api/v1/blob/dbd099080ce51798.

---
name: deploy-runbook
description: The exact three.ws production deploy runbook (clean worktree, build:gcp chain, submit gates, cron sync, CDN purge, verify). Load before any deploy.
when_to_use: Any production deploy, a deploy that died mid-build, or a question about build:gcp order and deploy gates.
---

# Deploy runbook (API/frontend)

Remember stop-and-ask gate 2: production deploys need owner approval unless the owner's current instruction is itself the approval. Full runbook incl. LB/DNS/TLS/env/rollback/recovery: `docs/ops/gcp-production.md`. GPU workers deploy from their own `workers/<name>/cloudbuild.yaml`.

0. **Reclaim disk first: `npm run clean:worktrees` (add `--apply` to act).** Deploy worktrees are created by every deploy and removed by nobody. On 2026-08-04 ten had accumulated and filled the 126 GB disk to 100%, which fails in a way that reads like a different bug entirely: `git worktree add` dies mid-checkout with `No space left on device` and leaves a half-written tree behind. The script only reclaims a linked worktree that is detached, clean, and idle past `--min-age-hours` (default 2, so a concurrent agent's live build is never deleted); anything with uncommitted work is reported and kept. It applies the same rules to the subagent worktrees under `.claude/worktrees/`, which additionally must have their `worktree-agent-*` branch already merged into HEAD; the merged branch is deleted with `git branch -d` (on 2026-10-08 nine of those held 31 GB).
1. **Prepare a CLEAN worktree: `npm run prep:worktree -- --apply`.** That is the whole step; it encodes everything below, and running it beats staging by hand. Plan first with a bare `npm run prep:worktree` (writes nothing), `--path <dir>` to stage your own tree instead of colliding with a concurrent agent on the default `/workspaces/.deploy-wt`, `--force` to replace an existing tree (refused if it holds uncommitted work). What it does: `git worktree add --detach <path> HEAD`, then hardlinks every artifact in the `ARTIFACTS` list of `scripts/prepare-deploy-worktree.mjs` with `cp -al` on the same filesystem (not just the first). Three of them are build-critical, `node_modules`, `chat/node_modules` and `character-studio/build`; the optional ones (`walk-sdk/dist`, `multiplayer/node_modules`, `animation-sources`) are what let the worktree also run the repo's own test suite, so a pre-ship `npm test` can be taken at the exact commit being shipped instead of in the shared tree. It then copies BOTH `.env` and `.env.local`. Omitting either nested one kills the build minutes in: an OOM exit 144 on the avatar-studio rebuild, or `Cannot find package '@sveltejs/vite-plugin-svelte'` on the chat build. A nested artifact that is missing from the source tree is BUILT first (`deps:chat`, `build:avatar-studio`) rather than skipped, which is the failure a hand-run `cp -al` produces on a machine that has never deployed. `.env.local` is the one that carries `DATABASE_URL`, and without it step 3's migration gate cannot read the schema state and refuses to submit. **Remove it when the deploy lands** (`git worktree remove --force <path>`); that is what step 0 exists to clean up after. See `docs/ops/gcp-production.md`.
2. **Build: `npm run build:gcp`.** ORDER IS LOAD-BEARING and `build:gcp` already encodes it. Never hand-run the steps in a different order. The full chain, in order:

   `check:conflicts` → `check:browser-graph` → `check:tdz-bootstrap` → `ensure:avatar-studio` → `build:info:snapshot` → `docs:freshness` → `build:lib:full` (the UMD lib) → `build:avatar-sdk` → `build:chat` → `build` (the frontend `vite build`) → `publish:lib` → `build:info` → `check:dist` → `check:pages`

   Why the order is load-bearing: `check:conflicts` runs first so an unresolved merge-conflict marker fails in seconds instead of after a full build. `check:tdz-bootstrap` sits beside it for the same reason: it refuses a browser module whose top-level bootstrap call runs above the `let`s that call writes. JavaScriptCore checks an assignment target's temporal dead zone before evaluating the right-hand side and V8 does not, so that ordering ships green on Chrome and throws "Cannot access uninitialized variable." on every Safari, which is how `/avatars/:id` went dead on iOS and macOS while every desktop check passed. `build:lib:full` writes `dist-lib/agent-3d.js`, `build:avatar-sdk` copies that bundle into `avatar-sdk/dist/index.mjs`, and the frontend `vite build` resolves `avatar-sdk/src/agent.js` against it, so both must come BEFORE `build`; skipping them shipped whatever stale `avatar-sdk/dist` the build machine happened to hold, and in a fresh worktree failed the build outright with `Could not resolve "../dist/index.mjs"` (2026-08-17). The frontend `vite build` runs with `emptyOutDir`, so it WIPES `dist/` and everything that writes into `dist/` must come after it: that is why `publish:lib`, which mirrors the UMD lib into `dist/`, follows `build` rather than sitting next to `build:lib:full`, and `check:dist` fails without that bundle present. `publish:lib` writes only the moving channels (`latest`, `1`, `1.5`) from the fresh build; every released version directory is downloaded from the write-once archive `gs://three-ws-lib-releases` and checked against `data/agent-3d-releases.json`, so the build needs network access to storage.googleapis.com, and it fails if `package.json` names a version that was never cut. A versioned URL is immutable and SRI-pinned by embedders, so it must never change bytes: cut a new version with `npm run release:lib` after bumping `package.json`, never by rebuilding an old one (2026-10-01: rebuilding 1.5.2 on every deploy broke every published SRI pin). `check:dist` also fails when a documented SRI pin disagrees with the ledger; `npm run check:sri-pins -- --fix` rewrites stale pins. `ensure:avatar-studio` consumes `character-studio/build` and `build:chat` consumes `chat/node_modules`, which is exactly why step 1 stages worktree artifacts. `docs:freshness` measures documentation drift against git history and writes `public/docs-freshness.json` plus its summary, so it must run before `build` copies `public/` into `dist/`; running it here is what keeps `/docs/freshness` and every docs-page freshness badge describing the commit actually deployed instead of whenever someone last ran it by hand. Do NOT hand-run `build:vercel` as the frontend build: it builds the SDK/lib sub-artifacts, NOT the static HTML pages, so `dist/` ends up without `/`, `/create`, etc. and `check:dist` fails.
3. **Submit: `npm run deploy:gcp:submit`.** That is `db:check` (exit 4 if any migration is pending), then `check:gcloudignore`, then `audit:deploy` (no committed symlinks, no unsatisfied peer deps, no undeclared `api/` imports, decoder assets present, and an installed `node_modules` that matches `package-lock.json`), then `check:api-imports` (imports every module under `api/` and fails on any that throws while loading, since the server imports a handler on first hit and a load-time throw is a 500 on every request to that route rather than a build error), then the same `gcloud builds submit --config server/cloudbuild.yaml --region us-central1 --project aerial-vehicle-466722-p5`. `check:gcloudignore` simulates the upload and fails when a file the server imports at runtime would be excluded from the build context: an omitted `services/home-relay/src/token.js` shipped revision 00412 answering `/api/healthz` with `ERR_MODULE_NOT_FOUND` on every request (2026-09-04). The lockfile match exists because the image installs from the lockfile while every local test, and every deploy worktree that hardlinks this `node_modules`, runs whatever is on disk: a dependency bump nobody reinstalled shipped `@x402/extensions` 2.25.0 untested and every paid x402 route answered 500 on import (2026-09-17). If it fails, run `npm install`, then `npm test`, before deploying. Do NOT hand-run the bare `gcloud builds submit`: it is the only step that would have caught new code shipping over an old schema, and skipping it is how production ended up serving handlers that query columns their migration never applied (`column "blocks_behind" does not exist`, live on 2026-08-14). If the gate says migrations are pending, apply them first (`npm run db:status`, then `npm run db:migrate`); if it says `DATABASE_URL is not set`, step 1 did not copy `.env.local`. EVERY cloudbuild config must pin `serviceAccount: .../three-ws-build@...` (the default compute SA was deleted), and manual submits need `--substitutions=SHORT_SHA=manual$(date +%s)` when the config tags images with `$SHORT_SHA`.
4. **Sync new crons, then purge CDN: `npm run deploy:gcp:sync-crons`, then `npm run deploy:gcp:purge-cdn`.** The sync creates a Cloud Scheduler job for every cron in `vercel.json` that has none and touches no existing job; without it a new cron never fires (four sat dead for up to three weeks until 2026-10-08). The purge is synchronous on purpose. Do not re-add `--async`, or post-deploy checks read stale edge content and report phantom failures.
5. **Verify the deploy landed.** `curl -s https://three.ws/api/version` returns the live commit SHA + Cloud Run revision, and `npm run smoke:prod` sweeps every page declared in `data/pages.json` against the live site (`deploy:gcp` runs it automatically after the purge).
6. **Check the workers too: `npm run check:worker-drift`.** `deploy:gcp:full` ships `three-ws-api` only. Every service under `workers/` deploys from its own `cloudbuild.yaml` and nothing else rebuilds it, so a merged worker fix can sit unbuilt for months (`remesh-service` segfaulted in production on a July image while its fix waited from 2026-08-11 to 2026-11). The check is read-only and needs a gcloud login: it compares each serving image's build time and commit tag with the git history of its `workers/<dir>`, prints DRIFTED/drifting/current/NOT-DEPLOYED per service and region, exits 1 when any worker has carried unbuilt source past the grace window (`--grace-days`, default 7), and prints the exact `gcloud builds submit` command per drifted worker. Each of those rebuilds is a production rollout under gate 2; the `(extra)` region copies it lists are config-only `gcloud run services update --image` rolls, which are pre-approved. It shells out to gcloud and git, so it runs on the operator machine before or after a deploy, not as a Cloud Run cron.

The whole build + submit + purge in one command: `npm run deploy:gcp:full`.

---