run-integ git:20260919.f99d2e7A

Immutable. This exact content never changes and is served at /api/v1/blob/66a4a2daaa9a0da0.

---
name: run-integ
description: Run integration tests (deploy + destroy) against real AWS. Use when you need to verify cdkd works end-to-end with actual AWS resources.
argument-hint: "<test-name|all> [--synth-only] [--no-destroy]"
---

# Integration Test Runner

Run integration tests against a real AWS account: deploy actual resources,
verify, clean up.

## Arguments

- `test-name`: which test (see `ls tests/integration/`). If unspecified, ask via
  `AskUserQuestion` showing the options.
- `all`: run all tests
- `--synth-only`: synthesis only, skip deploy/destroy
- `--no-destroy`: deploy but don't destroy (debugging)
- `--deploy-args "<args>"`: forward extra args to the `cdkd deploy` invocation
  verbatim (destroy is unaffected).

## Steps

1. **Rebase, then build**: `git fetch origin` and rebase the branch onto current
   `origin/main` BEFORE the run — a real-AWS run against a stale base verifies
   code that is not what will merge, and nothing warns you. Then `vp run build`
   so `dist/` is current.

2. **List available tests**: `ls tests/integration/` — never a hardcoded list.

3. **Determine state bucket**: account via
   `aws sts get-caller-identity --query Account --output text`, then
   `cdkd-state-{accountId}` (region-free default). If absent, fall back to the
   legacy `cdkd-state-{accountId}-us-east-1` and note the deprecation.

4. **Pre-flight orphan scan** (mandatory): a prior run killed mid-deploy leaves
   orphans whose names match the stack about to deploy; cdkd's diff does not see
   them (not in state), so the deploy attempts CREATE and collides. Synth first
   (to learn stack name + resource types), then scan:

   ```bash
   # Always (cheap, broadly applicable):
   aws s3 ls s3://<bucket>/cdkd/<StackName>/ --region us-east-1
   aws iam list-roles --query 'Roles[?contains(RoleName, `<StackName>`)].RoleName' --output text
   aws lambda list-functions --region us-east-1 \
     --query 'Functions[?contains(FunctionName, `<StackName>`)].FunctionName' --output text

   # When the template uses Lambda EventSourceMapping (orphan ESM = AlreadyExists + rollback):
   aws lambda list-event-source-mappings --region us-east-1 \
     --query 'EventSourceMappings[?contains(FunctionArn, `<StackName>`)].[UUID,FunctionArn]' --output text

   # When the template uses VPC + Lambda VpcConfig (hyperplane ENIs outlive the function):
   aws ec2 describe-network-interfaces --region us-east-1 \
     --filters "Name=description,Values=AWS Lambda VPC ENI-<StackName>*" \
     --query 'NetworkInterfaces[].[NetworkInterfaceId,Status]' --output text
   ```

   **Anything found → abort** with the orphan list and cleanup commands; do NOT
   deploy on top of orphans.

5. **Run the test(s)**

   **Dispatch**: a `verify.sh` in `tests/integration/<test-name>/` owns its own
   deploy + verify + destroy cycle; the standard flow below is for plain smoke
   tests. Pre-flight (step 4) and post-run verification (steps 6 + 7) apply to
   BOTH paths.

   **CHECK FOR `verify.sh` BEFORE PICKING THE FIXTURE — the standard-flow branch
   is effectively unreachable from an agent session**: the harness's
   auto-approval classifier refuses a direct `cdkd deploy`, so a fixture WITHOUT
   a `verify.sh` dead-ends after dispatch. (rc=127 from `bash verify.sh` means
   the file does not exist, and bash says so on STDERR — if that line is missing
   you redirected stderr; read the log, not the exit code alone.) Run the
   standard flow only when a human drives the shell.

   - `cd tests/integration/<test-name>/`; `npm install` if no `node_modules`.
   - **If `verify.sh` exists**:
     `AWS_REGION=us-east-1 STATE_BUCKET=<bucket> bash verify.sh` — the script does
     its own deploy + destroy; steps 6/7 STILL run after. Propagate its exit code
     so a non-zero exit drives the failure path; never swallow failures.
   - **Otherwise** (standard flow):
     - `node ../../../dist/cli.js synth --region us-east-1`
     - **Multi-stack apps**: if synth lists more than one stack, pass `--all` to
       deploy and destroy (otherwise they fail with `Multiple stacks found`).
     - `node ../../../dist/cli.js deploy [--all] [<extra-deploy-args>] --region us-east-1 --state-bucket <bucket> --verbose`
     - `node ../../../dist/cli.js destroy [--all] --region us-east-1 --state-bucket <bucket> --force`

   **Never run it unwatched, and do not reach for `timeout`** — it is not in
   stock macOS (its absence is exit 127 in 0s, which reads as instant
   completion), and a hung run is indistinguishable from a slow one. Shell
   watchdog, firing made visible:

   ```bash
   LOG=$(mktemp)   # assign HERE: a separate block is a separate shell, and
                   # `> ""` is a loud failure that costs you the whole run
   bash verify.sh > "$LOG" 2>&1 &
   VPID=$!
   ( sleep 1500; kill -9 $VPID 2>/dev/null; echo "WATCHDOG_FIRED" >> "$LOG" ) &
   WPID=$!
   wait "$VPID"; RC=$?
   kill "$WPID" 2>/dev/null
   grep -c WATCHDOG_FIRED "$LOG" || echo "watchdog did not fire"
   echo "verify.sh rc=$RC"   # the verdict steps 6-11 read; nothing else carries it out
   ```

   The `grep` and the `rc` line are load-bearing (`kill -9` surfaces as rc=137,
   otherwise just a crash). **Steps 6-11 are LATER calls that read this output**
   — a marker or a `PASS` ledger row chained into this same call is written
   before any verdict exists.

6. **Verify cleanup**
   - `aws s3 ls s3://<bucket>/cdkd/ --region us-east-1` — no leftover state.
   - **The state bucket is VERSIONED**, so that listing shows nothing while every
     prior version stays readable (`aws s3 rm` writes a delete marker). For any
     fixture that WRITES a secret into state (redaction / scrub / drift fixtures
     do, deliberately), "the object is gone" is not "the content is gone" — the
     difference is a disclosure. Check versions:
     ```bash
     # Per state/lock key the fixture touched. Non-empty = content still readable.
     aws s3api list-object-versions --bucket <bucket> --prefix "cdkd/<Stack>/<region>/state.json" \
       --query "([Versions, DeleteMarkers][])[?Key=='cdkd/<Stack>/<region>/state.json'].VersionId" \
       --output text
     ```
     If a fixture seeded a secret, grep the surviving versions for it.
   - Verify the AWS resources are gone, per stack name from synth output, for
     the types the test actually created — the per-service listing commands are
     in `/cleanup` step 4; use the same ones with this run's stack prefix.
   - FSx tests take an extra check: destroy keeps CFn parity, so
     `DeleteFileSystem` takes a chargeable FINAL BACKUP by default, and it
     usually carries NO tags, so a name scan reports clean over a live billing
     backup. Attribute by the persisted file-system id
     (`aws fsx describe-backups --region us-east-1 --query 'Backups[?FileSystem.FileSystemId==\`{fs-id}\`].[BackupId,Lifecycle]'`);
     if the run's fs ids are unknown, list ALL backups and flag any unattributed
     entry for manual review.

7. **Auto-cleanup orphans (mandatory when destroy didn't fully succeed)** —
   trigger when the destroy step reported errors, OR step 6 found leftover state
   or any resource matching the stack prefix:
   - VPC-attached Lambda failures (commonest), **in delete order**: (1)
     hyperplane ENIs (`describe-network-interfaces --filters
     "Name=vpc-id,Values=<vpc>"` → `delete-network-interface`; re-poll `in-use`
     until `available`), (2) SecurityGroups, (3) Subnets, (4) VPC.
   - S3 state orphans: `aws s3 rm s3://<bucket>/cdkd/<StackName>/ --recursive`
     (or `cdkd state orphan <StackName>`, which also handles the lock key).
   - Other types: infer delete order from CFn dependency rules (children before
     parents). Always pass `--region`. Re-run step 6 after cleanup.

   **Never** end the run with orphans present (a NAT GW alone is ~$1/hr). If a
   resource genuinely cannot be deleted after reasonable retries, surface it with
   the exact ID, region, and what was tried — but only after the auto-cleanup
   pass.

8. **Report results**: pass/fail per test, resource counts, timing. Always state
   "destroy completed: 0 errors, 0 orphans" or itemize what remained.

9. **Set the `integ-destroy` markgate marker (only on full clean success)** —
   when the destroy step finished with **0 errors**, step 6 found **0
   leftovers**, and step 7 was skipped or re-checked clean. `mise trust` is
   UNCONDITIONAL and part of the pasted block: an untrusted `.mise.toml` makes
   `markgate set` die with a config-parse error naming no cause, and here that
   discards a real-AWS run that cannot be cheaply repeated.

   ```bash
   mise trust
   mise exec -- markgate set integ-destroy || {
     echo "markgate set integ-destroy FAILED — the marker was NOT recorded." >&2
     exit 1
   }
   # The marker, not the rc. `grep` exits 0 even for `no marker`, so it cannot
   # fail the block — ABSENCE of the line is the signal, which is what the
   # untrusted-config case produces (markgate status itself dies).
   mise exec -- markgate status | grep integ-destroy \
     || echo 'NO integ-destroy LINE — markgate status itself failed' >&2
   ```

   **Read BOTH the exit code and the status line** — they fail in different
   directions. The gate runs markgate's `hash: diff` mode, where `set` exits **2**
   if `origin/main` is unresolvable in this worktree or the branch has no delta
   against the merge base; the remedy is `git fetch origin`, never re-running the
   integ. An untrusted `.mise.toml` is the other direction: `mise` writes its
   error to stderr and the rc can still read as success, so only `markgate
   status` says whether a marker exists. Run from the PR's own worktree on the PR
   branch. If any success condition failed, do NOT set the marker.

10. **Post-run Docker sweep (mandatory for every `local-*` test)**, on top of
    step 6's AWS checks. A local run leaves containers and networks behind the
    way a deploy leaves AWS resources behind, and the run is not clean until all
    three listings come back empty:

    ```bash
    # All three MUST return empty, else show the orphan IDs and clean them up.
    # `-a`, not bare `docker ps`: a print-and-exit task container is already
    # `Exited` when this runs, so a running-only sweep reports clean over a
    # real orphan.
    docker ps -a --filter name=cdkd-local- --format '{{.ID}}'
    docker network ls --filter name=cdkd-local-task- --format '{{.ID}}'
    docker network ls --filter name=cdkd-local-svc- --format '{{.ID}}'
    ```

    Subnet-overlap gotcha: `cdkd local start-service` uses the FIXED subnet
    `169.254.171.0/24`, so a `local-start-*` test can fail with `Pool overlaps`
    even when all three are empty — a foreign leftover network (e.g. cdk-local's
    `cdkl-svc-*`) may own the subnet. Diagnose with
    `docker network inspect $(docker network ls -q) --format '{{.Name}} {{range .IPAM.Config}}{{.Subnet}}{{end}} {{len .Containers}}'`
    and remove the holder ONLY at 0 attached containers.

    A purely local run never touches AWS, so it cannot satisfy step 9's destroy
    conditions and does not set the `integ-destroy` marker;
    `local-invoke-from-state` is the exception — it exercises a real deploy +
    destroy as well, so it both sweeps clean here AND qualifies for step 9.

11. **Record the run in the integ ledger (MANDATORY — every run, pass OR fail)**:
    `docs/_generated/integ-last-run.tsv` is a COMMITTED update-type ledger (one
    row per test) feeding `/pick-integ`. Write it on EVERY invocation, right
    after step 9 (or right after a failure).

    Columns (TAB): `test  last_run_iso  result  duration_s  flow  note`. `result`
    is `PASS` only at the same bar as step 9's marker (destroy 0 errors / 0
    orphans; verify.sh exit 0), else `FAIL`. `last_run_iso` is UTC; `flow` is
    `verify.sh` or `standard`.

    **Use an ABSOLUTE path into the feature worktree for `LEDGER`** — the
    session's Bash cwd can silently reset to the MAIN worktree, and a relative
    write then dirties the main tree on `main`. Verify with `pwd`.

    ```bash
    LEDGER="/path/to/repo/.claude/worktrees/<branch>/docs/_generated/integ-last-run.tsv"
    # The file already exists; if it does not, copy its header from git history
    # first — `>>` alone creates it headerless and the normalizer preserves
    # whatever header it finds (none).
    TEST="<test-name>"; TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
    RESULT="PASS"; DUR="<seconds>"; FLOW="verify.sh"; NOTE="rc ok, orph clean"
    printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$TEST" "$TS" "$RESULT" "$DUR" "$FLOW" "$NOTE" >> "$LEDGER"
    vp run integ-ledger-normalize
    ```

    Commit the ledger update with the branch's changes. The one-row-per-test
    invariant is CI-enforced. When two lanes recorded the SAME test, the rebase
    conflicts and keep-both leaves two rows: re-run `vp run integ-ledger-normalize`
    after any rebase touching this file **and commit the rewrite before pushing**.
    Confirm with `git status --porcelain -- docs/_generated/`, never the
    normalizer's own output.

## Choosing the fixture

Which fixture to run is a coverage judgement, not a marker lookup.

- **A cross-cutting deploy/destroy change → run a BROAD fixture.** A test is
  "broad" iff its name is one of:

  ```text
  bench-cdk-sample
  lambda
  microservices
  drift-revert
  drift-revert-vpc
  multi-stack-deps
  multi-resource
  remove-protection
  export
  ```

  **Only five of the nine carry a `verify.sh`; from an agent session the other
  four cannot be run at all** (step 5's dispatch note). Runnable from a session:
  **`lambda`**, `drift-revert`, `drift-revert-vpc`, `remove-protection`,
  `export`. Human-driven shell only: `bench-cdk-sample`, `microservices`,
  `multi-stack-deps`, `multi-resource`. `lambda` is the cheap default — ~100 s,
  9-resource DAG across SQS / IAM / Lambda / LayerVersion / DynamoDB Table +
  GlobalTable. Re-derive the split with `ls tests/integration/<name>/verify.sh`.
  Nothing compares the copies of this list — `/pick-integ` and `/verify-pr` carry
  the others.

  **A narrow feature fixture is NOT a substitute**: a 2-stack feature fixture
  destroys cleanly without ever reaching the broad VPC / Lambda / multi-resource
  / Custom-Resource paths a cross-cutting change can break.

- **A local-execution change → run a `local-*` fixture** and complete step 10's
  Docker sweep. A cross-cutting `src/local/` change wants at least `local-invoke`
  + `local-start-api`.

- **A state schema version bump → run the matching
  `schema-v<N>-to-v<N+1>-migration` fixture.** A bump MUST be transparently
  auto-migrated, and only a real-AWS round-trip proves it (deploy under vN → swap
  binary → read works → next write upgrades silently → destroy clean).

## Important

- **Run `/review-pr` (and apply its fixes) BEFORE this skill when both are
  planned for the same PR** — the `integ-destroy` marker is digest-bound to its
  src scope, so a post-integ review fix stales the marker and forces a full
  real-AWS re-run.
- Always `--region us-east-1`; always destroy after deploy; if deploy fails,
  still attempt destroy to clean up partial state.
- **A run blocked BEFORE its assertions is not a test failure — say which it
  was.** (`cdkd gc` refuses while ANY stack holds a lock — account-wide, by
  design — so a parallel session's lock can stop a gc fixture before its first
  assertion.) Record it as `FAIL` (the bar is exit-code-based) with a ledger note
  naming the blocker and any hand-removed AWS resources, clean up what the
  aborted run leaked, and WAIT for the blocker to clear. Never
  `cdkd force-unlock` a lock you did not take — it belongs to another session's
  in-flight deploy.
- **Never report success on a successful deploy alone** — destroy must complete
  and the orphan check must pass.
- **Do NOT restart Docker to fix a hung docker-dependent run (`local-*`, or an
  ECR asset push) — on Docker Desktop the restart IS the likelier cause**: a
  quit-and-reopen can leave the self-respawning backend up while the app that
  serves the daemon's registry proxy never finishes launching. Three paths fail
  INDEPENDENTLY — host networking, container networking, and the daemon's own
  pull path — so name which one is down first:

  ```bash
  curl -s -o /dev/null -w '%{http_code}\n' --max-time 15 https://registry-1.docker.io/v2/  # 401 = HOST networking fine
  docker run --rm --entrypoint curl <an already-cached image> -s -o /dev/null \
    -w '%{http_code}\n' --max-time 15 https://registry-1.docker.io/v2/   # 401 = CONTAINER networking fine
  docker pull hello-world                         # hangs while both 401s return = DAEMON pull path only
  pgrep -f 'Docker Desktop' >/dev/null && echo app-running || echo APP-NOT-RUNNING
  ```

  **On that third signature, WAIT — it recovers on its own and a restart does not
  fix it.** Do not pipe the waiting probe through `tail`, which buffers away the
  progress lines. Never escalate to a factory reset or deleting Docker data (it
  destroys local images and volumes) — ask the maintainer. Clean up your own
  probes: `kill`ing a `docker pull` wrapper leaves the `com.docker.cli` child.
- **A fixture that discards the CLI's stderr cannot report its own failure.**
  `RESULT=$(${CDKD} ... 2>/dev/null | tail -1)` under `set -euo pipefail` prints
  the arm header and exits 1 with NO error text. The shape is banned and fenced
  ([../../rules/abort-capture.md](../../rules/abort-capture.md)): a failing
  invoke prints `[verify] command exited N` plus the stderr tail. A log that ends
  at an arm header with no error text means a fixture outside the fence — re-run
  that command with stderr attached BEFORE concluding anything.
- **Never bypass this skill** with direct `cdkd deploy` / `cdkd destroy` — the
  orphan-cleanup contract is part of the test, not optional.