git:20260828.46c809f to git:20260903.5b063ff

- Run integration tests against a real AWS account. These tests deploy actual AWS resources, verify them, and clean up.

- - `test-name`: Which test to run. Run `ls tests/integration/` to see all available tests. If not specified, use the `AskUserQuestion` tool to ask which test to run, showing the available options.

- - `all`: Run all tests

- - `--synth-only`: Only run synthesis, skip deploy/destroy

- - `--no-destroy`: Deploy but don't destroy (for debugging)

- - `--deploy-args "<args>"`: Forward extra arguments to the `cdkd deploy` invocation (e.g. `--deploy-args "--aggressive-vpc-parallel"`). Use when validating an opt-in deploy flag end-to-end against `bench-cdk-sample` etc. — the destroy step is unaffected (opt-in flags so far are deploy-only).

- 1. **Build first**: Run `vp run build` to ensure dist/ is up to date.

-

- 2. **List available tests**: Run `ls tests/integration/` to discover all test directories dynamically. Do NOT rely on a hardcoded list.

-

- 3. **Determine state bucket**: Resolve dynamically via `aws sts get-caller-identity --query Account --output text` to get the account ID, then construct `cdkd-state-{accountId}` (region-free, the current default since PR #62 / v0.11.0). If that bucket doesn't exist, fall back to the legacy `cdkd-state-{accountId}-us-east-1` and note the deprecation in the report.

- 4. **Pre-flight orphan scan** (mandatory — fail fast on prior-run leftovers instead of going through CREATE + rollback):

- Before invoking deploy, scan AWS for resources matching the stack name from this test that have no business existing yet. The scenario this catches: a previous integration run was killed mid-deploy, leaving orphan Event Source Mappings / Lambda functions / ENIs / IAM roles whose names match the stack about to be deployed. cdkd's diff calculation does NOT see these (they're not in state), so the deploy attempts CREATE — which collides with the orphans, fails immediately with `ResourceAlreadyExists`, and forces a CREATE-then-rollback cycle. Failing at the start is much cheaper than partway through.

- Synth first (without deploy) to learn the stack name and the resource types in the template, then for each scenario in scope run a targeted scan:

- # Always run these (cheap, broadly applicable):

- # Run when the template uses Lambda EventSourceMapping (the orphan ESM

- # case bit cdkd in 2026-05-02: AlreadyExists + rollback cycle on a fresh deploy):

- # Run when the template uses VPC + Lambda VpcConfig (hyperplane ENIs

- # outlive their function):

- **If anything is found**, abort the test run with a clear report listing the orphans and the cleanup commands the user should run (`aws lambda delete-event-source-mapping --uuid …`, `cdkd state destroy <StackName> --yes`, etc.) — do NOT proceed with deploy. Resuming on top of orphans is the failure mode this step exists to prevent.

-

- **If nothing is found**, the deploy can proceed cleanly.

- **Dispatch**: a `verify.sh` in `tests/integration/<test-name>/` is for tests with non-standard flows (drift-injection, multi-stage validation, etc.) — the script owns its own deploy + verify + destroy cycle. The standard flow below is for plain "deploy this stack and destroy it" smoke tests. Pre-flight (step 4) and the post-run verification (steps 6 + 7) apply to BOTH paths — they are the safety net that catches a buggy `verify.sh` leaking resources.

- **CHECK FOR `verify.sh` BEFORE PICKING THE FIXTURE — the standard-flow branch is effectively unreachable from an agent session.** The branch tells you to invoke `node ../../../dist/cli.js deploy …` directly, and the harness's auto-approval classifier refuses a direct `cdkd deploy`, so a fixture WITHOUT a `verify.sh` dead-ends after the dispatch. Measured 2026-08-20 on `bench-cdk-sample` (no `verify.sh`): `bash verify.sh` returned **rc=127 in 0 seconds**, and the standard-flow fallback was then denied. **rc=127 means the file does not exist, and `bash` says so on STDERR** — `bash: verify.sh: No such file or directory`, verified on both bash 5.x and macOS system bash 3.2. If that line is missing from what you are reading, you redirected stderr to a log and are reading the tail; go look at it rather than interpreting the exit code alone. (This paragraph previously claimed the failure was silent. It is not — the report it came from had redirected the output.) The practical workaround is a SELECTION decision, made here rather than after the dead-end: when the goal is a marker (`integ-broad` / `integ-destroy`), pick a broad-set fixture that HAS a `verify.sh` (`ls tests/integration/<name>/verify.sh` before committing to the name). Only run the standard flow when a human is driving the shell and can approve the `deploy` / `destroy` invocations.

- - Navigate to `tests/integration/<test-name>/`

- - Ensure dependencies: `npm install` if node_modules doesn't exist

- - **If `tests/integration/<test-name>/verify.sh` exists**, run it instead of the standard flow:

- - `AWS_REGION=us-east-1 STATE_BUCKET=<bucket> bash verify.sh`

- - The script is responsible for its own deploy + destroy cycle. Steps 6 (verify cleanup) and 7 (auto-cleanup orphans) STILL run after — do not skip them.

- - Propagate the script's exit code: a non-zero exit must drive the skill into the failure path so step 7's auto-cleanup fires. Do NOT swallow `verify.sh` failures.

- - Skip the synth / deploy / destroy commands below (the script does its own).

- - **Otherwise (no `verify.sh`)**, run the standard flow (synth → deploy → destroy):

- - Run synth: `node ../../../dist/cli.js synth --region us-east-1`

- - **Detect multi-stack apps**: read the synth output. If it lists more

- than one stack (e.g. `multi-stack-deps`, `composite-stack`,

- `cross-stack-references`), pass `--all` to deploy and destroy.

- Without `--all`, deploy/destroy will fail with `Multiple stacks

- found: ... Specify stack name(s) or use --all`.

- - Run deploy: `node ../../../dist/cli.js deploy [--all] [<extra-deploy-args>] --region us-east-1 --state-bucket <bucket> --verbose`

- - When `--deploy-args "<args>"` was passed to the skill, splice those args into the deploy invocation verbatim. Don't apply them to destroy.

- - Run destroy: `node ../../../dist/cli.js destroy [--all] --region us-east-1 --state-bucket <bucket> --force`

- - Check `aws s3 ls s3://<bucket>/cdkd/ --region us-east-1` to confirm no leftover state

- - **The state bucket is VERSIONED** (`cdkd bootstrap` enables it), so that

- listing shows nothing while every prior version is still readable: `aws s3

- rm` writes a DELETE MARKER, it does not remove content. For any fixture that

- WRITES a secret into state — the redaction / scrub / drift fixtures do it

- deliberately, to simulate a pre-GHSA binary — "the object is gone" is not

- "the content is gone", and the difference is a disclosure. Check versions,

- not just the current object:

- Two bugs of this shape shipped on 2026-08-19 and BOTH survived a green run:

- a fixture's version sweep lived only in `cleanup`, which runs from the trap

- the success path DISARMS (so on the normal path it never ran — one key had

- accumulated 30 versions), and the sweep itself iterated with `printf '%s' |

- tr | while read`, which drops the LAST field because there is no trailing

- newline. Neither is visible from reading the script; both are obvious the

- moment you COUNT what S3 holds afterwards. If a fixture seeded a secret,

- grep the surviving versions for it rather than assuming the sweep worked.

- - Also verify actual AWS resources are gone by checking with stack name prefix filters. Get stack names from the synth output, then for each stack name query AWS APIs filtered by that prefix:

- - For VPC-based tests also check: `aws ec2 describe-vpcs --filters "Name=tag:Name,Values={StackName}/Vpc" ...`

- - For FSx-based tests also check **final backups** (issue #1113): destroy keeps CFn parity, so `DeleteFileSystem` runs with API defaults, which TAKE a chargeable final backup for Windows/ONTAP (observed on OpenZFS too; `AutomaticBackupRetentionDays: 0` does not prevent it). The backup usually carries NO tags (`CopyTagsToBackups` defaults to false), so a tag/name scan reports clean over a live billing backup; attribute by the backup's persisted `FileSystem.FileSystemId` instead:

- `aws fsx describe-backups --region us-east-1 --query 'Backups[?FileSystem.FileSystemId==\`{fs-id}\`].[BackupId,Lifecycle]'` (use the file-system id(s) the run created, from deploy output or state). If the run's fs ids are not known, list ALL backups (`aws fsx describe-backups --region us-east-1 --query 'Backups[].{Id:BackupId,FsId:FileSystem.FileSystemId,Type:FileSystem.FileSystemType,Created:CreationTime}'`) and flag any untagged/unattributed entry for manual review (do NOT assume clean). Delete a confirmed leftover with `aws fsx delete-backup --backup-id {id} --region us-east-1`.

- - Only check resource types relevant to the test being run

-

- 7. **Auto-cleanup orphans (mandatory when destroy didn't fully succeed)**:

-

- **Trigger this step whenever any of the following is true:**

- - The `destroy` step in step 5 reported a non-zero error count (e.g. "X failed to delete")

- - Step 6 found a leftover S3 state file

- - Step 6 found any AWS resource matching the stack name prefix

-

- **What to do:**

- - For VPC-attached Lambda failures (the most common pattern), the typical orphan set is, **in delete order**:

- 1. Lambda hyperplane ENIs (`aws ec2 describe-network-interfaces --filters "Name=vpc-id,Values=<vpc>"` → `aws ec2 delete-network-interface`). Some may be `in-use` initially — re-poll until they go `available`, then delete.

- 2. SecurityGroups (`aws ec2 delete-security-group`) — must come after the ENIs that reference them are gone.

- 3. Subnets (`aws ec2 delete-subnet`) — must come after every ENI in them is gone.

- 4. VPC (`aws ec2 delete-vpc`) — last.

- - For S3 state orphans: `aws s3 rm s3://<bucket>/cdkd/<StackName>/ --recursive`. (`cdkd state orphan <StackName>` is the cdkd-native equivalent and also handles the lock key.)

- - For other resource types, infer the right delete order from CloudFormation dependency rules (children before parents).

- - Always specify the correct region (`--region`).

- - Re-run step 6 after cleanup to confirm zero orphans remain.

-

- **Never** end the run with orphan resources still present in AWS. Cost (NAT GW alone is ~$1/hr) and account hygiene make this non-negotiable. If a resource genuinely cannot be deleted after reasonable retries, surface it to the user with the exact ID, region, and what was tried — but only after the auto-cleanup pass.

- 8. **Report results**: Show pass/fail for each test, including resource counts and timing. Always state explicitly "destroy completed: 0 errors, 0 orphans" or itemize what remained / what was force-cleaned.

- 9. **Set the `integ-destroy` markgate marker (only on full clean success)**:

- When — and ONLY when — all of the following hold:

- - the destroy step finished with **0 errors**,

- - step 6 found **0 leftover resources**,

- - step 7 was either skipped (because nothing to clean up) or completed with the post-cleanup re-check showing 0 orphans,

- record the gate so subsequent `gh pr merge` calls are unblocked:

- **Check the exit code; do not fire and forget.** Under the old

- `hash: files` mode this command could not fail. The gate now runs

- markgate 0.4's `hash: diff` (see `.markgate.yml`), where `set` exits

- **2** if `origin/main` is unresolvable in this worktree or the branch

- has no delta against the merge base — and it writes that to stderr,

- so an unchecked call looks silent and successful. Failing to notice

- means you burned a real-AWS deploy + destroy and recorded nothing:

- the merge is still blocked, and the natural reaction is to run the

- integ AGAIN rather than to `git fetch origin`. Run from the PR's own

- worktree on the PR branch, and if it exits 2, fix the base ref rather

- than re-running the integ.

-

- If any of the above failed, do NOT set the marker — that is the

- whole point of the gate. The hook

- `.claude/hooks/integ-destroy-gate.sh` will block `gh pr merge` for

- any PR that touches deletion-related code (see `.markgate.yml`

- `integ-destroy.include`) until this marker is fresh, so a

- destroy-untested change physically cannot reach main.

-

- 10. **Set the `integ-local` markgate marker (only for `local-*` tests, on full clean success)**:

-

- When the integ test name starts with `local-` (i.e. `local-invoke`,

- `local-start-api`, `local-run-task`, `local-invoke-container`,

- `local-invoke-from-state`, `local-invoke-layers`,

- `local-invoke-python` / `-ruby` / `-java` / `-dotnet` / `-provided`,

- `local-start-api-cors`, or any future `local-*` test), ALSO set

- the `integ-local` marker after a clean Docker run.

- Required cleanup verification BEFORE setting the marker (in

- addition to the conditions above for `integ-destroy`):

- # All three queries MUST return empty. If any lists IDs, the

- # marker is NOT set and the user is shown the orphan container /

- # network IDs to clean up via `docker rm -f` / `docker network rm`.

- # `-a`, not a bare `docker ps`: a print-and-exit task container is already

- # `Exited` by the time this runs, so a running-only sweep reports clean over

- # a real orphan. That blind spot let one survive every `local-run-task-from-state`

- # run until it was caught twice in a row while gating #2183.

- Subnet-overlap gotcha (seen 2026-07-27): `cdkd local start-service`

- creates its shared network on the FIXED subnet `169.254.171.0/24`,

- so a `local-start-*` test can fail with `Pool overlaps with other

- one on this address space` even when all three queries above are

- empty — a foreign leftover network (e.g. cdk-local's `cdkl-svc-*`

- from a crashed run) may own that subnet. Diagnose with

- `docker network inspect $(docker network ls -q) --format

- '{{.Name}} {{range .IPAM.Config}}{{.Subnet}}{{end}} {{len .Containers}}'`

- and remove the holder ONLY when it has 0 attached containers (a

- non-empty one may belong to a live parallel run).

-

- When all three return empty AND the integ test exited cleanly:

-

- ```bash

- mise exec -- markgate set integ-local

- ```

-

- The hook `.claude/hooks/integ-local-gate.sh` blocks `gh pr merge`

- (and `git merge`) for any PR that touches `cdkd local *` code

- (see `.markgate.yml` `integ-local.include`) until this marker is

- fresh, so an unverified local-execution change physically cannot

- reach main. Same TTL (14d) and same "do NOT call markgate set

- directly to bypass" rule as `integ-destroy`.

-

- Note: Non-`local-*` tests (e.g. `bench-cdk-sample`, `lambda`)

- leave the `integ-local` marker untouched. The two markers are

- independent — a `lambda` integ run does not refresh `integ-local`,

- and a `local-invoke` run does not refresh `integ-destroy` unless

- its `verify.sh` also exercises a real-AWS deploy + destroy (the

- `local-invoke-from-state` test does, so it can set BOTH).

- 11. **Set the `integ-broad` markgate marker (only for BROAD integ tests, on full clean success)**:

- The broad-integ set covers tests that exercise multi-resource

- deploy/destroy paths (VPC + NAT + Lambda hyperplane ENI, Custom

- Resource, DAG with 5+ types across 2+ levels). A test is "broad"

- **Only FIVE of those nine carry a `verify.sh`, and from an agent session

- the other four cannot be run at all.** Step 5's dispatch note explains why

- (the standard flow needs a direct `cdkd deploy`, which the harness's

- auto-approval classifier refuses), but it leaves the selection to be

- discovered one dead-end at a time. Measured 2026-08-26 while gating a

- `destroy.ts` change: `multi-stack-deps` is the obvious pick for an `--all`

- loop change and is one of the four without one.

-

- Runnable from a session: **`lambda`**, `drift-revert`, `drift-revert-vpc`,

- `remove-protection`, `export`.

- Human-driven shell only: `bench-cdk-sample`, `microservices`,

- `multi-stack-deps`, `multi-resource`.

-

- `lambda` is the cheap default — a ~100 s, 9-resource DAG across SQS / IAM /

- Lambda / LayerVersion / DynamoDB Table + GlobalTable, which is what makes it

- broad. Re-derive the split with

- `ls tests/integration/<name>/verify.sh` rather than trusting this list if a

- fixture has since gained one.

-

- (Keep this list in sync with `.claude/hooks/integ-broad-gate.sh`'s

- error message and the matching memory rule

- `feedback_cross_cutting_needs_broad_integ.md`. The sync is now

- enforced rather than requested: `tests/unit/scripts/cross-cutting-list-sync.test.ts`

- compares all seven copies of this list, after the hook's own header

- comment was found sitting at 8 entries while every other copy had 9.)

- When the integ test name is in the broad set AND the destroy step

- finished cleanly with 0 errors / 0 orphans (= the same conditions

- that flip `integ-destroy`), ALSO record the broad-integ sentinel

- and flip the marker:

- # Sentinel content is informational (human-readable, helps the

- # next /verify-pr run know which integ was last run). The

- # `integ-broad` markgate gate's include scope is just this file,

- # so writing the test name flips its digest naturally.

- The hook `.claude/hooks/integ-broad-gate.sh` blocks `gh pr merge`

- for any PR that touches cross-cutting deploy/destroy code (see the

- hook's `CROSS_CUTTING_REGEX` for the canonical list) until this

- marker is fresh. Same 14d TTL and same "do NOT call markgate set

- directly to bypass" rule as the other AWS-coupled gates.

-

- **Narrow feature integs do NOT set this marker** — e.g.

- `import-value-strong-ref` flips `integ-destroy` but leaves

- `integ-broad` alone. This is the structural fix for the PR #348

- incident: a 2-stack S3+SSM feature fixture is sufficient for the

- feature's correctness, but a cross-cutting change to

- `src/deployment/deploy-engine.ts` etc. needs the broader VPC /

- Lambda / multi-resource coverage that only the broad set provides.

- `schema-v*-to-v*-migration` tests, on full clean success)**:

-

- cdkd's S3 state schema is the actual user contract. A schema

- version bump (e.g. v5 -> v6) MUST be transparently auto-migrated

- by the new binary AND verified by a real-AWS integ test that

- proves the round-trip: deploy under vN -> swap binary -> read

- works against the vN state without re-deploying -> next write

- upgrades to vN+1 silently -> destroy clean / 0 orphans.

-

- A test is "schema-migration" iff its name matches the pattern

- `schema-v<N>-to-v<N+1>-migration` (e.g.

- `schema-v5-to-v6-migration`). Test fixtures live under

- `tests/integration/schema-v<N>-to-v<N+1>-migration/`.

-

- When the test name matches AND the destroy step finished cleanly

- with 0 errors / 0 orphans (= the same conditions that flip

- `integ-destroy`), ALSO record the schema-migration sentinel and

- flip the marker. Unlike `integ-broad`, this gate's marker uses

- `src/types/state.ts` as its include scope directly, so the

- sentinel is informational only (no separate sentinel file is

- needed for markgate's digest):

- The hook `.claude/hooks/integ-schema-migration-gate.sh` blocks

- `gh pr merge` for any PR that bumps the `StackState.version`

- literal type in `src/types/state.ts` (detected via precise `gh

- pr diff` grep — non-bump edits to state.ts pass through with no

- false positive) until this marker is fresh. Same 14d TTL and

- same "do NOT call markgate set directly to bypass" rule as the

- other AWS-coupled gates.

-

- **Non-schema-migration tests do NOT set this marker** — e.g.

- `lambda` flips `integ-destroy` + `integ-broad` but leaves

- `integ-schema-migration` alone. A schema-bump PR must run a

- test named exactly `schema-v<N>-to-v<N+1>-migration` to clear

- this gate. See memory rule

- `feedback_schema_version_migration_integ_required.md` for the

- full migration-test checklist + the absolute requirement that

- auto-migration must be transparent (no user action required

- on upgrade).

-

- 13. **Record the run in the integ ledger (MANDATORY — every run, pass OR fail)**:

- `docs/_generated/integ-last-run.tsv` is a COMMITTED (NOT gitignored), update-type

- ledger — one row per test — so anyone can see when each integ last ran and whether

- it passed. This answers "has this been run recently?" / "this hasn't run in months,

- it's risky to trust" without trawling CI history, and feeds `/pick-integ`. **Write it

- on EVERY `/run-integ` invocation, pass or fail**, right after the marker steps above

- (or right after a failure is recorded — never skip it on failure).

- Columns (TAB-separated): `test last_run_iso result duration_s flow note`

- - `result`: `PASS` only when the run finished cleanly (destroy 0 errors AND 0 orphans;

- verify.sh exited 0) — the SAME bar as the markgate markers. Otherwise `FAIL`.

- - `last_run_iso`: `date -u +%Y-%m-%dT%H:%M:%SZ` (UTC). `flow`: `verify.sh` or `standard`.

- - `duration_s`: optional wall-clock seconds. `note`: short reason / finding one-liner.

- Append the new row, then normalize the whole file (do NOT hand-drop the old row —

- the normalizer collapses to the newest row per test and re-sorts):

- **Use an ABSOLUTE path into the feature worktree for `LEDGER`.** The

- session's persistent Bash cwd can silently reset to the MAIN worktree

- (observed right after a background integ task completes — the cwd-race

- signature in `main-tree-git-cwd-detector.sh`), and a relative

- `docs/_generated/...` write then dirties the main tree on `main`, which

- the `main-tree-dirty-detector` hook flags and you must then repair.

- Verify with `pwd` immediately before the write, or hardcode

- `LEDGER=/abs/path/to/.claude/worktrees/<branch>/docs/_generated/integ-last-run.tsv`.

- # ABSOLUTE path into the feature worktree (see the cwd-race note above);

- # substitute your worktree path, do not rely on the session cwd.

- # Bootstrap the header if the file is somehow absent — `>>` alone would create it

- # headerless, and the normalizer preserves whatever header it finds (none), silently

- # dropping the invariant + do-not-hand-edit notice.

- Commit the ledger update with the branch's changes (it is part of the integ run record;

- the file is intentionally committed so the last-run history is shared across sessions).

- The one-row-per-test invariant is now **enforced by CI** (issue #1112): the

- `integ-last-run ledger is normalized` step re-runs the normalizer and fails on any

- diff. The whole-file rewrite makes a REPLAYED commit reproduce the file byte-for-byte

- instead of appending a duplicate row — but that covers the replay case only. When two

- lanes recorded the SAME test, the rebase is a CONFLICT, and resolving it keeps both

- rows: re-run `vp run integ-ledger-normalize` after any rebase that touched this file

- **and commit the rewrite before pushing**. Measured 2026-08-29: the normalizer was run

- after the push and its output never committed, so CI turned the PR red. Confirm with

- `git status --porcelain -- docs/_generated/` rather than the normalizer's own output —

- a grep over a check log that keeps only error lines reports a dirty tree as clean.

- (`--check` reports without writing.)

- planned for the same PR.** The `integ-destroy` / `integ-broad` /

- `integ-local` markers are digest-bound to their src scopes, so any

- post-integ review fix that touches a provider or deploy/destroy file

- stales the marker and forces a full re-run against real AWS. Review

- polish first, integ last (memory rule

- `feedback_apply_review_polish_before_integ` — recurred 2026-07-31 on the

- #1282 PR, costing a second CloudFront deploy + destroy cycle).

- - Always use `--region us-east-1` for integration tests

- - Always destroy after deploy to avoid leftover resources

- - If deploy fails, still attempt destroy to clean up partial state

- - **A run blocked BEFORE its assertions is not a test failure — say which it was.**

- `cdkd gc` refuses while ANY stack in the state bucket holds a lock (an

- account-wide guard, deliberate and documented at its site), so on an account

- with parallel agent sessions a gc fixture can be stopped before it reaches a

- single gc assertion. That happened twice in a row on 2026-08-19, from two

- different foreign stacks in a different region than the one under test. Both

- are recorded as `FAIL` rows because the bar is exit-code-based, and both notes

- say plainly that the fix was not at fault.

- When a run dies before its assertions, do all three: clean up as usual (an

- aborted run still leaks — one of these left a state file, a Lambda and an IAM

- role behind), write the ledger note so it names the blocker rather than

- implying the change is broken, and WAIT for the blocker to clear rather than

- forcing past it. Never `cdkd force-unlock` a lock you did not take: it belongs

- to another session's in-flight deploy.

- - **Never report success based on a successful deploy alone** — destroy must complete and orphan check must pass

- - **Never bypass this skill** by calling `cdkd deploy` / `cdkd destroy` directly from a shell — the orphan-cleanup contract above is part of the integration test, not optional