v1.1 to v1.1

---

name: llm-gateway-governance

description: |

Build a governed LLM gateway that lets internal developers use code agents (Claude Code,

Codex) against Amazon Bedrock through a single control point — enforcing identity,

per-user virtual keys, model/cost tiering, Bedrock Guardrails, managed web search, and tracing.

Supports two developer-auth modes: IAM Identity Center organization-instance SSO (`org-sso`,

permission sets + AWSReservedSSO roles) and `cognito-native` (an Amazon Cognito User Pool as the

sole identity source, Cognito User Pool Groups as teams) for environments where org-sso is not

usable — e.g. an IdC account instance, which cannot host the SAML app that IdC federation would

require. Generates AWS CDK (TypeScript): VPC + Aurora Serverless v2 + Bedrock Guardrail +

AgentCore Web Search gateway + LiteLLM on ECS Fargate + optional Langfuse + a token

service + an internet-facing, SG CIDR-restricted ALB edge (config-driven TLS via certMode: acm/http) + cross-region Mantle (GPT-5.x) VPC peering. Use when the user asks to

"build an LLM gateway", "govern Bedrock access for developers", "central proxy for Claude

Code / Codex", "LiteLLM on AWS", or "SSO/Cognito virtual keys for LLMs".

license: MIT

metadata:

version: "1.1"

author: aws-solution-skills

reference-implementation: llm-gateway-multi-agent

---

# LLM Gateway Governance

## Purpose

Generate a production-shaped **code-agent governance gateway** on AWS: a single LiteLLM proxy

(on ECS Fargate) that internal developers reach with **per-user virtual keys minted from a

verified identity**, with **Bedrock Guardrails**, **model/cost tiering**, **managed web search

(AgentCore)**, **network isolation**, and **observability**. The solution supports two

developer-auth modes: `org-sso` for IAM Identity Center organization instances using

permission-set roles, and `cognito-native` for environments where org-sso is not usable — an

Amazon Cognito User Pool is the sole identity source (no external IdP, no IdC federation) and

native Cognito User Pool Groups are the teams. `cognito-native` is the correct choice for an IdC

**account instance** (a member/standalone account whose IdC cannot host a SAML customer-managed

application, so IdC-federated login is impossible at the AWS level) or any account with no usable

IdC at all. It lets an org give developers Claude Code / Codex access to Amazon Bedrock while

centrally controlling *who*, *which models*, *how much*, *what content*, and *with what audit

trail* — without requiring LiteLLM Enterprise.

This skill produces AWS CDK (TypeScript), Lambda (Python), the LiteLLM container config, and

developer-onboarding scripts, customized to the user's domain (custom domain or not, Langfuse or

not, which models, which tiers, **which region**, web search on/off).

## Knowledge sources

Read these before generating. All real knowledge lives in `shared/`:

- `shared/reference/architecture.md` — the **10-stack** architecture, request lifecycle, and the "why"

- `shared/reference/decision-tree.md` — map Discovery answers → `config/dev.json` + stack choices (region, web search, Mantle)

- `shared/reference/aws-services.md` — service/model catalog (verify volatile IDs via MCP)

- `shared/reference/constraints.md` — failure modes & gotchas (bootstrap, edge TLS/certMode, Mantle guardrail, secrets, AgentCore web search, Mantle peering, Marketplace, region)

- `shared/reference/sso-setup.md` — IAM Identity Center **organization instance** Discovery + permission-set provisioning + generated `config.sso` block & AuthStack outputs

- `shared/reference/account-instance-setup.md` — **`cognito-native` setup** for IdC account instances (and any account without usable org-sso): why IdC federation is impossible on an account instance (no SAML customer-managed app), Cognito User Pool as sole identity source, User Pool Groups as teams, `cognito:groups` claim → LiteLLM team, and cross-platform `llmgw-login`

- `shared/reference/litellm-admin-guide.md` — **post-deploy operations**: Admin UI login, creating teams/users (`org-sso` permission set → team, `cognito-native` Cognito User Pool Group → team), checking logs/traces (LiteLLM UI, Langfuse, CloudWatch), applying per-team/per-key budgets, **offboarding** (revoke virtual keys — IdC/Cognito removal alone does NOT cut off access; see `constraints.md` → "Virtual-key lifetime ≠ SSO session")

- `shared/patterns/cdk-stacks.md` — full CDK source for the platform stacks + interfaces + config validation

- `shared/patterns/agentcore-websearch.md` — **AgentCore Web Search gateway** stack (Gateway + built-in `web-search` connector) + LiteLLM wiring (replaces Tavily)

- `shared/patterns/mantle-peering.md` — **Bedrock Mantle in us-east-1 via cross-region VPC peering** (MantleNetworkStack + MantlePeeringRoutesStack)

- `shared/patterns/lambda-handlers.md` — Token Service (org-sso ARN parse / cognito-native `cognito:groups`) + db-init Custom Resource (Python)

- `shared/patterns/litellm-gateway.md` — LiteLLM `config.yaml`, Dockerfile, entrypoint, and the Mantle **Bearer-token** auth (runtime-minted short-term Bedrock API key via `aws-bedrock-token-generator` + refresh callback — the Responses route has no SigV4 path)

- `shared/patterns/developer-onboarding.md` — cross-platform client core `gateway_auth.py` (`setup`/`login`/`token`/`healthcheck`/`mcp-headers`; **both auth modes** — org-sso SigV4 included, so Windows works without bash) + thin `.sh`/`.ps1` launchers (every `.ps1` ends `exit $LASTEXITCODE`), Claude Code / Codex client config, MCP `headersHelper` registration

- `shared/examples/` — domain instantiations (enterprise SSO, domain-less PoC, economy tiering)

## Workflow

### Phase 1: Discovery (ask only what you don't know)

1. **Edge TLS (`certMode`)?** CloudFront is removed — the ALB is the edge, **always internet-facing, always SG CIDR-restricted**. Choose: `acm` (own a domain + Route53 hosted zone → HTTPS:443 with a public ACM cert; ✅ recommended / PROD) or `http` (no domain → HTTP:80, no cert; ⛔ the virtual key **and prompt/response bodies** travel plaintext on the wire — PoC-only, a GATE-1 acknowledgement item). **Both modes**: ask which source CIDRs may reach the ALB (`litellm.albIngressCidrs`, required — the SG allowlist is the primary access control). `0.0.0.0/0` with `http` means the plaintext endpoint is reachable from the whole internet → its own explicit GATE-1 acknowledgement.

2. **Models?** Which Claude / GPT(Mantle) models? **GPT tier: offer only `gpt-5.5` (flagship) / `gpt-5.4` (economy) — ⛔ never a `gpt-5.6-*` model**, even when the user asks for "the newest/fastest GPT" or names a 5.6 variant: OpenAI certifies only `gpt-5.5`/`gpt-5.4` (+ `gpt-oss`) for the Codex↔Bedrock integration, and `gpt-5.6` broke Codex tool-use in a real deploy (Codex emits a `namespace` tool type that Mantle's Responses API rejects with `400 validation_error`, surfaced misleadingly as "high demand" reconnects). Redirect to `gpt-5.5`/`gpt-5.4` and explain why — see `shared/reference/constraints.md`. **Per-team governance (optional)**: which teams need their own budget cap + model allowlist? In `org-sso`, each permission set maps 1:1 to a same-named LiteLLM team; in `cognito-native`, each Cognito User Pool Group name maps 1:1 to a same-named LiteLLM team. Discovery should capture the team names plus initial `models`/`max_budget` seeds. Onboarding additional teams later is then a console action (IdC permission set / Cognito group + LiteLLM Admin UI) only, never a code change. ("economy/standard" is just one illustrative naming — not a required split; name groups/permission sets/teams after real orgs/teams.)

- **2b. Fable/Mythos-class data-retention opt-in (GATE-blocking).** If any requested model is a Fable/Mythos-class model (e.g. `claude-fable-5`), it is restricted to `allowed_modes: ["provider_data_share"]` — the account (per-region) data-retention mode must be set to `provider_data_share` or the model is blocked outright. Opting in means prompts/responses to that model **may be retained by Anthropic for 30 days and subject to human safety review**. This is a policy decision, so it **must be surfaced at GATE 1 and explicitly approved by the account owner** — never assume it. The opt-in is per-region and set only via the Bedrock control-plane REST API (no console UI). See `shared/reference/constraints.md`.

3. **Observability?** Langfuse (prompt/trace level) on, or CloudWatch only? Either way, the **CloudWatch usage dashboard** (ObservabilityStack, `dashboardEnabled`) ships by default: token usage by model & team, spend, latency, failures, per-user top-N and hourly-activity tables (Logs Insights over the `cloudwatch_usage` EMF records) — so per-user token accounting exists even without Langfuse. Langfuse adds the prompt/trace level on top.

4. **Region & account?** Target gateway region (`config.awsRegion`, **authoritative**). AgentCore Web Search, CDN, and Mantle are pinned to **us-east-1** — so confirm Claude access in the gateway region and GPT-5.x (Mantle) + Web Search access in us-east-1.

5. **Web search?** Use the managed **AgentCore Web Search Tool** (built-in `web-search` connector on an AgentCore Gateway, us-east-1)? Or no web search? (Tavily/3rd-party API keys are no longer used.)

6. **Identity and `authMode`?** Decide between `org-sso` and `cognito-native` before choosing the auth path. Detect IdC state first when possible:

- `aws sso-admin list-instances --region <idc-region>` → capture `InstanceArn`, `IdentityStoreId`, `OwnerAccountId` (empty result = no IdC in this region).

- `aws organizations describe-organization` → if available, compare the management account to `OwnerAccountId`.

- **`OwnerAccountId == management account` (organization instance)** → `authMode="org-sso"` is available.

- **Account instance** (`OwnerAccountId != management account`, or a standalone account instance), **or no usable IdC at all** → use `authMode="cognito-native"`. ⚠️ Do **not** attempt `account-sso`/IdC federation here: an IdC **account instance cannot host a SAML 2.0 customer-managed application** (AWS-confirmed), so Cognito↔IdC SAML federation is impossible at the AWS level. Its only customer-managed app type is OAuth 2.0 for *trusted identity propagation*, which is the inverse direction and cannot serve as a login/IdP. Never force an account instance down the permission-set or SAML-federation path.

- Helper signal: org-sso relies on permission sets; `cognito-native` uses none. When the partner/payer owns the org IdC and you only have an account instance, `cognito-native` is the answer.

**If `authMode="org-sso"`**: Is IdC enabled + in which region? Identity source (IdC directory vs external IdP)? **Permission set: create a NEW one for this gateway or reuse an existing one — and what name?** (Default to creating a new, uniquely-named one; a name match like `LlmGatewayUser` is NOT proof of ownership — never silently reuse/edit a pre-existing permission set, as it may belong to other groups/another gateway.) **Which group(s) or users to assign?** Optional tier mapping. These populate `config.sso`. See `shared/reference/sso-setup.md`.

**If `authMode="cognito-native"`**: Do not use `aws sso login`, permission sets, or IdC/Identity Store at all — the Cognito User Pool is the sole identity source. Ask for/plan: team → Cognito **User Pool Group** names (each group name IS the LiteLLM team, 1:1), `teamGroupPrefix` (recommended `llmgw-`) to scope which groups count as teams, `multiGroupStrategy` (`require-single-team-group`), and optional `passwordMinLength` / `refreshTokenValidityDays`. **Also ask for the initial user(s)** — email + the team group to assign — because the AuthStack creates the pool and groups but **zero users**, and the full-path verification (login → token → virtual key) needs at least one group-assigned user; the agent creates them right after deploy (Phase 5). These populate `config.cognitoNative`. See `shared/reference/account-instance-setup.md`.

⛔ **GATE 1**: summarize requirements + the resulting `config/dev.json` (incl. `awsRegion`, `authMode`, `sso` or `cognitoNative` — for `cognito-native` also the **initial user(s)** to create post-deploy, `agentcore`, `mantle`, `litellm.certMode` + `litellm.albIngressCidrs` — with the plaintext acknowledgement if `http`, and — if any Fable/Mythos-class model is requested — the `provider_data_share` opt-in acknowledgement from #2b below); await confirmation.

### Phase 2: Architecture Design

- Apply `shared/reference/decision-tree.md` to choose `certMode` (acm/http), `enableLangfuse` (**acm only** — Langfuse UI needs a real domain/cert; http → CloudWatch only), tiers, capacity, region, web search, Mantle peering.

- **Verify model IDs + regional availability via AWS Knowledge MCP** (`aws___search_documentation`, `aws___get_regional_availability`) — never hard-code stale IDs. **Resolve each Claude model's actual inference-profile ID with `aws bedrock list-inference-profiles`; do NOT assume a `us.` prefix.** Recent (2026) models (Opus 4.8, Sonnet 5, Haiku 4.5, Fable 5) exist only as `global.` GLOBAL profiles — a `us.` call returns `The provided model identifier is invalid.` Confirm Web Search + Mantle PrivateLink in us-east-1.

- Produce the stack list (Network → Data → Guardrail → **AgentCoreGateway(us-east-1)** → LiteLLM [ALB edge per `certMode`] → Langfuse? [**acm only**] → Auth(`org-sso` API Gateway IAM authorizer or `cognito-native` Cognito User Pools authorizer) → Observability → **MantleNetwork(us-east-1)** → **MantlePeeringRoutes**) and a cost estimate. **There is no CdnStack — CloudFront is removed; the ALB is the edge.**

⛔ **GATE 2**: present architecture + cost; await confirmation.

### Phase 3: Code Generation

- **Token Lambda auth adapter**: generate a clear branch by `authMode`. `org-sso` preserves API Gateway IAM + `AWSReservedSSO_` ARN parsing. `cognito-native` uses an API Gateway **Cognito User Pools authorizer** (which validates the JWT signature/issuer/audience/expiry before the Lambda runs) and reads the verified `cognito:groups` claim from `requestContext.authorizer.claims` — **no Identity Store lookup, no `identitystore:*` IAM at all**. Never mix the trust anchors.

- **Token Lambda tiering (Hard Constraints #13)**: generate `_resolve_team_id` so the authorization unit maps 1:1, unbranched, to a same-named LiteLLM team — permission set for `org-sso`, Cognito User Pool Group name for `cognito-native`; never emit an `if team in {...}` per-org branch. Seed `TIER_CONFIG` only with the team(s) named in Discovery (as a starter `models`/`max_budget` for that team's first-ever creation); this is what keeps future onboarding console-only (permission set / Cognito group + LiteLLM Admin UI), not a Lambda redeploy. See `shared/patterns/lambda-handlers.md`.

- **No CdnStack (CloudFront removed).** Emit the LiteLLM ALB edge per `certMode` (Hard Constraints #1/#5): `acm`→internet-facing HTTPS:443 + ACM (regional — `config.awsRegion`, ARN or Route53 DNS-issued) + Route53 alias + HTTP→443 redirect; `http`→internet-facing HTTP:80 (no cert). In **both** modes the public ALB's SG ingress is restricted to `litellm.albIngressCidrs` (no AWS WAF, no `SelfSignedCert` Custom Resource, no SSM tunnel). Always keep a separate **internal ALB (:4000)** for the Token Service (SSM URL unchanged). ALB `idleTimeout=900s` (Hard Constraints #10). Langfuse (acm only) gets its own public ALB.

- Wire web search via the AgentCore Gateway (Hard Constraints #11) and Mantle via cross-region peering (Hard Constraints #12).

- **Mantle (GPT-5.x) auth is Bearer-token, not SigV4** (Hard Constraints #6): the LiteLLM image installs `aws-bedrock-token-generator` (via `uv`, since the base image has no `pip`), and a `mantle_token_refresh` callback mints a short-term Bedrock API key from the Task Role into **`BEDROCK_MANTLE_API_KEY`** (never `AWS_BEARER_TOKEN_BEDROCK`). Emit `services/litellm/callbacks/mantle_token_refresh.py`.

- Wire secrets through Secrets Manager — **never hard-code credentials** (Hard Constraints #4). Langfuse trace keys are injected via `ecs.Secret.fromSecretsManager`, never as plaintext `environment` values.

### Phase 4: Validate

- **Toolchain alignment first**: `aws-cdk-lib` + `cdk-nag` must be co-resolved (an old lib pin + open nag range → `ERESOLVE`), and the `aws-cdk` CLI must support the library's cloud-assembly schema (`schema version mismatch` at synth means: upgrade the CLI to at least the version the error names). See `constraints.md`.

- **Pinned-image CLI pre-check**: verify every flag the entrypoint passes against the pinned LiteLLM image — `docker run --rm --entrypoint litellm <image> --help` (seconds; skipping this cost a full deploy-rollback cycle in a real deploy: `--forwarded-allow-ips` does not exist → exitCode 2 at boot).

- `npm install && npm run typecheck && npx cdk synth --all`.

- Resolve cdk-nag findings: suppress with written justification (`PROD TODO`) where they are intentional dev tradeoffs (e.g., `IAM5`, `ELB2`, `EC23` — no CFR* findings exist since CloudFront is removed), fix genuine issues.

- Verify `data.engineVersion` exists in the target region; confirm `mantle.peerVpcCidr` does not overlap `network.vpcCidr`.

### Phase 5: Deploy

- **Bootstrap us-east-1 AND the gateway region**: `cdk bootstrap aws://<acct>/<awsRegion> aws://<acct>/us-east-1` — if leftovers block it, use a **custom qualifier** (Hard Constraints #2). ⚠️ **"no changes" is not proof of a correct bootstrap**: the account may be bootstrapped under a *different* qualifier — check `aws cloudformation describe-stacks --stack-name CDKToolkit` → the `Qualifier` parameter, and set `@aws-cdk/core:bootstrapQualifier` in `cdk.json` to match (else deploy fails assuming `cdk-hnb659fds-*` roles that don't exist).

- `cdk deploy --all --require-approval never --outputs-file outputs.json` (or a subset by stack name). **If a fix is needed mid-deploy: never edit source while a deploy is still running/rolling back** — the running deploy keeps using the image built from the old source; wait for it to finish (or cancel), then start a fresh deploy. A container dying in seconds (exitCode N) → check **CloudWatch Logs first**, not ECS events.

- Run developer onboarding **automatically — do not ask the operator to assemble env vars or run scripts by hand**. Execute `scripts/setup-developer.sh` yourself (**Windows operator machine: `scripts\setup-developer.ps1` or `python scripts\gateway_auth.py setup` — the `.sh` needs bash/WSL**); both are thin wrappers over `gateway_auth.py setup`, the single cross-platform implementation. It is zero-touch: derives the gateway URL (scheme+host from the `GatewayUrl` output — the ALB domain; CloudFront is removed), Token Service URL, auth mode, and (org-sso) SSO profile values from `outputs.json`. For `cognito-native`, the same `setup` writes the `llmgw-login` config (Cognito Hosted UI domain, app client id, issuer, token endpoint). For `http`, the base URL is `http://<alb-dns>` — reachable only from the `albIngressCidrs` allowlist; no cert trust or tunnel setup exists.

- **SSO provisioning (`org-sso` only)**: per the Phase 1 decision, **create** the permission set(s) — name with **no underscore**, prefer a new uniquely-named one (do NOT reuse a pre-existing permission set just because the name matches) — with an `execute-api:Invoke`-only inline policy whose `Resource` is `arn:aws:execute-api:<config.awsRegion>:<account>:<tokenServiceApiId>/*` (region + API id MUST match the deployed Token Service, else every SSO call 403s), assign to the account (the user-specified **group**(s) or users), `provision-permission-set`, and hand off password activation (IdC console only). Follow `shared/reference/sso-setup.md`.

- **`cognito-native` provisioning (account instance / no-IdC only)**: do **not** create permission sets or any IdC/Identity Store resource. The AuthStack already creates the Cognito User Pool, Hosted UI domain, app client (Authorization Code + PKCE, loopback redirect), and the User Pool Groups that are the teams — but **zero users**. The agent then **creates the initial user(s) from the Phase 1 answer itself**: `aws cognito-idp admin-create-user --user-pool-id <pool> --username <email> --user-attributes Name=email,Value=<email> --desired-delivery-mediums EMAIL` + `aws cognito-idp admin-add-user-to-group --group-name <llmgw-team>` (optionally `admin-set-user-password --permanent` if the operator supplies a password for immediate activation instead of the email invite). Then run the **full-path verification** with that user: `llmgw-login` → access token → Token Service → virtual key → `/v1/models` — a master-key test alone proves nothing about this path. Ongoing on/offboarding stays Cognito console/CLI only. Follow `shared/reference/account-instance-setup.md`.

- **Usage-dashboard check**: after the verification calls, confirm the LiteLLM log group has a `"llmgw": "usage"` EMF line and `aws cloudwatch list-metrics --namespace <METRICS.NAMESPACE>` shows `TotalTokens` within ~2 min; then open the `DashboardUrl` output — the token/user widgets should populate. Log line present but no metric = malformed EMF (multi-line record or empty dimension value); see `litellm-gateway.md` §4. **Before** any of the above verification calls, confirm the ECS service actually reached steady state — `cdk deploy` returning success only means the CloudFormation stack update completed, not that the container is running: `aws ecs describe-services --cluster <cluster> --services <service> --query 'services[0].deployments[0].{running:runningCount,desired:desiredCount,rolloutState:rolloutState}'` should show `rolloutState: COMPLETED` with `running == desired`. If it is stuck `IN_PROGRESS` or rolled back, check CloudWatch Logs for `exec format error` first (Docker build/task architecture mismatch — see constraints.md) before assuming an application bug.

### Phase 6: Developer Onboarding — ALWAYS end by generating the two HTML docs

After a successful deploy you **MUST** generate **two self-contained HTML docs** with `scripts/gen-onboarding.py` (from `cdk deploy --outputs-file outputs.json`), per `shared/patterns/developer-onboarding.md §6`. This **replaces the old inline markdown guide**. The base URL is the **`GatewayUrl` output (the ALB domain — CloudFront is removed)**.

- **`developer-setup.html`** — hand to every developer. **No admin secrets.** **Script-first: the doc instructs developers to RUN the generated `.sh`/`.ps1` launchers — never to hand-edit config files** (the merged JSON/TOML content appears only as a reference appendix). Filled with deployed values:

- **Step 1 — one-shot setup**: run `scripts/setup-developer.sh` (Windows: `scripts/setup-developer.ps1`). It merges `~/.claude/settings.json` (`ANTHROPIC_BASE_URL=<gateway-url>`, `AWS_REGION=<deploy-region>`, **all four model aliases incl. `ANTHROPIC_DEFAULT_FABLE_MODEL`** — omitting Fable hides that tier from `/model` —, `apiKeyHelper` → the token helper, `permissions.deny: ["WebSearch"]` — use the AgentCore Web Search MCP instead of the unsupported built-in) and `~/.codex/config.toml` (`[model_providers.llm-gateway]`: `base_url=<gateway-url>/v1`, `wire_api=responses`, `model=<gpt-alias>`, `web_search="disabled"`, `.auth.command` → the token helper), **backing both up to `*.llmgw-backup-<timestamp>` first (merge, never overwrite)**.

- **Step 2 — login**: `org-sso` → `aws sso login --profile llm-gateway` (Token Service accepts only `AWSReservedSSO_` principals; a non-SSO caller is rejected 403 **by design**). `cognito-native` → `llmgw-login` / `llmgw-login.ps1` (Cognito Hosted UI; `aws sso login` is NOT used).

- **`http` only**: the base URL is `http://<alb-dns>` (plaintext — reachable only from the SG `albIngressCidrs` allowlist; no cert trust, no tunnel).

- **Step 3 — web search MCP** (client-side, required): `claude mcp add-json` at `<gateway-url>/mcp/` with **`headersHelper`** → `gateway_auth.py mcp-headers` (rotating virtual key injected dynamically). Tool: `websearch-web-search-tool___WebSearch`.

- **Step 4 — verify**: run `scripts/healthcheck.sh` / `.ps1` (token helper → `GET /v1/models`; expect the aliases incl. GPT-5.x — plus `GET /v1/mcp/tools` for the websearch tool). Region is never hardcoded (the helper derives the SigV4 region from the Token Service URL host / Cognito issuer); note the **`cognito-native` access-token gotcha** (sending the `id_token` → 401; must send the access token) and **Windows PowerShell launchers** — as detailed in `developer-onboarding.md`.

- **`admin-onboarding.html`** — **operator ONLY** (written `0600`; embeds the LiteLLM master key + Langfuse admin password). Add it to `.gitignore`; never share/commit. Contains: deployed endpoints + identity, **secret values**, **password-change** (master key → Secrets Manager `put-secret-value` + `ecs update-service --force-new-deployment`; Langfuse via its UI; Cognito `admin-set-user-password`), **developer on/offboarding** (`admin-create-user` + `admin-add-user-to-group`; `admin-disable-user`/`admin-delete-user`/`admin-remove-user-from-group` + **revoke the LiteLLM virtual key** + the DynamoDB key-cache **TTL** caveat), and a quick admin test (master key as bearer vs `<gateway-url>/v1`).

- **Hand the operator `shared/reference/litellm-admin-guide.md`** for ongoing ops (LiteLLM Admin UI `/ui/` login with the master key, team/budget management, request logs/traces via the LiteLLM UI · Langfuse [acm only] · CloudWatch). This is what the operator needs *after* the first developer is onboarded, distinct from the one-time deploy steps above.

## Hard Constraints

1. **The ALB is the edge — TLS via `certMode`, CloudFront removed.** The ALB is **always internet-facing and always SG CIDR-restricted** (`litellm.albIngressCidrs`). `acm` needs a domain (regional public ACM cert — an existing ARN, or Route53 DNS-issued) → HTTPS:443. `http` needs no domain → HTTP:80, ⛔ plaintext on the wire, PoC-only (GATE-1 acknowledgement). There is no `self-signed` mode, no CloudFront, no internal/VPN exposure variant, no SSM tunnel. (See `constraints.md`.)

2. **Bootstrap collisions** → bootstrap with a custom `--qualifier` + `@aws-cdk/core:bootstrapQualifier` in `cdk.json`; delete any empty `REVIEW_IN_PROGRESS` `CDKToolkit` stack. Bootstrap **both** us-east-1 and the gateway region. Never delete other apps' bootstrap resources.

3. **Bedrock Guardrails are bedrock-runtime only** — never attach them to `bedrock_mantle/` (GPT) models; cover Mantle with LiteLLM `hide-secrets` and document the gap.

4. **Never hard-code secrets.** Master key, DB creds, Langfuse admin password + project keys all go through Secrets Manager. Shared LiteLLM↔Langfuse trace keys must live in a stack created *before* LiteLLM and be injected with `ecs.Secret.fromSecretsManager` — **never as plaintext `environment` values**.

5. **The public ALB is the edge** (internet-facing in both modes: `acm` HTTPS:443, `http` HTTP:80) — access control is the `albIngressCidrs` SG allowlist (**no AWS WAF**; mention WAF only as an optional PROD hardening note for a fully open `acm` deployment). Always keep a **separate internal ALB (:4000)** for the Token Service (its SSM URL is unchanged, no hairpin). Langfuse (acm only) has its own public ALB. Never expose the internal ALB or the `:4000` listener to the internet. ⚠️ **Pass `open: false` on EVERY `addListener()`** — CDK's default `open: true` silently appends a `0.0.0.0/0` ingress rule for the listener port to the ALB SG, defeating `albIngressCidrs` (real-deploy incident; synth-time invisible). After any deploy touching listeners, verify the deployed SG (`aws ec2 describe-security-groups`) contains only the intended CIDRs. See `constraints.md`.

6. **Mixed model auth — Claude is tokenless SigV4, Mantle (GPT-5.x) is Bearer-token.** Claude (`bedrock/`) authenticates via the ECS Task Role (SigV4) — nothing to rotate. Mantle (`bedrock_mantle/`) has **no SigV4 code path** on the Responses route (verified by extracting the actual installed source from the pinned image); its `validate_environment()` requires a Bearer token. The image therefore installs `aws-bedrock-token-generator` and a `mantle_token_refresh` LiteLLM callback mints a **short-term Bedrock API key from the Task Role's own credentials** and keeps it fresh in-process (no long-term IAM user, no static secret, no external scheduler). ⚠️ **The token MUST be written to `BEDROCK_MANTLE_API_KEY`, never `AWS_BEARER_TOKEN_BEDROCK`.** The latter is a boto3-reserved name: set it and *every* `bedrock-runtime` client in the process (including Claude's SigV4 calls) switches to Bearer auth and Claude breaks with 403 (this exact mistake caused a 4-model outage). Before shipping any change to Mantle auth, run a 3-step regression: Claude call → GPT call (triggers callback) → Claude call again. See `shared/patterns/litellm-gateway.md`.

7. **Production posture** — `removalPolicy: RETAIN` + backups, per-AZ NAT, scoped IAM (no `*`/`bedrock-mantle:*`), access/flow logs. Dev sample uses the opposite; tag each with `PROD TODO`.

8. **UIs must not redirect to a dead host — and NEVER pass `--forwarded-allow-ips`** (LiteLLM Admin UI + Langfuse). The pinned LiteLLM image's CLI **does not have** `--forwarded-allow-ips` (nor a `FORWARDED_ALLOW_IPS` env fallback — verified against the actual image); passing it kills the container at boot (`No such option`, exitCode 2 → circuit-breaker rollback; a real-deploy incident). Redirect correctness comes from `PROXY_BASE_URL`: for `acm`, set it to the gateway URL (`https://<custom-domain>`, known at synth); for `http`, leave it empty (the ALB DNS is not known when the container definition is synthesized) — the `/ui`→`/ui/` 307 scheme quirk is cosmetic only. `NEXTAUTH_URL` for Langfuse = its acm domain. Verify any doc-sourced CLI flag against the pinned image (`docker run --rm --entrypoint litellm <image> --help`) before emitting the entrypoint. See `constraints.md`.

9. **Auth mode is explicit — `org-sso` or `cognito-native`.** Default `authMode="org-sso"` preserves the organization-instance path: Token Service accepts only `AWSReservedSSO_` principals, and provisioning creates a no-underscore permission set with `execute-api:Invoke` on the Token Service API. For an IAM Identity Center **account instance** or any account without usable org-sso, use `authMode="cognito-native"`: an Amazon Cognito User Pool is the sole identity source (no external IdP, no IdC federation, no Identity Store), the Token Service sits behind a Cognito User Pools authorizer, and it reads the verified `cognito:groups` claim to resolve the team. ⚠️ Do **not** attempt IdC↔Cognito SAML federation on an account instance — account instances cannot host a SAML customer-managed application (AWS-confirmed), so that path is impossible. (`account-sso` remains in the schema only as a deprecated no-op for old config snapshots; do not generate it.) See `shared/reference/sso-setup.md` and `shared/reference/account-instance-setup.md`.

10. **Region is config-driven** — `config.awsRegion` is authoritative (`bin/app.ts`: `config.awsRegion ?? CDK_DEFAULT_REGION ?? AWS_REGION`); never require editing `constants.ts`. AgentCoreGateway and MantleNetwork are pinned to us-east-1 (bootstrap us-east-1 **and** the gateway region). **Long completions are governed by the ALB `idleTimeout` (`config.litellm.albIdleTimeoutSeconds`, default 900s, max 4000s) — the old CloudFront hard 120s VPC-Origin ceiling that 504'd Opus/Fable extended thinking (with no LiteLLM access-log line) is gone.** ACM certs for the ALB are regional (`config.awsRegion`), not the former us-east-1 CloudFront cert.

11. **Web search = AgentCore Web Search Tool** — provision `AWS::BedrockAgentCore::Gateway` (MCP, `AWS_IAM` inbound) + `GatewayTarget` (`connectorId: web-search`, `GATEWAY_IAM_ROLE`) in us-east-1; LiteLLM calls it with SigV4 (`bedrock-agentcore:InvokeGateway`). No Tavily, no 3rd-party API key. See `shared/patterns/agentcore-websearch.md`.

12. **Mantle (GPT-5.x) = us-east-1 via cross-region VPC peering** — `MantleNetworkStack` (peer VPC + `bedrock-mantle` endpoint + peering + acceptance custom resource + cross-region PHZ) + `MantlePeeringRoutesStack` (primary-side routes); `MANTLE_REGION=us-east-1`; Task Role needs `aws-marketplace:Subscribe` (first-call auto-subscribe). `mantle.peerVpcCidr` must not overlap `network.vpcCidr`. See `shared/patterns/mantle-peering.md`.

13. **Authorization unit = LiteLLM team_alias, unbranched.** In `org-sso`, the permission set name maps 1:1 to a same-named LiteLLM team. In `cognito-native`, the Cognito User Pool Group name maps 1:1 to a same-named LiteLLM team (filtered by `teamGroupPrefix`; `require-single-team-group` rejects a caller with zero or ambiguous matching groups). Never hard-code an `if team in {...}` org/tier list in `handler.py`. Onboarding a new team after deploy must be achievable via a permission set / Cognito group (+ membership) + LiteLLM Admin UI team settings, with **no Lambda code change or redeploy**. `TIER_CONFIG` may seed a team's *first-ever* creation with initial `models`/`max_budget` from Discovery answers, but is never consulted again once the team exists. See `shared/patterns/lambda-handlers.md`.

14. **Verify model IDs, never assume a `us.` prefix.** Resolve each Claude model's actual inference-profile ID with `aws bedrock list-inference-profiles` at Design time. Recent (2026) models (Opus 4.8, Sonnet 5, Haiku 4.5, Fable 5) are published only as `global.` GLOBAL profiles — `bedrock/global.anthropic.<model-id>`; a `us.` call returns `The provided model identifier is invalid.` IAM must grant the GLOBAL profile ARN **plus** the unqualified `arn:aws:bedrock:::foundation-model/<model>` fan-out target (and the gateway-region foundation-model ARN), not `us-east-1`/`us-west-2` regional ARNs.

15. **Fable/Mythos-class models need `provider_data_share` opt-in.** These models are restricted to `allowed_modes: ["provider_data_share"]`; the account data-retention mode must be set to `provider_data_share` **per region** (Bedrock control-plane REST API, no console UI) or calls are blocked. This permits 30-day Anthropic retention + human review, so it is a **GATE-1 approval item**, never assumed. See `shared/reference/constraints.md`.

16. **Do not plant a boto3-reserved env var name.** Before setting any process-global env var that an SDK might special-case, check it is not reserved. Specifically, the Mantle token goes in `BEDROCK_MANTLE_API_KEY`, never `AWS_BEARER_TOKEN_BEDROCK` (boto3 intercepts the latter for *all* bedrock-runtime clients and breaks Claude's SigV4). In a shared-process gateway, always regression-test the providers you did **not** change. See Hard Constraint #6.

17. **`http` certMode = plaintext, PoC-only, SG-locked.** The virtual key **and prompt/response bodies** travel unencrypted from client to ALB. The deploy MUST restrict the public ALB SG to known source CIDRs (`litellm.albIngressCidrs` — a required Discovery answer, not a synth-time guess); `0.0.0.0/0` means the plaintext endpoint is reachable from the whole internet and requires its own explicit GATE-1 acknowledgement. Prefer `acm` whenever a domain is available or the traffic is sensitive.

18. **Langfuse UI requires `certMode='acm'`.** `enableLangfuse=true` is valid only with a domain (public ALB + ACM); `http` deploys are CloudWatch-only (schema fail-fast). When acm, whether to deploy Langfuse is a free `enableLangfuse` choice exactly as before.

19. **GPT models are `gpt-5.5`/`gpt-5.4` only — never `gpt-5.6-*`.** `constants.ts` `MODELS` defines `GPT55`/`GPT54` and never a `GPT56_*` entry; `config.yaml` routes only those two. `gpt-5.6-*` is not on OpenAI's certified Codex↔Bedrock model list and broke Codex tool-use in a real deploy (`namespace` tool type → Mantle `400 validation_error`; the client shows misleading "high demand" reconnects). If a user insists on `gpt-5.6-*`, warn about this failure mode explicitly and get confirmation before deploying it. See `shared/reference/constraints.md`.

## Generation rules

- Single source of truth in `lib/config/constants.ts`; runtime-validate `config/dev.json` (`lib/config/schema.ts`, incl. `awsRegion`/`authMode`/`sso`/`cognitoNative`/`agentcore`/`mantle`) so deploy fails fast.

- Append-only cross-stack `*Exports` interfaces; runtime-only wiring via SSM by name; cross-region wiring via `crossRegionReferences: true`.

- **SG ownership rule (cyclic-reference, real-deploy lesson)**: NetworkStack owns only shared-infra SGs (Aurora/endpoints/Lambda) with **CIDR-based ingress**; app stacks (LiteLLM/Langfuse) own their service + ALB SGs. Never let a Network-owned SG hold a rule referencing an app-stack SG — and remember `addTargets()` auto-wires SG rules. See `constraints.md` + `cdk-stacks.md` §1.

- **Onboarding scripts merge, never overwrite**, the user's shared config files (`~/.claude/settings.json`, `~/.codex/config.toml`): backup + JSON load-update-save / TOML block-replace only (`constraints.md`). The merge logic exists **once**, in `gateway_auth.py setup` — `.sh`/`.ps1` are thin launchers; never re-implement it in shell/PowerShell. The merge also **removes** direct-Bedrock remnants (`CLAUDE_CODE_USE_BEDROCK`, `AWS_BEARER_TOKEN_BEDROCK`, top-level `model`) — leaving them causes 403/"team not allowed to access model" (real Windows incidents, `constraints.md`). Keep `:?` error messages apostrophe-free (macOS bash 3.2).

- ARM64/Graviton for all Fargate/Lambda.

- Every cdk-nag suppression carries a written reason.

## When to call MCP

| Trigger | MCP call |

|---|---|

| Choosing/confirming model IDs (Claude + GPT-5.x) | `aws___search_documentation`, `aws___get_regional_availability` |

| Resolving a Claude model's inference-profile ID (`global.` vs `us.`) | (CLI) `aws bedrock list-inference-profiles` in the target region |

| Confirming Fable/Mythos data-retention (`provider_data_share`) requirement | `aws___search_documentation` (data-retention + model card) |

| Confirming AgentCore Web Search is in us-east-1 / CFN resources exist | `aws___get_regional_availability` (`AWS::BedrockAgentCore::Gateway`, `...::GatewayTarget`) |

| Confirming `bedrock-mantle` PrivateLink + GPT-5.x in us-east-1 | `aws___get_regional_availability`; CLI `ec2 describe-vpc-endpoint-services` |

| Confirming a service/feature is in the target region | `aws___get_regional_availability` |

| Aurora engine version validity | (CLI) `rds describe-db-engine-versions` in the target region |