--- name: zitadel-idp metadata: version: 0.14.0 description: "Self-hosted Zitadel v4 OIDC field guide — a numbered quirk catalogue, working docker-compose and an idempotent TypeScript bootstrap. High-friction traps: FirstInstance env placement, JWT/JWKS over self-signed HTTPS, TLS-terminating proxy (`--tlsMode external`) and the console's mixed-content `Failed to fetch`, v2.66→v4 upgrade, API v1→v2, silent-renew, 401 storms. Triggers — zitadel, oidc self-hosted, silent-renew, JWKS, masterkey, v2.66→v4, tlsMode external, pre-cutover check." --- # Zitadel IdP — Field Guide This skill captures patterns and pitfalls discovered while integrating Zitadel `v4.x` self-hosted as the IdP for the JRC Brasil ERP. It exists to prevent every new project from re-discovering the same gotchas the hard way. **Scope**: Zitadel `v4.x` (latest as of 2026-04) is the default target. v3 differs in a few places (notably no `OrganizationService.AddOrganization` v2 endpoint) — the references flag breaking changes when relevant. **v2.66.x** is also covered for one specific edge case (Quirk 24, masterkey via flag) because legacy stacks still run it; the rest of the skill remains v4-first. **Out of scope**: Zitadel Cloud (managed), v3 setup, SAML flows, federation IdPs (login with Google/GitHub), Login UI v2 customization. The skill assumes self-hosted, OIDC-only, with the bundled Login UI v1. ## When to use this skill Read this skill before you start any of the following: - Drafting a `docker-compose.yml` that includes Zitadel - Writing a script that programmatically creates Orgs, Projects, Roles, Apps via Management API - Implementing JWT validation for Zitadel access tokens in any backend (Node/Go/Java/Python) - Wiring an SPA or mobile app to Zitadel via Auth Code + PKCE - Renaming or retiring a project role (the key is immutable — Quirk 48), or reconciling grants after the declarative config gains a role - Diagnosing 401 / 403 / 404 / 400 against Zitadel — including: `Instance not found`, `User with state initial`, `Method Not Allowed`, `Organisation doesn't exist`, `Errors.Instance.Domain.AlreadyExists`, `404 {"code":5,"message":"Not Found"}` na hosted UI, 400 em `/oauth/v2/authorize?prompt=none`, 401 com JWT cujos `iss`/`aud`/`exp` parecem corretos (especialmente atrás de proxy TLS local com cert self-signed), ou 400 `COMMAND-1m88i "No changes"` em bootstrap idempotente - Wiring an SPA hosted **fora de `localhost`/`127.0.0.1`** (acesso via LAN, IP `.sslip.io`, hostname custom) — exige HTTPS por causa do `crypto.subtle` (PKCE), e o consumer típico de `react-oidc-context` engole a rejeição do `signinRedirect` If you are merely calling a pre-existing Zitadel deployment from your code, you probably only need `references/token-validation.md` and `references/api-cheatsheet.md`. ## How to use the bundled material Drill into the relevant reference based on the immediate task: | Task | Read | |------|------| | Bring up Zitadel locally for the first time | `references/docker-compose-bootstrap.md` + use `assets/docker-compose.zitadel.yml` as starting point | | Reset a broken local Zitadel | `scripts/reset-zitadel.sh` | | Programmatically create Org/Project/Roles/App | `references/api-cheatsheet.md` + `assets/bootstrap-zitadel.ts` | | Validate Zitadel JWT in Node | `references/token-validation.md` | | Map domain `tenantId` to Zitadel `orgId` | `references/tenant-org-mapping.md` | | Wire an SPA (React + `react-oidc-context`) — boot-time silent renew, F5 retention, logout flow | `references/spa-recipes.md` | | Apply org branding to the hosted Login UI v1 (logo, colors, custom texts) | `references/branding.md` | | Plan a v2.66.x → v4.x upgrade (pre-flight, snapshot, login UI v2 container, validation, rollback) | `references/migration-v2-to-v4.md` | | Refactor callers from Management API v1 to v2 (Connect protocol, payload diffs, idempotence patterns) | `references/api-v1-to-v2-mapping.md` | | Run prod + staging Zitadels side by side, or move users between instances | `references/multi-instance-and-user-migration.md` | | Run a role rename/migration across environments (declare-first, the grant tool, per-environment state) | `references/role-migration.md` | | Hit a confusing error and want a quick lookup | `references/troubleshooting.md` | | Prove a bootstrap actually wrote what the YAML declares (no PAT needed) | `references/troubleshooting.md` §"The bootstrap logs `reuse` for everything" | | One account loops back to the login screen while others log in fine | `references/troubleshooting.md` §"One account loops back…" | The references are designed to be readable in isolation — open the one you need without slogging through the rest. ## The quirks (the headline list) These are the issues that consistently bite first-time Zitadel integrators. Each has a dedicated section in the references; this list is the trigger map so you know which file to open. 1. **`ZITADEL_FIRSTINSTANCE_*` env vars must live on the `zitadel` service, not on `zitadel-init`** — the init container only runs schema migrations; setup (which honors FirstInstance) runs from `start-from-init` on the main service. → `docker-compose-bootstrap.md §1`. 2. **The `/current-dir` volume must be writable by uid 1000** — Zitadel runs as a non-root user. A previous root-owned init leaves the volume unwritable and setup silently fails with `permission denied` in a restart loop. → `docker-compose-bootstrap.md §2`. 3. **`ZITADEL_EXTERNALDOMAIN` is enforced on every request via the `Host` header** — calling `http://127.0.0.1:8080` (the bind port) when external domain is `127.0.0.1.sslip.io` returns `Instance not found`. Node `fetch` does not let you override the `Host` header at runtime — use the URL with the external domain literally. → `docker-compose-bootstrap.md §3`. 4. **`/admin/v1/orgs/_setup` requires a human admin user; use `/zitadel.org.v2.OrganizationService/AddOrganization` to create an org without one** — v2 endpoints accept JSON over Connect protocol. → `api-cheatsheet.md §"Create org"`. 5. **`AddHumanUserRequest` requires `userName` and `profile.firstName`/`lastName` (not `givenName`/`familyName`)** — payload shape changed across versions; v4 also rejects `clockSkew > 5s` on OIDC apps. → `api-cheatsheet.md §"Create human user"` + `§"Create OIDC app"`. 6. **Users created without `initialPassword` enter state `initial` and cannot be `_deactivate`d** — Zitadel rejects with `COMMAND-ke0fw`. In production the user exits this state by completing invite; in tests/seeds set an `initialPassword`. → `troubleshooting.md`. 7. **Domain `tenantId` (stable string like `JRC`) ≠ Zitadel `orgId` (numeric like `370503937624637443`)** — every Management API call needs the numeric `orgId` in `x-zitadel-orgid`. Build a translation layer in your adapter. → `tenant-org-mapping.md`. 8. **User grants search is at `/management/v1/users/grants/_search` (global, filter by `userIdQuery`), not `/management/v1/users/{id}/grants/_search`** — the latter returns `405 Method Not Allowed`. assignRole / revokeRole should be implemented as search-then-PUT/DELETE for idempotency, not POST a fresh grant each time. → `api-cheatsheet.md §"User grants"`. 9. **`loginV2.required=true` is the default instance feature flag in Zitadel ≥ v3, but Login UI v2 is a separate Next.js app you must deploy yourself** — without it, every `/oauth/v2/authorize` redirects to `/ui/v2/login/login` and returns `{"code":5,"message":"Not Found"}`. App-level `loginVersion: {loginV1: {}}` alone does NOT override the instance flag — you must `PUT /v2/features/instance {loginV2: {required: false}}` or deploy v2. → `troubleshooting.md`. 10. **Silent-renew redirect URI must be in the OIDC app's `redirectUris`** — most bootstrap scripts only register `/auth/callback`. Without `/silent-renew`, every `prompt=none` request returns `400` and the SPA loops forever in "verifying session…". Add it at bootstrap time, not after the first failure. → `troubleshooting.md` + `api-cheatsheet.md §"Create OIDC application"`. 11. **JWT access tokens DO NOT carry profile claims** (`name`, `preferred_username`, `email`, `urn:zitadel:iam:org:id`) — those live in the id_token and `/oidc/v1/userinfo` only. Backend mappers that hard-require them silently 401 every request with "Token inválido" even when `iss`/`aud`/`exp` are perfect. Always fall back to `sub` for operatorName and `defaultTenant` for tenantId. → `token-validation.md §"Access token vs ID token"`. 12. **Node backends fetching JWKS over HTTPS with a self-signed cert (mkcert/dev CA) need `NODE_EXTRA_CA_CERTS`** — `createRemoteJWKSet(new URL(jwksUrl))` does a normal Node `fetch`, which uses the OS trust store. With a local CA, the TLS handshake fails before signature check; jose surfaces it as a generic verification error. **Symptom**: 100% of `/api` requests return 401, JWT decoded by hand looks perfect, SPA falls into a silent-renew loop and rate-limit (429) follows. → `token-validation.md §"Trusting a self-signed JWKS endpoint"` + `troubleshooting.md §"401 storm with apparently-valid JWT"`. 13. **Zitadel volume reset (`down -v` / `--reset-zitadel`) regenerates `projectId` and `clientId`** — any backend env (`AUTH_AUDIENCE`/`OIDC_AUDIENCE`/`VITE_OIDC_CLIENT_ID`) cached from a previous bootstrap goes stale silently. Same 401-storm symptom as quirk 12, no clear error. Fix: re-derive from `bootstrap.json` on every boot, never hardcode. → `api-cheatsheet.md §"Re-reading bootstrap output after volume reset"` + `troubleshooting.md §"401 storm with apparently-valid JWT"`. 14. **`PUT /management/v1/projects/{p}/apps/{a}/oidc_config` returns `400 COMMAND-1m88i "No changes"` when the body matches current state** — idempotent bootstrap scripts that always PUT the OIDC config crash on second run unless they catch this code. → `api-cheatsheet.md §"Create OIDC application"` + `assets/bootstrap-zitadel.ts` (idempotent template). 15. **Running Zitadel behind a TLS-terminating reverse proxy (Caddy/NGINX/Traefik) requires THREE settings, not two** — `ZITADEL_EXTERNALSECURE=true` + `ZITADEL_TLS_ENABLED=false` + the start flag `--tlsMode external`. Without `--tlsMode external` the binary still tries to bind a TLS listener on its internal port and refuses traffic. → `docker-compose-bootstrap.md §"TLS terminated by reverse proxy"`. 16. **Browsers expose `crypto.subtle` (used by PKCE in oidc-client-ts) only in secure contexts — `localhost`/`127.0.0.1` are the only HTTP exceptions** — accessing the SPA via LAN IP / `.sslip.io` / custom hostname over HTTP makes `signinRedirect()` throw. `react-oidc-context` callers typically `void` the promise, so the failure is silent — clicking "Entrar" produces no console error, no navigation, no network request. Fix: serve everything via HTTPS even for dev/LAN testing (mkcert + reverse proxy). → `troubleshooting.md §"Entrar / Login button does nothing"`. 17. **F5 with `InMemoryWebStorage` requires a boot-time `signinSilent` — and `` wrapping the silent-renew route makes that recursive** — `automaticSilentRenew` only fires on `accessTokenExpiring`, which needs an existing `User`; in-memory storage has no `User` after F5, so the lib never recovers the session even with the IdP cookie alive. The fix is an active `auth.signinSilent()` at boot — but if your provider mounts above the `` tree, the iframe loads `/silent-renew`, re-mounts the provider, fires another `signinSilent`, and the parent Promise never settles. **Symptom**: SPA stuck on "Verifying session…" indefinitely with no IdP-side error. Fix requires three guards (route check, iframe check, ref-gated useEffect) and a watchdog `setTimeout` — and crucially, no closure-scoped `cancelled` flag, because StrictMode flips it before the Promise resolves. → `spa-recipes.md §"Recipe 1"` + `troubleshooting.md §"SPA stuck on Verifying session…"`. 18. **`post_logout_redirect_uri` is byte-matched just like `redirect_uri`** — registering `${WEB_BASE}/` while the SPA sends `${WEB_BASE}/login` (a common drift introduced when the SPA's `VITE_OIDC_POST_LOGOUT_REDIRECT_URI` is set to `/login` for UX but the bootstrap registered just the root) makes Zitadel respond with `{"error":"invalid_request","error_description":"post_logout_redirect_uri invalid"}` and the user is stranded. Trailing slash, scheme, and port all matter. Fix: register both URIs the SPA might send, comma-joined: `${WEB_BASE}/login,${WEB_BASE}/`. → `troubleshooting.md §"Logout returns post_logout_redirect_uri invalid"`. 19. **Branding aplicado na org não pinta a Login UI sem `privateLabelingSetting` no projeto** — `POST /management/v1/policies/label` + `_activate` na org JRC retorna 200, GET subsequente confirma `primaryColor: "#ed1c24"`, mas browser em `/ui/login/login` continua azul Zitadel default `#5469d4`. Causa: Zitadel só usa a label policy da org dona do projeto se este tiver `PRIVATE_LABELING_SETTING_ENFORCE_PROJECT_RESOURCE_OWNER_POLICY`. O default `_UNSPECIFIED` cai pra label policy da instância. Fix: setar o flag no payload do `POST/PUT /management/v1/projects/{p}` e checar via GET. → `branding.md §"Quirk 19"`. 20. **POST vs PUT na primeira label policy + dois error IDs distintos pra "no-op"** — `PUT /management/v1/policies/label` quando `policy.isDefault === true` retorna `404 "Private Label Policy not found (Org-0K9dq)"` (precisa POST primeiro pra criar override). Re-runs do bootstrap idempotente disparam `400 "Private Label Policy has not been changed (Org-8nfSr)"` — diferente do `COMMAND-1m88i` genérico (quirk 14). Bootstrap deve: GET prévio → ramificar por `isDefault` → tratar **ambos** error IDs como no-op. → `branding.md §"Quirk 20"`. 21. **Path de assets em Zitadel v4 mudou para `/assets/v1/org/policy/label/...`** — singular `org`, sem `s/me`. O path `/assets/v1/orgs/me/policy/label/logo` que aparece em docs/exemplos antigos do Zitadel v1/v2/v3 retorna **HTTP 405 Method Not Allowed** em v4. Confunde porque parece "endpoint existe mas método errado" — na verdade existe em outro path. Endpoints corretos: `logo`, `logo/dark`, `icon`, `font`. Header `x-zitadel-orgid` continua obrigatório. Multipart com field name `file`. → `branding.md §"Quirk 21"`. 22. **Custom login text usa `/management/v1/text/login/{lang}`, e Zitadel só aceita códigos curtos** — não `/policies/custom_login_text/{lang}` (path antigo, retorna 404). E só aceita `pt`, `en`, `de` etc. — `pt-BR` retorna `400 LANG-lg4DP "Language is not supported"`. Resolução de Accept-Language `pt-BR → pt` é feita server-side. PUT é mergeable: campos não enviados preservam i18n default; pra zerar um campo mande string vazia explícita. → `branding.md §"Quirk 22"`. 23. **Refatoração single-app → multi-app YAML: env vars do `dev.sh` precisam DOMINAR o YAML** — quando você evolui o bootstrap pra ler `applications[].redirectUris` declarativos do YAML, o YAML naturalmente lista hosts canônicos (`localhost:5173`, `app.example.com`). Mas dev workflows com LAN HTTPS (mkcert + proxy reverso, host dinâmico via IP `.sslip.io`) populam `OIDC_REDIRECT_URIS=https://.sslip.io:5443/auth/callback,...` — host que muda a cada IP, **nunca está no YAML**. Se YAML wins, dev login bate em `{"error":"invalid_request","error_description":"The requested redirect_uri is missing in the client configuration"}` no callback do Zitadel. Família dos quirks 10 (silent-renew em redirectUris) e 18 (byte-match), mas o gatilho é diferente: **regressão de refatoração**, não config inicial. Fix canônico: precedência **env > YAML > hardcoded fallback** — env do `dev.sh` (quando setada explicitamente) domina; YAML é fallback de produção (onde env unset, hosts são canônicos). → `troubleshooting.md §"redirect_uri missing in client configuration after multi-app refactor"`. 24. **Zitadel v2.66.x `start-from-init`: env-var `ZITADEL_MASTERKEY` não é lida com confiabilidade — passe via flag CLI** — em v2.66.10 (e potencialmente outras patches da linha 2.66) o subcomando `start-from-init` falha com `panic: No master key provided ... masterkey must either be provided by file path, value or environment variable` mesmo quando `ZITADEL_MASTERKEY` está injetada corretamente no container (validável via `docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}'` — 32 chars exatos, sem CR, sem leading whitespace, sem null bytes). Em v4.x o fallback `os.Getenv` funciona; em v2.66 não. **Sintoma**: container em loop de restart com `RestartCount` crescendo, exit 2, e o panic acima a cada ciclo (~60s, devido ao retry policy). Fix canônico: passar `--masterkey ${ZITADEL_MASTERKEY}` no `command:` do compose (a flag tem precedência sobre env). Trade-off: a flag aparece em `docker inspect` e `ps aux`, então o `.env` precisa ser `chmod 600` e o host dedicado/de confiança. Se isso virar problema, migrar para `--masterkeyFile /run/secrets/zitadel_masterkey` + Docker secret. → `docker-compose-bootstrap.md §"Quirk 24 — masterkey via flag em v2.66.x"`. 25. **Login UI v2 em v4 é um container Next.js separado** — diferente de Login UI v1, que vive embutida no binário do Zitadel em `/ui/login/`, a Login UI v2 (`/ui/v2/login`) é servida pela imagem `ghcr.io/zitadel/zitadel-login`. Em v3+ o instance flag `loginV2.required` default é `true` — o `signinRedirect` da SPA cai em `/ui/v2/login` por padrão, e se você só tem o container `zitadel` no compose recebe `404 {"code":5,"message":"Not Found"}`. Duas saídas: (A) deploy do container `zitadel-login` + reverse proxy roteando `/ui/v2/login` → `zitadel-login:3000` e tudo o mais → `zitadel:8080`; (B) `PUT /v2/features/instance {"loginV2":{"required":false}}` e seguir com Login UI v1 indefinidamente. Escolha uma — oscilar entre as duas sem pensar gera 404 esporádicos. → `migration-v2-to-v4.md §3.2-3.3` + `docker-compose-bootstrap.md §8`. 26. **Idempotência em v2 via IDs determinísticos em vez de search-then-create** — em v1 o padrão idiomático era `POST /resource/_search` filtrando por nome → se 0 hits, `POST /resource` (Zitadel gera o ID). Em v2 você pode passar seu próprio `userId`/`applicationId`/`projectId` no body; tentar criar com ID já existente retorna `ALREADY_EXISTS`, que você trata como sucesso. Resultado: 1 round-trip em vez de 2 por recurso. Vale a pena para bootstraps multi-app (5 apps × 2 round-trips × N boots = ruído mensurável em hot-deploy). Pra recursos com nome humano-legível e sem ID estável (`org "JRC"`, `project "ERP-JRC"`), o padrão search-then-create v1 continua válido em v2 também. → `api-v1-to-v2-mapping.md §5`. 27. **Contextual `orgId` mudou de header para body em v2** — v1 exigia `x-zitadel-orgid: 379...` em todo call org-scoped. v2 coloca o equivalente no body, geralmente como `organizationId` (às vezes nested em `org.id` ou `resourceOwner`). Modo de falha mais comum em refactor: dropar o header e esquecer de adicionar o campo no body — symptom é `INVALID_ARGUMENT: missing organization_id`. O header é **inofensivo em calls v2** (ignorado), então durante a transição você pode manter o header setado globalmente no HTTP client sem quebrar nada. → `api-v1-to-v2-mapping.md §3`. 28. **Login UI v2 auto-provisioning is broken in v4.15.0** — the FirstInstance envs that should auto-create the `IAM_LOGIN_CLIENT` service user + write its PAT to a shared volume don't work in v4.15. Two failure modes, no winner: **(A)** setting only `ZITADEL_FIRSTINSTANCE_LOGINCLIENTPATPATH` is a no-op — the server has no service user to create, so the volume stays empty and the `zitadel-login` container loops on stderr `ZITADEL_SERVICE_USER_TOKEN_FILE=/login-client/login-client.pat is set. Awaiting file and reading token.` indefinitely; `/ui/v2/login` returns 404. **(B)** adding the `ZITADEL_FIRSTINSTANCE_ORG_LOGINCLIENT_MACHINE_USERNAME` / `MACHINE_NAME` / `PAT_EXPIRATIONDATE` envs to actually provision the service user causes setup migration `03_default_instance` to fail with `Errors.Instance.Domain.AlreadyExists` / `unique_constraints_pkey` (instance domain reserved twice — Human admin + LoginClient race). Container enters restart loop and never becomes healthy. Tracked in zitadel/zitadel#8910 and #9293; partial fix in PR #10518 may not be in every v4.x patch. **Pragmatic mitigations**: (1) **Path B of Quirk 25** — `PUT /v2/features/instance {"loginV2":{"required":false}}` and pin OIDC apps with `loginVersion: { loginV1: {} }`. Login UI v1 stays default with branding intact via label policy. The `zitadel-login` container can stay deployed (idle, looping benignly) so promoting it is a single config flip when upstream stabilizes. (2) **Provision in bootstrap** — after Zitadel is healthy, your bootstrap creates the machine user `login-client` + role `IAM_LOGIN_CLIENT` + PAT via the regular UserService/AuthorizationService API, then writes the PAT to a bind-mounted file the `zitadel-login` container reads. More work but independent of upstream. → `troubleshooting.md §"zitadel-login: Awaiting file and reading token (forever)"` + `migration-v2-to-v4.md §3.4`. 29. **OIDC `client_id` is the numeric `clientId` from `oidcConfiguration`, NOT the deterministic `applicationId` UUID you supplied** — even when you pass your own `applicationId` (UUID v7) in `CreateApplicationRequest` for idempotency, Zitadel auto-generates a *separate* numeric `clientId` (e.g. `371898282416275459`) for OAuth/OIDC. The `applicationId` is the resource handle for v2 APIs (`UpdateApplication`, `GetApplication`, `DeleteApplication`); the `clientId` is what `/oauth/v2/authorize?client_id=…` requires. Frontend `VITE_OIDC_CLIENT_ID` MUST be the numeric value. **Symptom**: OIDC authorize returns `400 invalid_request "Errors.App.NotFound"` with the UUID; SPA login fails right after a clean cutover even though the bootstrap log shows `[app] created appId=`. **Fix in CD**: bootstrap output exposes both (`appId=… clientId=…`); a CD step extracts the numeric `clientId` from `oidcConfiguration` of the bootstrap response and overwrites `VITE_OIDC_CLIENT_ID` (image-rebuild required — VITE_* are baked at build time). Backend `AUTH_AUDIENCE` for JWT validation can be the deterministic `projectId` (it appears in the JWT `aud` claim alongside the clientId). → `troubleshooting.md §"OIDC client_id mismatch — Errors.App.NotFound"`. 30. **`ZITADEL_BOOTSTRAP_ENV`-style env that picks dev-vs-prod deterministic IDs from YAML must be set explicitly in CD — silent default to `dev` is a footgun** — when your bootstrap script has `applications[].ids.dev` and `applications[].ids.prod` blocks and selects between them via `process.env.ZITADEL_BOOTSTRAP_ENV ?? 'dev'`, a CD pipeline that doesn't set the env creates prod entities with **dev IDs**. The frontend secret has prod IDs, so the SPA's `client_id` doesn't match anything in the IdP — same `Errors.App.NotFound` symptom as Quirk 29 but with a different root cause and trickier fix (already-created entities have wrong IDs and must be wiped + recreated). **Mitigations**: (1) require the env at script start, throw on undefined rather than defaulting; (2) set `ZITADEL_BOOTSTRAP_ENV: prod` (or equivalent) explicitly on the bootstrap container in `docker-compose.prod.yml`; (3) document the env in the runbook. **Related**: `bootstrap.json` (or equivalent output file) is **only refreshed when the bootstrap actually runs** — after editing the YAML to add a new `applications[]` entry, a downstream consumer who reads `bootstrap.json` to extract `clientId` will see the *previous* boot's snapshot until bootstrap is re-executed (the new app simply won't appear). Always re-run bootstrap before extracting IDs from its output; consider `--dry-run` semantics if extraction must be cheap. **Three-environment corollary**: the `ids.{env}` block generalizes past dev/prod — a `ids.staging` alongside them is the common shape once staging gets its own Zitadel (Quirk 47), and it makes the "silent default to `dev`" footgun worse, because two of the three environments are now wrong instead of one. **And note what the YAML does NOT contain**: only the deterministic `applicationId` is yours to declare. The numeric `clientId` is minted by Zitadel and is **not recoverable from the YAML at all** — it exists only in the bootstrap log and in the output file, so losing both means reading it back via `GetApplication` (needs a PAT) or the Console. Treat the bootstrap output as an artifact worth keeping, not a log line. → `troubleshooting.md §"Wrong-environment IDs in prod IdP"`. 31. **`ZITADEL_DEFAULTINSTANCE_FEATURES_LOGINV2_REQUIRED=false` at FirstInstance time breaks the chicken-and-egg of Quirks 25 + 28 in CD cutovers** — when the upstream Login UI v2 auto-provisioning bug (Quirk 28) blocks `/ui/v2/login`, *and* the operator can't login to the console to create the IAM_OWNER PAT manually because the OIDC redirect to `/ui/v2/login` 404s — the cleanest break is to set the feature flag in **DefaultInstance** config (env on the `zitadel` server) so the instance is born with `loginV2.required=false` from boot zero. The OIDC authorize endpoint then redirects to `/ui/login` (v1, embedded in the binary) immediately, no PAT required. Bootstrap's `PUT /v2/features/instance` call against the same flag becomes a no-op idempotency check. **Why this is better than waiting for bootstrap to flip the flag**: bootstrap needs a PAT, PAT requires console login, console login redirects to broken `/ui/v2/login` → without DefaultInstance pre-config, you're stuck. With it, the loop opens at the right place. → `docker-compose-bootstrap.md §"DefaultInstance feature flags pre-config"`. 32. **nginx-proxy: when 2 containers share a `VIRTUAL_HOST` and one declares `VIRTUAL_PATH` while the other doesn't, the no-`VIRTUAL_PATH` container is silently ignored** — only the more-specific path-route is registered in the generated `nginx.conf`. Every request that doesn't match the prefix returns 404 (e.g., `/.well-known/openid-configuration`, `/oauth/v2/*`, `/ui/console`). Symptom in nginx logs: trailing `"-"` upstream means no route was matched. Fix: declare `VIRTUAL_PATH=/` + `VIRTUAL_DEST=/` on the "default" container (the one serving root) so nginx-proxy registers it as a less-specific location alongside the prefix-routed sibling. Typical trigger: adding the `zitadel-login` container (`VIRTUAL_PATH=/ui/v2/login`) to a host that previously had only `zitadel` (no VIRTUAL_PATH). Applies to any nginx-proxy split (e.g., API + admin UI on same host). → `docker-compose-bootstrap.md §"nginx-proxy: split VIRTUAL_HOST + VIRTUAL_PATH"`. 33. **Console v4 "Add Human User" form auto-truncates `Username` to the email's local-part** — typing the email into the **E-Mail** field auto-fills **Username** with everything before `@` (e.g., `jane.doe@example.com` → `jane.doe`). Combined with `userLoginMustBeDomain=false` (the instance default), the canonical `loginName` becomes just the local-part. The end user then tries to login with their full email and the IdP responds **"O usuário não pôde ser encontrado" / "User could not be found"** — even though the user exists, has the right grants, and email is correct in the contact section. **API creation does NOT trigger this** (`AddHumanUserRequest` honors whatever `username` you pass). Operationally this is the #1 cause of "I created the user but they can't login" reports in self-hosted setups where TI provisions humans via the Console. **Fix at creation time**: after typing the e-mail, *manually* clear the Username field and re-type the full email. **Retroactive fix**: detail page → PROFILE → click `modify` next to Username → enter full email → Save. The loginName is recomputed on save. → `troubleshooting.md §"User created via Console can't login by email"`. 34. **`userLoginMustBeDomain=true` (Settings → Domain settings → "Add Organization Domain as suffix to loginnames") stamps loginNames irreversibly on existing users** — flipping the org-level checkbox rewrites every existing user's `loginName` to `userName + "@" + orgPrimaryDomain`. **Disabling and clicking "Reset to Instance default" does NOT undo the stamp** — affected users keep the suffix indefinitely. Doubly hostile in self-hosted Zitadel because the org's `primaryDomain` auto-generates as `.` (e.g., `acme.idp.example.com`) — completely unrelated to the company's email domain (e.g., `example.com`). So enabling the flag without first adding a custom domain via Organization Domains → Add (and verifying it) produces nonsensical loginNames like `user@acme.idp.example.com`, or worse, double-suffixed `user@example.com@acme.idp.example.com` if the userName already had `@`. **Recovery for stamped users**: edit the affected user's `userName` to a temporary value, Save, then change it back to the desired value, Save again — this forces Zitadel to recompute the loginName under the (now disabled) policy. Last resort: delete + recreate. **Lesson**: do NOT enable this flag in production until you have added and verified a custom domain matching your users' email domain. The setting is only sensible as a multi-tenant disambiguator across orgs that share usernames. → `troubleshooting.md §"Login Name stamped with org domain after policy toggle"`. 35. **Console "Add Human User" leaves "E-mail Verificado" / "Email verified" unchecked by default — first login then triggers an SMTP verification code that never arrives if SMTP isn't configured** — the form has the verified checkbox immediately under the E-Mail field, and it's easy to miss. When unchecked, the freshly-created user logs in fine up to the password step, then gets stuck on a "Verifique seu e-mail / Code sent to your email" screen. In self-hosted deployments where SMTP is commonly deferred ("we'll wire it up later"), no code is ever delivered and the user is permanently stuck. **API creation avoids this** entirely — `AddHumanUserRequest` lets you set `email.isVerified: true` explicitly in the payload, which bootstrap scripts typically do. **Fix retroactively**: detail page → CONTACT INFORMATION → pencil icon next to email → modal opens → tick **"Is verified"** → Save. Do NOT use "Resend Code" — that hits the same broken SMTP path. **Better long-term**: configure SMTP (the entire invite-and-self-service flow depends on it: password reset, MFA enrollment notifications, email change verification). → `troubleshooting.md §"User stuck on email verification code after manual creation"`. 36. **Backend container that validates JWT needs `extra_hosts: idp.:host-gateway` when IdP is on the same host with unreliable hairpin NAT** — `jose.createRemoteJWKSet(new URL(jwksUrl))` does an ordinary Node `fetch` on every JWKS reload (default `cacheMaxAge: 600s`). When the backend container resolves `idp.` via public DNS, it gets the host's external IP — and on VPS providers where hairpin NAT (DNS → external IP → loopback to local nginx-proxy) is flaky, every reload after the 10-minute TTL fails with a network error. The failure surfaces as `JOSEError` from the catch block, which the validator throws as `InvalidTokenError` → 401. **Symptom: backend works perfectly for ~10 minutes after each restart, then 401-storms every authenticated request even though JWTs are decoded-by-hand correct (`iss`/`aud`/`exp`/`kid` all valid; the `kid` is verifiably present in the JWKS endpoint when fetched from outside the container).** Family of quirks 12 (self-signed JWKS) and 13 (volume reset stale clientId) — same symptom, different cause; this is the third documented cause of "401-storm with apparently-valid JWT". **Fix**: add `extra_hosts: ["idp.:host-gateway"]` to the backend service in `docker-compose.prod.yml` so the container resolves the IdP hostname to the docker-bridge IP — same path external traffic takes. **Diagnostic in 3 commands**: `docker exec getent hosts idp.` (should be the bridge IP), `docker exec wget -qO- $JWKS_URL | head -c 200` (should be valid JWKS), `docker logs --tail 100 | grep '\[auth\] JOSE'` (any `code=ERR_JOSE_*` confirms the JWKS-after-TTL failure mode). → `token-validation.md §"Network reachability to the IdP from the JWT validator"` + `troubleshooting.md §"401 storm starting ~10 min after backend restart"`. 37. **Frontend defense-in-depth against the 401 → silent-renew → RT-reuse → session-revoke cascade requires THREE coordinated layers, not just dedupe** — Zitadel has refresh-token rotation with reuse detection enabled by default; if the SPA fires `signinSilent` twice with the same RT (race between concurrent 401-retry callers OR race between 401-retry and `addAccessTokenExpiring` handler), Zitadel detects reuse and **revokes the entire session** — every access token AND refresh token in that session becomes invalid. After that, re-login creates a fresh session but if the underlying root cause persists, a new storm revokes that too → infinite loop. The three layers that close the cascade: **(L1) Dedupe lives in `ApiClient`, not in the provider** — instance field `pendingRenew: Promise|null` cleared in `.finally()`; expose `apiClient.refreshToken()` public method that returns the deduped Promise; survives provider remount (test harnesses, Storybook), is testable in isolation, AND is shared between 401-retry path and `addAccessTokenExpiring` handler. Provider-level `useRef` dedupe is fragile because the expiring handler typically calls `signinSilent()` directly, bypassing the dedupe. **(L2) TanStack Query `retry` predicate filters `ApiError 401`** — 401 is not transient (the user needs re-auth, not retry), and retrying amplifies the storm by N requests/refresh attempts. Predicate: `(failureCount, error) => error instanceof ApiError && error.status === 401 ? false : failureCount < N`. **(L3) Listener for the `apiclient:unauthorized` CustomEvent in `` with `isAuthRoute()` early-return + `state: { returnTo }` on `signinRedirect`** — without the guard, a 401 bubbling up from `/auth/callback` or `/login` itself triggers another `signinRedirect` → IdP rejects → 401 → loop; without `state.returnTo` the user lands on home/`/` after re-auth instead of the page they were on. The three layers are independent and addressing only one of them leaves a known leak path. → `spa-recipes.md §"Recipe — Defense in depth against 401-storm-revokes-session"`. 38. **CI bind mount perms make `ZITADEL_FIRSTINSTANCE_PATPATH` fail with EACCES, then *cascades* into `unique_constraints_pkey` on retry — looks like Quirk 28 but isn't** — `ZITADEL_FIRSTINSTANCE_PATPATH=/current-dir/admin.pat` writes via the container's uid (1000) into a host bind mount; on a GitHub Actions `ubuntu-latest` runner that directory is created post-checkout owned by uid 1001 with mode `0755`, so the first boot fails on `03_default_instance` with `permission denied`, `setup failed, skipping cleanup` leaves partial state in the eventstore, and every retry under `restart: always` then fails with `unique_constraints_pkey` — the same tail as Quirk 28 (#8910/#9293), caused by host perms instead. Scroll up to the *first* migration attempt: if it failed with `permission denied`, this is Quirk 38, not 28, and the cure is a `chmod 0777` on the bind mount before `docker compose up` (CI-only; dev gets your user's perms). → `troubleshooting.md §"migration failed name=03_default_instance err.parent=...permission denied"` + `docker-compose-bootstrap.md §"Smoke-e2e plumbing checklist for GHA"`. 39. **Default Zitadel password policy requires all FOUR character classes — `openssl rand -hex` is lowercase-only and trips it deterministically** — fresh instances enforce upper + lower + digit + symbol on `AddHumanUser` (and any password-bearing API); the common CI shortcut `openssl rand -hex 16` only emits `[0-9a-f]` and dies with `400 invalid_argument: Password must contain upper case (COMMA-VoaRj)` on the seed-user step, and patching in one class at a time just trips the next. Fix: a structured generator with a 4-class prefix (`Aa1!` + random alphanumeric tail — the reference has the one-liner). Have the bootstrap read the seed password from env and fail fast when it is missing or short; CI re-runs and direct `docker compose --profile bootstrap run` invocations must export it explicitly. → `troubleshooting.md §"Bootstrap fails with COMMAND-VoaRj"` + `docker-compose-bootstrap.md §"Smoke-e2e plumbing checklist for GHA"`. 40. **`zitadel-login` (Login UI v2 Next.js container) takes 90+ seconds to first healthcheck render on small CI runners — `up --wait` for the WHOLE stack times out before bootstrap or REST tests get to start** — its healthcheck (`wget --spider http://localhost:3000/`) only passes after Next.js bootstrap completes; on `ubuntu-latest` (2 vCPU) that is ~90 s+, so `--wait-timeout 120` with `start_period: 30s` abandons before the container is Healthy even though `zitadel`, `zitadel-init` and `zitadel-db` were Healthy long before. Bootstrap and REST tests don't need the Login UI, so scope `--wait` to `zitadel-db zitadel-init zitadel` (the reference has the step); if a Playwright login spec does need it, raise `--wait-timeout` to ~240 s instead. Include `zitadel-login` in the on-failure log dump even when you don't wait for it. → `troubleshooting.md §"zitadel-login container never goes Healthy in CI"` + `docker-compose-bootstrap.md §"Smoke-e2e plumbing checklist for GHA"`. 41. **Idempotent bootstrap creates initial user grants but does NOT reconcile existing ones when the YAML evolves** — a declarative bootstrap shaped like `seedUser: { roles: [...] }` POSTs the grant on first run and treats `ALREADY_EXISTS` as a no-op afterwards, so when you add a new app + role to the YAML the pre-existing seed user's `roleKeys` never gains it: JWTs keep going out with the day-0 role set and the new SPA's `requireRole` returns 401. A brand-new user works fine — the operator most likely to test first is the one locked out. Easily confused with quirks 11 (access vs id token claims), 12/13/36 (the 401-storm family) and 28 (Login UI v2 blocking). Diagnostic: decode the JWT and inspect `urn:zitadel:iam:org:project:roles`. Fix now: search-then-PUT the grant with the FULL desired `roleKeys` (PUT replaces, not merges — Quirk 8's global search first); fix durably: make the bootstrap reconcile each seed grant whose current `roleKeys` is a strict subset of the YAML. → `troubleshooting.md §"Seed user JWT missing role added later to YAML"`. 42. **Browser SPA → backend Express needs CORS — preflight `OPTIONS` hits `authJwt` and 401s, mimicking quirks 12/13/36** — `localhost:5173` (Vite) and `localhost:3000` (Express) are cross-origin, so every `Authorization`-bearing request is preceded by a preflight `OPTIONS` that carries no Bearer; with no CORS middleware ahead of the auth chain the preflight 401s (`jwt_ausente`), the browser aborts the real request, and the SPA observes "every API call fails 401" — identical to the 401-storm family (Quirks 12/13/36/37). The diagnostic is the asymmetry: the same backend answers 200 to `curl -H "Authorization: Bearer "` (no Origin → no preflight). MSW/supertest tests never issue a real preflight, so only a browser (Playwright smoke, manual) catches it. Fix: a CORS middleware as the FIRST app-level middleware — before auth, body parsing and logging — short-circuiting `OPTIONS` with 204 (the `cors` package or the inline one in the reference). → `troubleshooting.md §"Browser 401 storm on cross-origin API, same backend OK via curl"`. 43. **Zitadel's own admin Console 404s all its API calls with "[unknown] Failed to fetch" when the generated `environment.json` renders `api: http://…` while `issuer: https://…`** — the Console is an Angular SPA served by the Zitadel binary at `/ui/console`; it reads `/ui/console/assets/environment.json` (generated in-memory, **no file on disk**) for its API base. `issuer` is built from the **persisted instance config** (written at FirstInstance) so it stays `https`, but `api` is computed **per-request** from the connection scheme the binary detects. Run the binary with **`--tlsMode disabled`** behind a TLS-terminating proxy and it does NOT trust the proxy's `X-Forwarded-Proto: https`, so `api` comes out `http://` — and an HTTPS page calling an `http://` origin is **active mixed content**, blocked by the browser before the request leaves the page (surfaces as `TypeError: Failed to fetch`). The `issuer`/`api` scheme **split** is the fingerprint that separates this from a plain proxy misconfig (Quirk 15): OIDC login and downstream apps keep working (they use `issuer`), so only the Console breaks and it's easy to dismiss as "the console is just flaky". **Compounding trap**: `docker start`/`restart` revives a container's **create-time** `Args`/env, NOT the current compose files — a container first created from a base compose (`--tlsMode disabled`, `EXTERNALSECURE=false`) stays broken forever even after you add the override with `--tlsMode external`; the cure is `docker compose … up -d --force-recreate `, never `docker start`. **Diagnose in 3**: `curl -sk https:///ui/console/assets/environment.json` (compare `api` vs `issuer` scheme) · `docker inspect --format '{{json .Args}}'` (`--tlsMode` `external` or `disabled`?) · in-browser `fetch('http:///oauth/v2/keys')` → `Failed to fetch` while the `https://` twin returns 200. **Fix**: ensure the running container has `--tlsMode external` + `EXTERNALSECURE=true` (Quirk 15's triad) and was **recreated** from that config; then `api` renders `https` and matches `issuer`. The browser may still hold the old `environment.json` via service worker/cache → hard-reload. → `troubleshooting.md §"Zitadel admin Console: [unknown] Failed to fetch (environment.json api http vs issuer https)"` + `docker-compose-bootstrap.md §7`. 44. **You can prove `client_id` + `redirect_uri` are correct WITHOUT any credential — hit `/oauth/v2/authorize` and look for the login redirect** — every other client check in this skill needs a PAT (`GetApplication`, Console), which makes them useless in the one moment you most need them: a pre-cutover check on a fresh environment where nobody has logged in yet. Zitadel validates `client_id` and byte-matches `redirect_uri` **before** rendering anything, so building the authorize URL by hand (with the RFC 7636 test-vector `code_challenge` — you never complete the flow, so no verifier is needed) and getting `302 https:///ui/login/login?authRequestID=…` proves both are registered. Anything else is a finding: a `400`/error page means the app isn't registered under that `client_id` (Quirk 29 — you probably passed the `applicationId` UUID instead of the numeric `clientId`) or the `redirect_uri` isn't in `redirectUris` (Quirk 18's byte-matching applies, trailing slash included). Pair it with Quirk 45 to close the loop: 45 proves the SPA *asks* for the right values, 44 proves the IdP *accepts* them. → `docker-compose-bootstrap.md §"Como detectar que o roteamento está OK"`. 45. **The OIDC config a deployed SPA actually uses is inside its JS bundle — fetch the bundle over HTTPS and grep it, don't infer it from the workflow** — `VITE_*` (and `NEXT_PUBLIC_*`) are inlined at build time, so "the secret is set" and "the bundle contains it" are different claims: a build that ran before the secret existed, a cached layer, a rebuild that never happened, or an image tag that resolved to another environment's build all produce a bundle whose `client_id` disagrees with your CI config, and the symptom is a login that simply never completes. Grep the **served** bundle (not the one inside the container) for the authority host, the numeric `client_id`, the redirect URI and the API base URL, and check that none of them belong to another environment — the served copy also catches a stale container still registered in the reverse-proxy upstream pool (Quirk 32's neighbourhood). Fix: rebuild the frontend image — updating the secret alone is a no-op, since it is read at the *next* build. → `troubleshooting.md §"OIDC client_id mismatch"` + Quirk 29. 46. **When two applications share one Zitadel, `ZITADEL_SEED_USER_ROLE` must be a LIST — `CreateAuthorization` takes it literally, and the bug only surfaces when the IdP is rebuilt** — bootstrap scripts commonly grant the seed admin whatever roles that env var names. Pin it to the new app's role (`quote.admin`) and the *other* app's admin grant silently disappears the next time the Zitadel volume is reset and the bootstrap runs from scratch, because the create path writes the literal list it was given. What hides it is the update path: `UpdateAuthorization` performs a **union**, so on an existing instance the old roles survive and everything looks fine — the defect is dormant for as long as nobody recreates the IdP, which is exactly when you least want to discover it. **Distinct from Quirk 41**, which is the adjacent problem on a *live* instance (bootstrap creates the grant but doesn't reconcile `roleKeys` when the YAML gains an app). **Fix**: make the var a comma-separated list of every role the seed user needs across all apps (`battery.admin,quote.admin`) and derive it from the YAML rather than hardcoding per-deployment; then prove it by resetting a throwaway instance and re-running the bootstrap. **Tell** on a rebuilt instance: login completes and the API answers `401 role_nao_reconhecida` for the app you did *not* name. → `troubleshooting.md §"Seed user JWT missing role added later to YAML"` (Quirk 41) + `api-v1-to-v2-mapping.md §4 AuthorizationService`. 47. **Running two Zitadel deployments in parallel (prod + staging) is fine; migrating users between them is asymmetric — importing a password hash is supported, extracting one is not** — separate instances mean separate `clientId`s, separate project IDs (so separate audiences) and separate PATs; the failure mode is a token from one validated against the other, which surfaces as `JWSSignatureVerificationFailed` and reads like a key-rotation problem. Keep audience and client id per environment and never let a "shared" secret span instances. The migration question follows quickly, and the answer is lopsided: `AddHumanUser` (API v2) accepts a `hashedPassword` in standard crypt formats (bcrypt, argon2), so the **destination** can take users verbatim — but there is no API that returns a user's hash from the **source**. It lives in the eventstore (`user.human.password.changed`), and reading it means depending on internal schema that changes across versions. **Low-risk path**: recreate the accounts through the API (username, email, profile) with `changeRequired: true` so each person sets a password on first login, then re-grant roles in the destination project — the grants do not travel with the user. **Measure before deciding**: how many users actually exist in the source (below a few dozen, recreating beats any extraction scheme) and the Zitadel version on both ends. → `references/multi-instance-and-user-migration.md`. 48. **A project role key is immutable — renaming one is a four-step migration, and the app has to accept both names while it runs** — `UpdateProjectRole` changes `displayName` and `group` only; upstream is explicit that *"the key is not editable. If a key should change, remove the role and create a new one"*. So the rename is: create the new role → grant it to everyone holding the old one (search-then-PUT per Quirk 8 — `roleKeys` **replaces**, so send the union) → drop the old key from those grants → delete the old role **last**. Two things make this bite harder than it reads. **The declarative bootstrap is additive**: `ensureRoleSet`-shaped code creates what is missing and no-ops on what exists, so the old key survives in the YAML and is **recreated on every fresh instance** until someone deletes the declaration *and* the role — the same blind spot as Quirks 41 and 46, one level up (those are about grants, this is about the role's lifecycle). And **the ordering turns into a cross-repo dependency** whenever the IdP is declared in a different repository from the app, which is the normal shape once two products share one Zitadel: deploy the app before the migration, or delete the old role before granting the new one, and every user gets `401 role_nao_reconhecida` at the same moment, because the claim now carries a key the app does not know. What removes the dependency is an **alias at the claim boundary** — the backend's claim parser and the SPA's role extractor accept *both* keys and normalize to the new one, with a dated removal. But be precise about how much it buys: **the alias frees the order of *granting*, never of *withdrawing*.** It lives in the new build, so until that build is serving in a given environment, dropping the old key from someone's grant leaves them holding only a key the running binary has never heard of — the same `401 role_nao_reconhecida`, now aimed at exactly the people you just touched. And because `roleKeys` replaces, "grant the new" and "withdraw the old" are physically the same call, which is what makes this easy to get wrong: the API invites a single `PUT` carrying the final desired set, and that single `PUT` is the outage. Split it across the deploy boundary on purpose. Measured 2026-09-18: with the alias in place the two keys coexisted in a live project (`roles/_search` returning both) and the app's browser suite passed against an account genuinely re-granted to the new key — while the migration runbook written from the un-qualified version of this quirk shipped with the withdrawal step ahead of the deploy, and a PR reviewer caught it. → `api-cheatsheet.md §"Renaming a project role"` (the API calls) + `role-migration.md` (running it across environments). 49. **`ListProjectRoles` has no Connect/v2 equivalent — and a guard built on it fails for the wrong reason, silently** — listing a project's roles by the path you would guess by analogy (`/zitadel.management.v1.ManagementService/ListProjectRoles`) returns `404 {"code":5,"message":"Not Found"}` in v4.15.0; the working call is still REST v1, `POST /management/v1/projects/{projectId}/roles/_search`, with the project in the path and the org in the `x-zitadel-orgid` header (harmless on v2 calls, per Quirk 27). Most of the v1→v2 migration is a mapping table, so the assumption that *every* call has a v2 twin is the natural one to carry — and it is the assumption that bites here. **Why this is worse than a 404 in a log**: when the read backs a guard — *"refuse a role that is not declared in the project"* — the guard now refuses **everything**, for a cause that has nothing to do with the role, while printing the same message it prints when working correctly. A refusal by the wrong cause is a refusal that lies, and it keeps lying until someone exercises the guard with a role that **does** exist. General rule this leaves behind: test a guard's positive path against a live instance, not only the input it should reject. ⚠️ And when scripting the check, remember `script | grep` returns **grep's** exit code — a sabotage test reading `$?` through a pipe reports `0` while the script exits `1`. → `role-migration.md §4`. 50. **A config field the schema accepts and the bootstrap never reads is worse than a missing one — it is an answer, and it is wrong** — declarative bootstraps grow a config schema faster than the code that consumes it, so a field like `seedUser.roles` ends up declared, validated (`z.array(z.string().min(1)).default([])`) and never read, because the value is still taken from an env var written earlier. A missing field is a question someone asks; a field that parses cleanly reads as the source of truth, and reviewers reasonably treat it as one. Meanwhile the real source is an env var living in compose files — plural — so the environment whose compose forgot it falls back to the default with no error at all. **Measured**: prod set `ZITADEL_SEED_USER_ROLE=battery.admin,quote.admin`, staging did not set it, and staging's seed user was born with `battery.admin` alone; nothing failed, and the first person to notice was whoever tried to use the second product there. **Fix is precedence, not replacement** — `env > YAML > default`, because flipping the source outright changes behaviour at the next deploy in the one place you least want a surprise; treat a **defined-but-empty** env as "no choice" and fall through (an empty var crosses `envsubst`, compose interpolation and every `??` without complaint — Quirk-13's family, in config form); and log which source won, since precedence is invisible in a compose file. Pair it with a `superRefine` that rejects a seed role absent from `applications[].roles[].key`, naming the offending value and the valid set — otherwise reading the field merely moves a typo's blast radius to `CreateAuthorization`, far from the file that caused it. **The smell generalizes past this field**: ask of each config key *what breaks if I delete it?* — if the answer is "nothing", it is not configuration, it is a comment that validates. Family of Quirks 41 and 46 (grants that the bootstrap creates but never reconciles); this is the layer above — the **declaration** that never reached the code. → `role-migration.md §2`. 51. **An idempotent bootstrap cannot tell you it succeeded — `reuse` is printed both when the declaration was satisfied and when the declaration itself is stale** — every bootstrap in this skill's shape logs `reuse`/`ALREADY_EXISTS`/`no changes` per resource, and that vocabulary is identical in two opposite worlds: the role really was there, or the script is reading a months-old YAML (`COPY`ed into the image and never rebuilt — Quirk 30's family, one layer down: 30 warns that the *output* file is only refreshed when the bootstrap runs, this is the **input** going stale the same way). The tell is a *negative* — no `created` lines — and nobody notices a line that is not printed. **Prove it against the instance, by the path that needs no credential**: the API answer is `POST /management/v1/projects/{projectId}/roles/_search` (REST v1; there is no Connect/v2 twin — Quirk 49), but it needs a PAT, which is exactly what you lack on a fresh or third-party environment. The Postgres projection answers with no token and no expiry — `select role_key, creation_date from projections.project_roles4 where role_key like '.%'` — and the **`creation_date` is what makes it decisive**: roles dated at the instance's birth, in a deploy meant to add two, say the bootstrap applied an old list rather than failed. ⚠️ Discover the projection's numeric suffix from `information_schema` instead of hardcoding it (`project_roles4` in v4.15.0; it moves across releases, and a wrong guess fails with `relation does not exist`, which reads like a broken database). Measured 2026-09-18: four green CD steps, zero roles created, image from 13 June. **Then fix the gate**: a deploy whose product is *state* needs an assertion about state — the cheapest is derived from the declaration (every `roles[].key` in the YAML must appear in the bootstrap's own output), which also catches the stale-input case that querying the instance does not. → `troubleshooting.md §"The bootstrap logs `reuse` for everything"`. 52. **Every role-related quirk in this skill ends in the same `401` loop — and they all assume the reader that rejects the token is the build you are reading** — Quirks 41, 46 and 48 tell you how a claim ends up carrying a role the app does not know (grant not reconciled, seed role list overwritten, key withdrawn before the deploy). The symptom they share is worth naming on its own, because it names neither roles nor permissions: the login page accepts the credentials, the browser reaches the callback and **bounces straight back to the IdP**, over and over. That happens because the API answers `401` for an unrecognised role and the SPA is built to react to `401` by re-authenticating — so an *authorization* failure is reported as an *authentication* failure and the app tries to fix it by logging in again, forever. **The cheapest triage fact is free**: another account with a different role logs in fine, which rules out the instance, JWKS, audience, redirect URIs and Login UI in one stroke and points at the role in the claim. Then decode the token (role absent → 41/46) and read the response *body*, not the status (`401`-with-a-role-code means the reader parsed the token and rejected the role; `403` would mean it understood the role and denied the route). ⚠️ **And then prove the process serving the port runs the code you are reading — the step the other quirks assume.** Measured 2026-09-18: a dev backend whose file watcher had silently stopped was serving a build from before the role existed; grant, claim and source were all correct, and only that one role 401'd. Reasoning from timestamps actively misled — the running child was *younger* than the commit that added the role and still did not have it. What settles it is behavioural: `touch` a source file and watch whether the PID changes (dev), or read which image tag the container is actually running (prod). Compare the artifact, never the intent. **Design corollary worth acting on**: an unrecognised role means *authenticated but not permitted*, so `403` is the honest status — returning `401` is what manufactures the loop, and it is your contract doing it, not Zitadel's. → `troubleshooting.md §"One account loops back to the login screen while the others log in fine"`. ## Migration v2.66 → v4 + API v2 When upgrading from v2.66.x to v4.x and refactoring callers from API v1 to v2: - **Start with the runbook** in `references/migration-v2-to-v4.md` — pre-flight (Postgres required since v3, advisory A10015), upgrade path (direct v2.66 → v4 OK if Postgres in place — no v3 stop), schema migration runs automatically in the v4 image's `setup` phase, validation matrix, rollback. - **Use the mapping table** in `references/api-v1-to-v2-mapping.md` — covers all 15 v2 services (Organization, Project, Application, User, Authorization, Action, Feature, Settings, OIDC, IDP, Group, SAML, Session, WebKey, Instance), payload diffs (`firstName/lastName` → `givenName/familyName`, `userName` → `username`, `email.isEmailVerified` → `email.isVerified`, language `pt-BR` → `pt`), and idempotence patterns. - **Login UI v2 is a separate container** — see Quirk 25 above. Reverse proxy must route `/ui/v2/login` to `zitadel-login:3000`, everything else (including OIDC discovery, OAuth, JWKS) to `zitadel:8080`. Path B (sticking with Login UI v1) is supported — Login UI v1 keeps working at `/ui/login/` indefinitely. - **Bootstrap idempotence in v2 differs** (Quirk 26) — deterministic IDs in body replace `_search`-then-create round-trip. Existing v1-shaped bootstrap scripts keep working in v4; refactor only when there's a reason. - **Connect protocol auth is unchanged**: same `Authorization: Bearer ` header. JSON variant uses `Content-Type: application/json`. Binary variant uses `application/connect+proto` (only if you have a generated client). ## Implementation flow (suggested order) For a fresh project, this sequence avoids most reordering: 1. Copy `assets/docker-compose.zitadel.yml` and adjust ports / external domain. Read `docker-compose-bootstrap.md §1-3` first — the env layout is non-obvious. 2. `docker compose up -d`, wait for healthy, verify the PAT was written to `/admin.pat`. If not, see troubleshooting. 3. Copy `assets/bootstrap-zitadel.ts` and run it to create your Org/Project/Roles/App. The script is idempotent — re-runs are safe. 4. Capture the `projectId` and `clientId` from the output JSON. These become `OIDC_AUDIENCE` and the SPA's client_id respectively. 5. In your backend, wire JWT validation per `token-validation.md`. The audience must equal the `projectId`, NOT the `clientId`. 6. In your SPA, configure Auth Code + PKCE pointing at `http:///.well-known/openid-configuration`. 7. Test login end-to-end. If anything 4xx's, jump straight to `troubleshooting.md`. ## What this skill explicitly does NOT cover - Production hardening (TLS, masterkey rotation, Postgres backup, SMTP) — these are real concerns but vary by environment. Quick pointer: terminate TLS at NGINX/Caddy/Traefik and set `ZITADEL_EXTERNALSECURE=true` + `ZITADEL_TLS_ENABLED=false` + start flag `--tlsMode external` (full triad — see `docker-compose-bootstrap.md §"TLS terminated by reverse proxy"`, quirk 15). Backup the Zitadel Postgres database like any other Postgres. - v3 → v4 migration. Greenfield projects should start on v4. The skill covers v2.66 → v4 (the supported direct hop when Postgres is already in place — see `migration-v2-to-v4.md`), not v3 as an intermediate stop. - Federation, SAML, SCIM, Actions, Webhooks. Add them when needed — they are mostly straight reads of the docs once Zitadel is bootstrapped correctly. ## Source of truth - Bundled working references: `assets/bootstrap-zitadel.ts` (idempotent bootstrap) and `assets/docker-compose.zitadel.yml`. - Upstream docs: and the steps file at . When the references in this skill diverge from upstream, trust the upstream docs but raise a note — Zitadel evolves quickly and these references will drift.