cosec-troubleshoot git:20261013.0496a70A

Immutable. This exact content never changes and is served at /api/v1/blob/ec855616e42f5c90.

---
name: cosec-troubleshoot
description: Diagnose CoSec authentication and authorization failures, including unexpected 401/403/429 responses, policy loading, JWT rejection, matcher mismatches, rate limit rules, and unexplained access decisions. Do not change behavior unless the user asks for a fix.
---

# CoSec Troubleshooting

The human-facing guide is https://cosec.ahoo.me/guide/troubleshooting; this skill adds the agent workflow.

Find the first incorrect transition from request parsing to authentication, policy loading, matching, and response mapping. Gather evidence before proposing a policy or code change.

## Start with observable facts

Capture the request method/path, response status and CoSec JSON reason, principal ID/authenticated state, app/space/tenant IDs, effective configuration, loaded policy IDs, and relevant startup/request logs. Redact tokens and secrets.

Enable focused logging:

```yaml
logging:
  level:
    me.ahoo.cosec.authorization.SimpleAuthorization: DEBUG
    me.ahoo.cosec.audit: DEBUG
```

`SimpleAuthorization` identifies the matched policy/statement or role/permission. When `AuditingAuthorization` is active, the default sink emits structured deny events at WARN and allow events at DEBUG, including principal, request, decision, and matched rule when available. A custom `Authorization` bean is not wrapped automatically.

When `cosec.explain.enabled=true`, prefer explaining the decision over reading logs: inject `AuthorizationExplainer` (or call `POST /actuator/cosecExplain` with `principal` and `request` objects) to get the real status, the deciding rule (omit `principal` and pass the caller's `Authorization` token or `X-API-Key` in `request.headers` to explain a real credential; rejected credentials add a `TOKEN_INVALID` warning), every statement's and permission's outcome (`MATCHED`, `ACTION_MISS`, `CONDITION_MISS`, `NOT_EVALUATED`, `ERROR`), overridden allows, matching rate limit rules, and warnings such as `MISSING_POLICY` or `NO_APP_PERMISSION`. It spends no rate-limit permits. Treat `TRACE_INCONSISTENT` as a CoSec bug and trust the reported decision.

## Interpret the status first

| HTTP status | CoSec meaning |
|---|---|
| 400 | Invalid normalized request path |
| 401 | Authorization denied while the parsed principal is anonymous; invalid/expired tokens also fall back to anonymous and preserve the token reason |
| 403 | An authenticated principal was denied, or a regex matcher timed out and failed closed |
| 429 | An allowed request exceeded a rate limit rule (`RateLimitingAuthorization`) |
| 500 | Unexpected authorization error, including Redis being unavailable for policy/permission reads (those caches fail closed); inspect the server exception |

The authorization result alone does not choose 401 versus 403; the filter maps a denied anonymous principal to 401 and a denied authenticated principal to 403.

## Decision order

`SimpleAuthorization` evaluates:

1. Root principal ID (`cosec` by default, overridden by JVM system property `cosec.root`) → allow, unless `cosec.authorization.root-enabled=false` or a custom `RootPrincipalMatcher` bean decides otherwise.
2. Blacklist → explicit deny.
3. Global policies.
4. Principal-attached policies.
5. Role permissions for `request.appId` + `request.spaceId`.
6. No match → implicit deny.

Each policy tier evaluates matched policy-level conditions once, pools all statements, and checks every deny before any allow. Tiers are combined deny-first (CoSec 6+): an explicit deny in any tier wins over an allow from any other tier, so a principal or role deny overrides a global allow. Only if no tier denies does a matched allow grant access.

## Symptom guide

### Unexpected 401

- Check whether the request actually carried a bearer token and whether parsing produced an authenticated principal. Tokens are read from the `Authorization` header or `authorization` cookie; the `authorization` query parameter is ignored unless `cosec.authorization.allow-query-token=true` (default `false` since CoSec 6).
- Match `cosec.jwt.algorithm` and secret to the issuer; only HMAC256/384/512 are supported.
- Check expiration and the response reason (`Token Expired` versus `Token Invalid`).
- If revocation is enabled, verify the cache capability supplied a real Redis-backed `TokenStore`; otherwise the starter warns that logout is ineffective.
- If the token is valid but the request remains anonymous, inspect the active `SecurityContextParser` and transport filter.

### Unexpected 403 or missing allow

- Read the debug/audit match. An explicit deny is a matched deny; an implicit deny means no rule in any tier matched.
- Confirm a global policy loaded and its policy-level condition matched.
- Check the action's HTTP method, path, trailing slash, and captured variables.
- Check condition parts and the actual principal roles/attributes/tenant.
- Remember that `inTenant.value` is `default`, `user`, or `platform`, not a tenant ID.

### Request allowed unexpectedly

- Check root bypass first.
- Find which tier matched an allow (the debug log and audit event name the policy statement or role permission). If you expected a deny, check that the deny's action and condition both match: a matched deny in any tier would have won.
- Verify the deny action and condition both match; array order cannot make an allow beat a same-tier deny.
- Confirm the intended policy was loaded rather than skipped or shadowed by a duplicate ID.

### Policy file has no effect

- Confirm `cosec.authorization.local-policy.enabled=true` and the file matches `local-policy.locations`.
- Inspect startup errors: `LocalPolicyLoader` logs and skips malformed JSON, missing required fields, unknown matcher types, and constructor-time configuration errors rather than failing the application.
- Deferred matcher errors still load. For example, an unsupported `part` throws during authorization and the transport returns 500; reproduce it with `DefaultPolicyEvaluator` or the failing request.
- Local policies are deduplicated by ID after location expansion; assert that the expected ID and content loaded.
- `LocalPolicyLoader` does not serve authorization decisions directly. Enable `init-repository` or prepopulate `PolicyRepository`; use `force-refresh` only when overwriting repository state is intended.

### Matcher mismatch

- Path variables use `{id}` and conditions read `request.path.var.id`; `:id` and `request.pathVariable.id` are invalid.
- Path SpEL templates use `#{principal.id}`. SpEL condition expressions use root properties such as `context.principal.id`; they are different evaluation roots.
- Valid part prefixes are singular `request.header.`, `request.attributes.`, `request.path.var.`, and `context.principal.attributes.`. Unsupported parts throw `IllegalArgumentException`.
- Regex timeouts fail closed as 403. Simplify or bound the pattern instead of increasing exposure to ReDoS.

### Rate limiting differs across instances

- Without the cache capability and a `StringRedisTemplate` bean, the `RateLimiter` is `LocalRateLimiter`: per JVM, so N instances allow N times the limit. Check which `RateLimiter` bean is active.
- Counter-store outages skip a rule by default. Set the rule's `strictFailure: true` only when availability loss should return 429.
- Denied requests (401/403) never spend permits. A rule that never applies usually has a non-matching `action`/`condition`, or its file is outside `cosec.limiter.locations`.
- A rule without `key` shares one bucket among all matching requests; requests whose `key` part is empty also share one bucket. Quotas belong to the rule `id`.
- After upgrading to 6.1, policies using `rateLimiter`, `groupedRateLimiter`, `redisRateLimiter`, or `redisGroupedRateLimiter` fail to parse: migrate them to rate limit rules.

## Reproduce narrowly

Prefer the smallest existing test layer that crosses the failing boundary:

- matcher behavior: create the real factory with `mapOf(...).asConfiguration()` and test `match`;
- policy parsing: load the exact resource with `LocalPolicyLoader`, assert its ID, then run `DefaultPolicyEvaluator.evaluate`;
- decision order: test `SimpleAuthorization` with the smallest repositories needed;
- transport mapping: use the WebFlux, WebMVC, or Gateway filter tests for 401/403/429 behavior;
- Redis behavior: run the cache integration test with Redis.

In this repository, run a focused class first, for example:

```bash
./gradlew :cosec-core:test --tests "me.ahoo.cosec.policy.DefaultPolicyEvaluatorTest"
```

Use `$cosec-policy-author` to rewrite a confirmed-bad policy and `$cosec-integration` to change confirmed-bad Spring configuration. For diagnosis-only requests, report the root cause and evidence without modifying files.