72 added, 251 removed. Audit A to A.
---
name: cosec-troubleshoot
- description: "Use when diagnosing CoSec authentication or authorization failures such as unexpected 401/403 responses, denied requests that should be allowed, policies not loading, JWT token rejection, matcher mismatches, or unclear access decisions."
+ description: Diagnose CoSec authentication and authorization failures, including unexpected 401/403/429 responses, policy loading, JWT rejection, matcher mismatches, Redis limiters, and unexplained access decisions. Do not change behavior unless the user asks for a fix.
---
- # CoSec Troubleshooting Guide
-
- This skill helps you debug authorization issues in CoSec. When a request gets an unexpected result (403, 401, 429, or is allowed when it shouldn't be), follow this systematic approach.
+ # CoSec Troubleshooting
- ## Step 1: Enable Debug Logging
+ Find the first incorrect transition from request parsing to authentication, policy loading, matching, and response mapping. Gather evidence before proposing a policy or code change.
- The fastest way to understand authorization decisions is debug logging on `SimpleAuthorization`:
+ ## Start with observable facts
- ```yaml
- logging:
- level:
- me.ahoo.cosec.authorization.SimpleAuthorization: debug
- ```
+ Capture the request method/path, response status and CoSec JSON reason, principal ID/authenticated state, app/space/tenant IDs, effective configuration, loaded policy IDs, and relevant startup/request logs. Redact tokens and secrets.
- This logs every matched statement, the policy/statement it came from, and the final result, e.g. `Verify [request] [context] matched Policy[globalPolicy] Statement[2][RequestOriginDeny] - [EXPLICIT_DENY].` (statement index is 0-based)
+ Enable focused logging:
- For more granular tracing:
```yaml
logging:
level:
- me.ahoo.cosec.policy: debug
- me.ahoo.cosec.authentication: debug
- me.ahoo.cosec.jwt: debug
- ```
-
- ## Step 2: Understand the Evaluation Order
-
- `SimpleAuthorization.authorize` evaluates in this order, falling through with `switchIfEmpty` when a step produces no match:
-
- ```
- 1. Root user check
- └─ If principal.id == root ID (default "cosec") → ALLOW (bypass everything)
-
- 2. Blacklist check
- └─ If principal is blacklisted → EXPLICIT_DENY
-
- 3. Global policies (type: "global")
- └─ Skip policies whose policy-level condition doesn't match
- └─ Pool ALL statements from ALL matched policies, then deny-first:
- a. Check every DENY statement → EXPLICIT_DENY if any matches
- b. Check every ALLOW statement → ALLOW if any matches
-
- 4. Principal-specific policies
- └─ Policies attached to the user (via policy IDs on the principal)
- └─ Same pooled deny-first evaluation as global policies
-
- 5. Role-based app permissions
- └─ Only when the principal has roles; permissions are looked up
- by request.appId + request.spaceId, then evaluated deny-first
-
- 6. Default → IMPLICIT_DENY
- ```
-
- Key consequence: a DENY statement in ANY policy of the same tier beats an ALLOW statement in ANY other policy of that tier — policy order within a tier does not matter.
-
- ## Step 3: Common Issues and Fixes
-
- ### All requests return 403
-
- **Symptoms:** Every endpoint returns 403, even public ones.
-
- **Likely causes:**
- 1. No policy files loaded — check `cosec.authorization.local-policy.enabled=true`
- 2. Policy files don't match the location pattern — default is `classpath:cosec-policy/*-policy.json`
- 3. Policy JSON syntax error — check startup logs for deserialization errors
-
- **Fix:**
- ```yaml
- cosec:
- authorization:
- local-policy:
- enabled: true
- locations: classpath:cosec-policy/*-policy.json
- ```
-
- ### Specific endpoint returns 403 when it should be public
-
- **Symptoms:** Most endpoints work, but a new public endpoint returns 403.
-
- **Cause:** No ALLOW statement matches the endpoint. By default, CoSec uses implicit deny — anything not explicitly allowed is denied.
-
- **Fix:** Add a statement for the endpoint:
- ```json
- {
- "name": "NewPublicEndpoint",
- "action": "/api/new-endpoint"
- }
- ```
-
- ### Request allowed when it should be denied
-
- **Symptoms:** A request that should be blocked gets through.
-
- **Likely causes:**
- 1. DENY statement doesn't match — check action pattern and condition
- 2. An ALLOW statement in a LATER tier matches (tiers fall through: global → principal → role permissions) — a DENY only overrides ALLOW within the same tier
- 3. Root user bypass — check if the user ID equals the root ID (default `"cosec"`, override with the `cosec.root` system property)
-
- **Debug:** Enable debug logging and check which statement matched.
-
- ### JWT token rejected
-
- **Symptoms:** Requests with valid JWT tokens return 401.
-
- **Likely causes:**
- 1. `cosec.jwt.secret` doesn't match the token issuer's secret
- 2. `cosec.jwt.algorithm` doesn't match the token's algorithm (supported: `hmac256`, `hmac384`, `hmac512`)
- 3. Token is expired (`token-validity.access` defaults to 10 minutes)
- 4. Token was revoked (logout) while `cosec.jwt.token-revocation.enabled=true`
-
- **Check:**
- ```yaml
- cosec:
- jwt:
- algorithm: hmac256 # must match the signing algorithm
- secret: exact-same-secret-used-by-issuer
- token-validity:
- access: 10m # access token TTL
- refresh: 7d # refresh token TTL
- token-revocation:
- enabled: false # when true, revoked (logged-out) tokens are rejected
+ me.ahoo.cosec.authorization.SimpleAuthorization: DEBUG
+ me.ahoo.cosec.audit: DEBUG
```
- ### Policies not loading from local files
-
- **Symptoms:** Startup succeeds but policies don't take effect.
-
- **Checklist:**
- 1. File location: `src/main/resources/cosec-policy/` (not `resources/main/...`)
- 2. File naming: must match `*-policy.json` pattern
- 3. Property: `cosec.authorization.local-policy.enabled=true`
- 4. JSON validity: parse errors are logged at startup
- 5. Policy type: must be `"global"` for the policy to apply to all requests
-
- ### Rate limiter not working
-
- **Symptoms:** Limits are per-instance instead of shared across the cluster.
-
- **Cause:** Both `rateLimiter` and `groupedRateLimiter` keep state in memory, per JVM instance — there is no built-in distributed limiter, and `cosec-cocache` does not provide one (it only caches policies/permissions/tokens).
-
- **Fix:** Implement a custom `ConditionMatcher` backed by a shared store such as Redis — see the `cosec-custom-matcher` skill. Also note: a tripped limiter produces `TOO_MANY_REQUESTS` (HTTP 429), not a plain 403.
-
- ### Path variables not matching
-
- **Symptoms:** `/user/123` doesn't match `/user/{id}`.
-
- **Check:**
- 1. Use `{varName}` — the `:varName` colon syntax is not supported by CoSec's path patterns
- 2. Access the variable via `request.path.var.varName` in conditions
- 3. Ensure the path pattern is correct (no trailing slash mismatch)
-
- ### SpEL template not evaluating
+ `SimpleAuthorization` identifies the matched policy/statement or role/permission. When `AuditingAuthorization` is active, the default sink emits structured deny events at WARN and allow events at DEBUG, including principal, request, decision, and matched rule when available. A custom `Authorization` bean is not wrapped automatically.
- **Symptoms:** `#{principal.id}` is treated as a literal string.
+ ## Interpret the status first
- **Cause:** SpEL templates use `#{}` syntax. `{}` alone is a path variable, not SpEL.
+ | HTTP status | CoSec meaning |
+ |---|---|
+ | 400 | Invalid normalized request path |
+ | 401 | Authorization denied while the parsed principal is anonymous; invalid/expired tokens also fall back to anonymous and preserve the token reason |
+ | 403 | An authenticated principal was denied, or a regex matcher timed out and failed closed |
+ | 429 | A local or Redis rate limiter threw `TooManyRequestsException` |
+ | 500 | Unexpected authorization error; inspect the server exception |
- **Fix:** Use `#{principal.id}` not `{principal.id}`.
+ The authorization result alone does not choose 401 versus 403; the filter maps a denied anonymous principal to 401 and a denied authenticated principal to 403.
- ### Condition part path is wrong
+ ## Decision order
- **Symptoms:** Condition always returns false, or evaluation throws `IllegalArgumentException: Unsupported part`.
+ `SimpleAuthorization` evaluates:
- **Valid part paths:**
- - `request.path` — full request path
- - `request.path.var.{name}` — path variable
- - `request.method` — HTTP method
- - `request.remoteIp` — client IP address
- - `request.origin` — Origin; `request.origin.host` — origin host
- - `request.referer` — Referer; `request.referer.host` — referer host
- - `request.appId` / `request.spaceId` / `request.deviceId` — identifiers
- - `request.header.{name}` — request header (singular `header`)
- - `request.attributes.{key}` — request attributes
- - `context.tenantId` — tenant ID
- - `context.principal.id` — user ID
- - `context.principal.attributes.{key}` — principal attribute
+ 1. Root principal ID (`cosec` by default, overridden by JVM system property `cosec.root`) → allow.
+ 2. Blacklist → explicit deny.
+ 3. Global policies.
+ 4. Principal-attached policies.
+ 5. Role permissions for `request.appId` + `request.spaceId`.
+ 6. No match → implicit deny.
- Common mistakes:
- - `request.headers.X-Foo` (wrong) → `request.header.X-Foo` (correct)
- - `request.ip` (wrong) → `request.remoteIp` (correct)
- - `principal.id` (wrong) → `context.principal.id` (correct)
- - `request.pathVariable.id` (wrong) → `request.path.var.id` (correct)
+ Each policy tier evaluates matched policy-level conditions once, pools all statements, checks every deny before any allow, and stops at the first tier with a result. A deny only overrides allows in the same tier; a global allow prevents later principal or role denies from running.
- ## Step 4: Testing Policies Locally
+ ## Symptom guide
- These patterns mirror the real tests in `cosec-core/src/test/kotlin/me/ahoo/cosec/authorization/SimpleAuthorizationTest.kt`.
+ ### Unexpected 401
- ### Unit test a full authorization decision
+ - Check whether the request actually carried a bearer token and whether parsing produced an authenticated principal.
+ - Match `cosec.jwt.algorithm` and secret to the issuer; only HMAC256/384/512 are supported.
+ - Check expiration and the response reason (`Token Expired` versus `Token Invalid`).
+ - If revocation is enabled, verify the cache capability supplied a real Redis-backed `TokenStore`; otherwise the starter warns that logout is ineffective.
+ - If the token is valid but the request remains anonymous, inspect the active `SecurityContextParser` and transport filter.
- ```kotlin
- @Test
- fun `test policy evaluation`() {
- val globalPolicy = mockk<Policy> {
- every { id } returns "globalPolicy"
- every { condition } returns AllConditionMatcher.INSTANCE
- every { statements } returns listOf(
- StatementData(
- name = "PublicEndpoints",
- effect = Effect.ALLOW,
- action = PathActionMatcherFactory.INSTANCE
- .create(mapOf("pattern" to "/api/users/*").asConfiguration()),
- ),
- )
- }
- val policyRepository = mockk<PolicyRepository> {
- every { getGlobalPolicy() } returns Mono.just(listOf(globalPolicy))
- every { getPolicies(any()) } returns Mono.empty()
- }
- val permissionRepository = mockk<AppRolePermissionRepository> {
- every { getAppRolePermission(any(), any(), any()) } returns Mono.empty()
- }
+ ### Unexpected 403 or missing allow
- val authorization = SimpleAuthorization(policyRepository, permissionRepository)
- val request = mockk<Request> {
- every { path } returns "/api/users/123"
- every { method } returns "GET"
- }
- // relaxed: evaluation writes path variables and the verify context into the context
- val securityContext = mockk<SecurityContext>(relaxed = true) {
- every { principal.id } returns "user-123"
- }
+ - Read the debug/audit match. An explicit deny is a matched deny; an implicit deny means no rule in any tier matched.
+ - Confirm a global policy loaded and its policy-level condition matched.
+ - Check the action's HTTP method, path, trailing slash, and captured variables.
+ - Check condition parts and the actual principal roles/attributes/tenant.
+ - Remember that `inTenant.value` is `default`, `user`, or `platform`, not a tenant ID.
- authorization.authorize(request, securityContext)
- .test()
- .expectNext(AuthorizeResult.ALLOW)
- .verifyComplete()
- }
- ```
+ ### Request allowed unexpectedly
- Imports worth knowing: `me.ahoo.cosec.configuration.JsonConfiguration.Companion.asConfiguration` for building matchers from maps, `me.ahoo.cosec.policy.StatementData` / `me.ahoo.cosec.policy.action.PathActionMatcherFactory` for real (non-mock) matchers, and `reactor.kotlin.test.test` for stepping through the `Mono`.
+ - Check root bypass first.
+ - Find the first matched tier. A broad global allow short-circuits stricter principal and role rules.
+ - Verify the deny action and condition both match; array order cannot make an allow beat a same-tier deny.
+ - Confirm the intended policy was loaded rather than skipped or shadowed by a duplicate ID.
- ### Smoke-test a local policy file
+ ### Policy file has no effect
- `LocalPolicyLoader` takes a set of resource patterns and exposes loaded policies as a property; `DefaultPolicyEvaluator` is a stateless object that dry-runs every matcher of a policy against a mock request to surface configuration errors (it swallows rate-limit and regex-timeout errors):
+ - Confirm `cosec.authorization.local-policy.enabled=true` and the file matches `local-policy.locations`.
+ - Inspect startup errors: `LocalPolicyLoader` logs and skips malformed JSON, missing required fields, unknown matcher types, and constructor-time configuration errors rather than failing the application.
+ - Deferred matcher errors still load. For example, an unsupported `part` throws during authorization and the transport returns 500; reproduce it with `DefaultPolicyEvaluator` or the failing request.
+ - Local policies are deduplicated by ID after location expansion; assert that the expected ID and content loaded.
+ - `LocalPolicyLoader` does not serve authorization decisions directly. Enable `init-repository` or prepopulate `PolicyRepository`; use `force-refresh` only when overwriting repository state is intended.
- ```kotlin
- val policies = LocalPolicyLoader(setOf("classpath:cosec-policy/test-policy.json")).policies
- policies.forEach { DefaultPolicyEvaluator.evaluate(it) }
- ```
+ ### Matcher mismatch
- ### Test a specific matcher
+ - Path variables use `{id}` and conditions read `request.path.var.id`; `:id` and `request.pathVariable.id` are invalid.
+ - Path SpEL templates use `#{principal.id}`. SpEL condition expressions use root properties such as `context.principal.id`; they are different evaluation roots.
+ - Valid part prefixes are singular `request.header.`, `request.attributes.`, `request.path.var.`, and `context.principal.attributes.`. Unsupported parts throw `IllegalArgumentException`.
+ - Regex timeouts fail closed as 403. Simplify or bound the pattern instead of increasing exposure to ReDoS.
- ```kotlin
- @Test
- fun `test path action matcher`() {
- val matcher = PathActionMatcherFactory.INSTANCE
- .create(mapOf("pattern" to "/api/users/*").asConfiguration())
+ ### Rate limiting differs across instances
- val request = mockk<Request> {
- every { path } returns "/api/users/123"
- every { method } returns "GET"
- }
+ - `rateLimiter` and `groupedRateLimiter` are in-memory and per JVM.
+ - `redisRateLimiter` and `redisGroupedRateLimiter` are cluster-wide and require cache support plus `StringRedisTemplate`.
+ - Redis limiter outages fail open by default. Set the matcher's `strictFailure: true` only when availability loss should return 429.
+ - Check `cosec.limiter.key-prefix`, `windowSeconds`, grouping `part`, and whether multiple policies intentionally share an identical quota configuration.
- matcher.match(request, SimpleSecurityContext.anonymous()).assert().isTrue()
- }
- ```
+ ## Reproduce narrowly
- ## Step 5: Request Attributes for Debugging
+ Prefer the smallest existing test layer that crosses the failing boundary:
- When debugging, inspect the request attributes that CoSec sets:
+ - matcher behavior: create the real factory with `mapOf(...).asConfiguration()` and test `match`;
+ - policy parsing: load the exact resource with `LocalPolicyLoader`, assert its ID, then run `DefaultPolicyEvaluator.evaluate`;
+ - decision order: test `SimpleAuthorization` with the smallest repositories needed;
+ - transport mapping: use the WebFlux, WebMVC, or Gateway filter tests for 401/403/429 behavior;
+ - Redis behavior: run the cache integration test with Redis.
- - `COSEC_SECURITY_CONTEXT` — the parsed security context
- - `request.attributes.ipRegion` — IP geolocation (if `cosec-ip2region` is enabled)
+ In this repository, run a focused class first, for example:
- In a WebFlux handler:
- ```kotlin
- @GetMapping("/debug/whoami")
- fun whoami(exchange: ServerWebExchange): Mono<Map<String, Any?>> {
- val context = exchange.getAttribute<SecurityContext>(COSEC_SECURITY_CONTEXT)
- return Mono.just(mapOf(
- "principal" to context?.principal?.id,
- "authenticated" to context?.principal?.authenticated,
- "roles" to context?.principal?.roles,
- "tenant" to context?.tenant?.tenantId
- ))
- }
+ ```bash
+ ./gradlew :cosec-core:test --tests "me.ahoo.cosec.policy.DefaultPolicyEvaluatorTest"
```
- ## Quick Reference: Authorization Results
-
- | Result | `authorized` | Meaning |
- |--------|-------------|---------|
- | `ALLOW` | `true` | Explicitly allowed by a policy statement |
- | `EXPLICIT_DENY` | `false` | Explicitly denied by a DENY statement or blacklist |
- | `IMPLICIT_DENY` | `false` | No statement matched (default deny) |
- | `TOKEN_EXPIRED` | `false` | JWT token has expired |
- | `TOO_MANY_REQUESTS` | `false` | Rate limiter exceeded |
+ Use `$cosec-policy-author` to rewrite a confirmed-bad policy and `$cosec-integration` to change confirmed-bad Spring configuration. For diagnosis-only requests, report the root cause and evidence without modifying files.