--- name: ops-investigate description: "Investigate one environment of the platform with a read-only credential: the alarms, the error rate, the latency, the worker outcomes and the work items that failed for good, the pool, the queue and its dead letter, the identity provider (the sign-in and its webhook), the cost against the budget, and the platform's size, then report what is wrong and what to do next. Every read goes through the signals' own APIs (CloudWatch, X-Ray, the error tracker in the cloud; Prometheus, Jaeger, GlitchTip locally). Use when something looks off, when an alarm fires, or as the daily look. Never writes." allowed-tools: Read, Grep, Glob, Bash(aws:*), Bash(curl:*), Bash(docker compose:*), Bash(uv run acme-ops size:*), Bash(sleep:*) --- # ops-investigate One environment, one window, one credential that can only read. The skill looks at every signal the platform emits and says what is wrong, how big the platform is, and which skill runs next. It changes nothing. Read `../_shared/ops-preamble.md`, a path from this skill's folder, before the first step: the profiles, the account check, and the env file are there. ## Input `--env local|staging|production [--since 1h] [--request-id ] [--alarm ]` `--env` is required; ask for it when missing. `--since` is the window, a duration ending now, one hour by default. `--request-id` narrows the look to one request across every signal. `--alarm` starts from one alarm by name and reads the platform's size before anything else. `local` reads the compose stack's twins (Prometheus, Jaeger, GlitchTip, `docker compose logs`) and needs no cloud. The skill is testable with no account. ## Role and credential `--env local` needs the compose stack with the `devx` profile up (`make devx-up`) and the env file below. No cloud credential. `--env staging` and `--env production` run under the investigate profile of that environment, `acme--investigate`, checked with `sts get-caller-identity` before any other command as the preamble states. Refuse any profile wider than the investigate role. Every `aws` command below carries `--profile acme--investigate`. The env file `~/.config/acme/ops/.env` gives the API's URL, the `read` operator token, and the error tracker's URL, token, org, and project. It holds no `write` token: the traffic generator's has a file of its own, which this skill never sources. Never read the env file; a command that needs a value sources it in the same command, as every block below does. Never print a token. On a `401` the token has expired: stop, and name the refresh the preamble gives. ## Procedure The processes are `api` and `maintenance`, as `deployment/README.md` lists them. Each is an ECS service of that name in the cluster `acme-`, with the log group `/acme//`. The queue is `acme--webhooks` (the identity provider's deliveries, which the API checked at `/webhooks/identity` and queued), with a dead-letter queue named with `-dead` after it. The deliveries come in through `/acme//api`, and their handling is in `/acme//maintenance`. 1. Verify the credential as Role and credential states. Compute the window: `--since` back from now, as epoch seconds for the cloud and as a Prometheus range for local. 2. Read the platform's size first: ```bash uv run acme-ops size --env ``` When it refuses the env file for holding the provisioner's token, stop, and give the person the line it printed: no later step runs, and none sources the file. It prints tenants, users, and the events produced in the last day, through `GET /v1/admin/size` with the env file's operator token (`uv run acme-ops size --env `, which leaves the traffic generator's own tenants out). The size is the maintenance worker's latest count, made every five minutes, and the command prints how long ago it counted. A count older than ten minutes says the worker's sweep has stopped counting, which is a finding of its own; a `404` says no worker has counted in this environment yet. A platform of one tenant and one user is the developer. Every finding below is read against this number and against whose traffic it was. 3. Alarms. Cloud: ```bash aws cloudwatch describe-alarms --alarm-name-prefix acme-- \ --profile acme--investigate ``` Local has no alarm topic: run the alarm conditions as Prometheus queries against `$ACME_PROMETHEUS_URL/api/v1/query`, over the window `[]`, each `curl` sourcing the env file in the same command: - 5xx ratio: `sum(rate(acme_http_requests_total{status=~"5.."}[])) / sum(rate(acme_http_requests_total[]))`, alarm above 0.01 - targets up: `up{job!="prometheus"}`, alarm on any 0 (a host process and a container are two targets of one job; one of them is down by design) - p95 latency: `histogram_quantile(0.95, sum by (le) (rate(acme_http_request_seconds_bucket[])))`, alarm above 1 s - running processes: the `up` targets again, one per process - the database's CPU and free storage: no local exporter; report them as not read - work backlog: `max(acme_work_oldest_ready_seconds)`, alarm above 600 (the item ready longest waited ten minutes: no worker claims) - work dead letter: `max(acme_work_failed_recently)`, alarm at 1 or more (an item failed for good in the last fifteen minutes) - outbox lag: `max(acme_outbox_oldest_pending_seconds)`, alarm above 300 (the oldest row not yet relayed landed five minutes ago) - outbox dead letter: `max(acme_outbox_failed_recently)`, alarm at 1 or more (a row failed for good in the last fifteen minutes) The last four are the worker's sweep reads, one per pass; a local stack whose worker is not running has none, which is "not read". With `--alarm `, start here and apply the first responder rule of step 11 before reading anything else. 4. Request rate, error ratio, p95, by route. Cloud, the queries of the dashboard `acme-`, namespace `Acme`. The p95 is the load balancer's, so read its ARN first: ```bash aws elbv2 describe-load-balancers --names acme- \ --query 'LoadBalancers[0].LoadBalancerArn' --output text \ --profile acme--investigate ``` `` below is what follows `loadbalancer/` in that ARN, `app/acme-/`. One call then reads the three signals: ```bash aws cloudwatch get-metric-data --profile acme--investigate \ --start-time --end-time --output text \ --query 'MetricDataResults[?length(Values) > `0`].[Id,Label,sum(Values),max(Values)]' \ --metric-data-queries '[ {"Id":"req","Period":60,"Expression":"SUM(SEARCH('"'"'{\"Acme\",OTelLib,environment,method,route,service,status} MetricName=\"acme_http_requests_total\" environment=\"\"'"'"', '"'"'Sum'"'"', 60))"}, {"Id":"by","Period":60,"Label":"${PROP('"'"'Dim.status'"'"')} ${PROP('"'"'Dim.method'"'"')} ${PROP('"'"'Dim.route'"'"')}","Expression":"SEARCH('"'"'{\"Acme\",OTelLib,environment,method,route,service,status} MetricName=\"acme_http_requests_total\" environment=\"\"'"'"', '"'"'Sum'"'"', 60)"}, {"Id":"p95","Label":"p95","MetricStat":{"Metric":{"Namespace":"AWS/ApplicationELB","MetricName":"TargetResponseTime","Dimensions":[{"Name":"LoadBalancer","Value":""}]},"Period":60,"Stat":"p95"}} ]' ``` It prints one line for each series with a datapoint in the window: its id, its label, the sum of its minutes, and the highest of them. - `req` is every request: its sum is the window's count, and the rate is that count over the window's seconds. - A `by` line is one status, method, and route. The error ratio is the sum of the lines whose status starts with `5` over the sum of `req`; one route's ratio is the same over that route's lines. - `p95` is the load balancer's target response time over every route, in seconds. The report's p95 is its highest minute times 1,000, in milliseconds, written once, as every route together; with no `p95` line, no request crossed the load balancer, and the report writes `p95 none`. The cloud has no p95 by route: the app's histogram reaches CloudWatch as a statistic set, which holds no percentile. Every series carries all its labels as dimensions, and the exporter adds `OTelLib`. CloudWatch matches dimensions exactly, so a `MetricStat` naming only some of them finds nothing, and neither does a `SEARCH` whose schema leaves one out: it answers with no datapoint and no error. The schema is the dashboard's own (`deployment/terraform/modules/dashboard/`). The load balancer's health checks are requests, so a call that prints no `req` line is "not read", never "no traffic". Local: ```bash set -a; . ~/.config/acme/ops/.env; set +a curl -sG "$ACME_PROMETHEUS_URL/api/v1/query" \ --data-urlencode 'query=sum by (route, status) (rate(acme_http_requests_total[5m]))' curl -sG "$ACME_PROMETHEUS_URL/api/v1/query" \ --data-urlencode 'query=histogram_quantile(0.95, sum by (le, route) (rate(acme_http_request_seconds_bucket[5m])))' ``` 5. Workers, queue, pool, cache: one counter carries every outcome, `acme_outcomes_total{subsystem, outcome}` (subsystems `worker`, `work`, `outbox`, `queue`, `deliveries`, `cache`, `rate_limit`, `admission`, `idempotency`, `orchestrations`), read as `sum by (subsystem, outcome) (increase(acme_outcomes_total[]))`. A work item that failed for good counts `work`/`dead_letter`: its attempts ran out (`worker`/`failed` beside it), or its handler was refused, a failure no retry changes (`worker`/`refused`, a provider refusing the call itself). Each one is an `ERROR` line in `/acme//maintenance`, `work item () in org failed for good: `, which step 7 finds; `work`/`requeued` counts the ones an operator sent back. The long-running records are `orchestrations`, one outcome per kind and state, `_`: `_running` when a record starts, `_parked` when a step waits on a provider that did not answer, `_resumed`, `_succeeded`, and `_failed`. The mechanism's own kind is `noop` (`noop_succeeded`); a product adds its kinds beside it. A parked record is woken by the event that clears its reason (`WAKE_PARKED`) or by a person; one that stays parked is read beside the provider's lines of step 8. A `_failed` is `defect`, a step that still failed on its item's last attempt, which the maintenance log names at level `ERROR` with the record's id and the org's (` in org failed: defect ...`), so step 7 finds it. A running record whose `_succeeded` never follows is a step the queue keeps retrying: read the worker's `failed` outcomes beside it. The work queue and the outbox are Postgres tables, and each sweep pass reads four numbers of them across every tenant: `acme_work_oldest_ready_seconds` (how long the item ready longest has waited), `acme_work_failed_recently` (items failed in the last fifteen minutes and still failed), `acme_outbox_oldest_pending_seconds` (how long ago the oldest row not yet relayed landed), and `acme_outbox_failed_recently` (rows failed for good in the last fifteen minutes). Locally they are gauges on Prometheus. In the cloud they are metrics of the same names in `Acme` with no dimension, which the alarms `acme--work-backlog`, `-work-dead-letter`, `-outbox-lag`, and `-outbox-dead-letter` read, filtered from the worker's sweep line: ```bash aws cloudwatch get-metric-data --profile acme--investigate \ --start-time --end-time \ --metric-data-queries '[{"Id":"ready","MetricStat":{"Metric":{"Namespace":"Acme","MetricName":"acme_work_oldest_ready_seconds"},"Period":60,"Stat":"Maximum"}},{"Id":"failed","MetricStat":{"Metric":{"Namespace":"Acme","MetricName":"acme_work_failed_recently"},"Period":60,"Stat":"Maximum"}},{"Id":"outbox","MetricStat":{"Metric":{"Namespace":"Acme","MetricName":"acme_outbox_oldest_pending_seconds"},"Period":60,"Stat":"Maximum"}},{"Id":"dead","MetricStat":{"Metric":{"Namespace":"Acme","MetricName":"acme_outbox_failed_recently"},"Period":60,"Stat":"Maximum"}}]' ``` The outcomes and the pool are one more call in the cloud: the counter by the dashboard's schema, the twin of the `sum by` above, and the database's connection count: ```bash aws cloudwatch get-metric-data --profile acme--investigate \ --start-time --end-time --output text \ --query 'MetricDataResults[?length(Values) > `0`].[Id,Label,sum(Values),max(Values)]' \ --metric-data-queries '[ {"Id":"out","Period":60,"Label":"${PROP('"'"'Dim.service'"'"')} ${PROP('"'"'Dim.subsystem'"'"')} ${PROP('"'"'Dim.outcome'"'"')}","Expression":"SEARCH('"'"'{\"Acme\",OTelLib,environment,outcome,service,subsystem} MetricName=\"acme_outcomes_total\" environment=\"\"'"'"', '"'"'Sum'"'"', 60)"}, {"Id":"pool","Label":"connections","MetricStat":{"Metric":{"Namespace":"AWS/RDS","MetricName":"DatabaseConnections","Dimensions":[{"Name":"DBInstanceIdentifier","Value":"acme-"}]},"Period":60,"Stat":"Maximum"}} ]' ``` An `out` line is one process, subsystem, and outcome, and its sum is the window's count. `pool` is read by its highest minute: the most connections the database held. A backlog with the maintenance service at its desired count is a worker that claims nothing: read its log for `claim failed`. An outbox lag is a relay that keeps failing: its log names each attempt, `outbox relay of () failed on attempt : `. An outbox dead letter is the last of those attempts: the log says `outbox row () failed for good after attempts: `, and the org's diary holds an `outbox.row.failed` event that names the row's kind and target. A work item failed for good is step 7's line. The inbound queue's depth and oldest age, and pool checkouts, have no metric of the worker's; the cloud reads the pool from the database's connection count, as above, and the queue and its dead letter from SQS: ```bash for q in webhooks webhooks-dead; do url="$(aws sqs get-queue-url --queue-name acme--$q \ --query QueueUrl --output text --profile acme--investigate)" aws sqs get-queue-attributes --queue-url "$url" --profile acme--investigate \ --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible done ``` A message in a `-dead` queue is a delivery the worker could not handle after its retries: a finding, with the queue's name. The worker counts each delivery it handles as `deliveries`/`` (`applied`, `duplicate`, `unowned`, `malformed`, `failed`); a `failed` stays on the queue and comes back. Cloud also reads the running count against the desired count: ```bash aws ecs describe-services --cluster acme- \ --services api maintenance \ --profile acme--investigate ``` 6. Errors. There is one tracker project for the product, and every environment reports into it, so the read names that project and filters on the environment. The error tracker's REST API at `$ACME_ERROR_TRACKER_URL` with the token as a bearer, the issues last seen in the window in this environment, newest first, with `local` as the environment locally: ```bash set -a; . ~/.config/acme/ops/.env; set +a case "$ACME_ERROR_TRACKER_URL" in *sentry.io*) window=" lastSeen:-" ;; *) window="" ;; esac curl -s -H "Authorization: Bearer $ACME_ERROR_TRACKER_TOKEN" --get \ --data-urlencode "query=environment:$window" \ "$ACME_ERROR_TRACKER_URL/api/0/projects/$ACME_ERROR_TRACKER_ORG/$ACME_ERROR_TRACKER_PROJECT/issues/" ``` The tracker decides how the window is asked, not `--env`. Sentry takes it as the search term `lastSeen:-`, in minutes, hours, days, or weeks (`90m`, `1h`, `2d`, `1w`). GlitchTip, the local stack's tracker or a deployed one, reads `lastSeen:` as a tag no issue carries and would answer with no issue. So only Sentry's own host gets the term, and any other tracker gets the query without it. Either way, keep the issues whose `lastSeen` field is inside the window. The window is never `statsPeriod`: that parameter sizes each issue's graph, takes only `24h` and `14d`, and answers `400` to any other value. An issue in that project can hold events of more than one environment, so an issue the query returned is not by itself this environment's: read its events (`/api/0/issues//events/`) and keep the ones whose `environment` tag is ``. Dropping the filter reports every environment's errors as this one's, which is wrong, not wider. The org's slug is what `GET /api/0/organizations/` lists (locally `acme`): ```bash set -a; . ~/.config/acme/ops/.env; set +a curl -s -H "Authorization: Bearer $ACME_ERROR_TRACKER_TOKEN" "$ACME_ERROR_TRACKER_URL/api/0/organizations/" ``` When the env file names no tracker, this step reads nothing and the report says "not read", never "no errors"; the other steps stand on their own, and an investigation is not stopped by it. 7. Logs. Cloud, one log group per process, `/acme//`: ```bash aws logs start-query --profile acme--investigate \ --log-group-names /acme//api /acme//maintenance \ --start-time --end-time \ --query-string 'fields @timestamp, level, request_id, @message | filter level = "ERROR" | sort @timestamp desc | limit 100' aws logs get-query-results --query-id --profile acme--investigate ``` Poll `get-query-results` at most 10 times for one query, each poll after `sleep 5` in the same command, so ten polls cover about a minute. When the status is still `Scheduled` or `Running` after the tenth, stop polling: the report reads that query's logs as "not read: the query did not finish in 10 polls", with the query id, and the next step starts. Step 8's query is polled the same way. Local: `docker compose -f deployment/local/docker-compose.yml -f deployment/local/docker-compose.full.yml logs --since api maintenance` from the repository root when the processes run in containers. When they run on the host (`scripts/dev.sh` writes no file; it logs to its terminal), `grep` the file the process was started with, and say "not read" when there is none. A local line carries the request id in brackets, `[]`, or as `"request_id"` when `ACME_LOG_JSON` is on. With `--request-id`, filter every source on it: `filter request_id = ""` in the cloud, `grep` locally. A failed work item is a finding with its id, its kind, its org, and its reason, all off the `failed for good` line. The reason says whether it is the provider's or a person's to fix; a `DELETE_ACCOUNT` one leaves a deleted person's personal org in place until it runs. Once the cause is fixed, a person sends it back with `uv run acme-ops work requeue --env --org `, in their own terminal: it signs them in with the second factor and mints a `write` token for that one call. This skill names the command in the report and never runs it. 8. The identity provider. It says in its own log when it is not configured, and the skill reads that, never a secret: the secrets are denied to the role, and their names are enough. Cloud, over the same window: ```bash aws logs start-query --profile acme--investigate \ --log-group-names /acme//api /acme//maintenance \ --start-time --end-time \ --query-string 'fields @timestamp, @log, @message | filter @message like /identity provider|WorkOS application|WorkOS credential check|ACME_WORKOS_WEBHOOK_SECRET|webhooks\/identity (400|503)|identity delivery/ | sort @timestamp desc | limit 50' ``` What each line means, and the secret it points to: - `identity provider: none (ACME_WORKOS_API_KEY is not set)` and `every sign-in through one answers 503`: nobody can sign in through WorkOS. `acme//workos_api_key` is `off`, or the API's tasks started before it was written. - `ACME_WORKOS_API_KEY is not the API key of the WorkOS application`, and the API's tasks stop at start: the secret holds a key WorkOS refuses as the Acme App's (the environment's API key, another application's, or the other WorkOS environment's). The key belongs on the Acme App's own API keys tab. `the WorkOS credential check did not finish` or `answered` is a warning only: WorkOS could not say, and the API started. - `ACME_WORKOS_WEBHOOK_SECRET is not set` on `POST /webhooks/identity`, a `503` there in the access lines of `/acme//api`: `acme//workos_webhook_secret` is `off`, or the API's tasks started before it was written. Every delivery is refused, and WorkOS sends it again later. - A `400` (`webhook_signature_invalid`) on `/webhooks/identity` in the access lines of `/acme//api`: the secret does not match the signing secret of the endpoint in the WorkOS dashboard, so every delivery is refused. - `identity delivery names no org` or `names org , which is gone` in `/acme//maintenance`: one delivery about an org this environment does not hold, dropped. Not a finding about a secret. The start lines are written once, when a task starts, so a window after the last rollout holds none: report "not in the window", never "configured". A finding here names the secret and `docs/runbooks/providers/workos.md`; writing the value is a person's step under their own sign-in, never this skill's. Locally, `grep` the same lines in each process's own output, as step 7 reads it; a laptop that sets no `ACME_WORKOS_API_KEY` signs people in by the local sign-in on purpose, so `identity provider: none` there is not a finding. 9. Traces. Cloud: ```bash aws xray get-trace-summaries --profile acme--investigate \ --start-time --end-time \ --filter-expression 'service("acme-api") AND responsetime > 1' ``` Local, Jaeger's v3 API (the service is the process name, `api`, and the request id is the span attribute `acme.request_id`): ```bash set -a; . ~/.config/acme/ops/.env; set +a curl -sG "$ACME_JAEGER_URL/api/v3/traces" \ --data-urlencode query.service_name=api \ --data-urlencode "query.start_time_min=" \ --data-urlencode "query.start_time_max=" \ --data-urlencode query.duration_min=1s ``` and filter the spans on the attribute client-side; the query API ignores attribute filters. An empty answer means the process ran with no `ACME_OTEL_ENDPOINT`, which is a finding, not an error. With `--request-id`, filter on the `acme_request_id` annotation in the cloud and the `acme.request_id` attribute locally instead. 10. Cost, cloud only. The month to date against the budget: ```bash aws ce get-cost-and-usage --profile acme--investigate \ --time-period Start=,End= \ --granularity MONTHLY --metrics UnblendedCost \ --filter '{"Tags":{"Key":"environment","Values":[""]}}' aws budgets describe-budgets --account-id \ --profile acme--investigate ``` 11. The first responder rule. In production nothing is suppressed: a new production's one tenant is its first customer, so every alarm and finding there is reported as a finding, with the request ids that prove it. Outside production, an alarm or a finding is read against the size of step 2, and it is suppressed only when the traffic behind it is the team's own: one tenant and one user, and that user is the team's. A suppression is never silent: it is reported as suppressed, with the reason, the size, and the request ids it rests on. 12. Write the report. Name the next skill: `ops-root-cause` with an org id when one tenant's rows explain it, `ops-watch` when the signal is still moving, `ops-infra-as-code` when the fix is a resource, and, for a failed work item whose cause is fixed, the requeue command of step 7 for a person to run. A session follows at most 2 hops of Next. The skill it starts with is hop zero; the report of the second hop still names its next skill, and the session stops there and reports. ## What it never does - No write to the cloud: no `aws` verb that is not `get`, `describe`, `list`, `start-query`, `get-query-results`, or `tail`. - No secret value read or printed: the tokens stay in the env file, which is sourced and never read, and `aws secretsmanager get-secret-value` is denied to the role and never attempted. - No tenant data: the signals carry no tenant id, and this skill reads no tenant's rows. That is `ops-root-cause`, for one named tenant. - No `terraform apply`, no console clicks, no scaling by hand. - No re-reading a wider window than asked; a longer look is a second run with a longer `--since`. - No unbounded poll: never more than 10 polls of a query. ## Output ```markdown # Investigation: , last **Credential.** **Size.** tenants, users, events in the last day, counted ago ## Alarms - : , ## Signals - Requests: , error ratio , p95 by route (cloud: one p95 , every route together) - Workers: , oldest ready item , failed in the last fifteen minutes , oldest pending outbox row - Failed work items: - Orchestrations: per kind, defects - Queue: webhooks (dead ), deliveries ; services api, maintenance / - Identity provider: sign-in /workos_api_key | not in the window>, webhook /workos_webhook_secret | signature refused: acme//workos_webhook_secret | not in the window> - Pool and cache: - Errors: , top issue (<request id, or none>), or "not read: the environment names no error tracker" - Cost: <month to date> of <budget> USD (cloud only) ## Findings - <what is wrong, one sentence, with the request id that proves it> ## Next <the skill to run next, with its arguments, or "nothing"> ```