git:20261014.62aa70e to git:20261014.7748a1f

71 added, 140 removed. Audit A to A.

---

name: ops-investigate

- description: "Investigate one environment of the system with a read-only credential: the alarms, the error rate, the latency, the worker outcomes and the work items that failed for good, the pool, the queue and its dead letter, the identity provider (the sign-in and its webhook), the cost against the budget, and the system's size, then report what is wrong and what to do next. Every read goes through the signals' own APIs (CloudWatch, X-Ray, the error tracker in the cloud; Prometheus, Jaeger, GlitchTip locally). Use when something looks off, when an alarm fires, or as the daily look. Never writes."

+ description: "Investigate one environment of the system with a read-only credential: the alarms, the error rate, the latency, the outcomes, gauges, queues, and log lines each namespace's signals file names, the pool, the cost against the budget, and the system's size, then report what is wrong and what to do next. Every read goes through the signals' own APIs (CloudWatch, X-Ray, the error tracker in the cloud; Prometheus, Jaeger, GlitchTip locally). Use when something looks off, when an alarm fires, or as the daily look. Never writes."

allowed-tools: Read, Grep, Glob, Bash(aws:*), Bash(curl:*), Bash(docker compose:*), Bash(uv run acme-ops size:*), Bash(sleep:*)

---

# ops-investigate

One environment, one window, one credential that can only read. The

skill looks at every signal the system emits and says what is wrong,

how big the system is, and which skill runs next. It changes

nothing.

Read `../_shared/ops-preamble.md`, a path from this skill's folder,

before the first step: the profiles, the account check, and the env

file are there.

## Input

`--env local|staging|production [--since 1h] [--request-id <id>] [--alarm <name>]`

`--env` is required; ask for it when missing. `--since` is the window,

a duration ending now, one hour by default. `--request-id` narrows the

look to one request across every signal. `--alarm` starts from one

alarm by name and reads the system's size before anything else.

`local` reads the compose stack's twins (Prometheus, Jaeger, GlitchTip,

`docker compose logs`) and needs no cloud. The skill is testable with

no account.

## Role and credential

`--env local` needs the compose stack with the `devx` profile up

(`make devx-up`) and the env file below. No cloud credential.

`--env staging` and `--env production` run under the investigate

profile of that environment, `acme-<env>-investigate`, checked with

`sts get-caller-identity` before any other command as the preamble

states. Refuse any profile wider than the investigate role. Every

`aws` command below carries `--profile acme-<env>-investigate`.

The env file `~/.config/acme/ops/<env>.env` gives the API's URL, the

`read` operator token, and the error tracker's URL, token, org, and

project. It holds no `write` token: the traffic generator's has a file

of its own, which this skill never sources. Never read the env file; a

command that needs a value sources it in the same command, as every

block below does. Never print a token. On a `401`

the token has expired: stop, and name the refresh the preamble gives.

## Procedure

The processes are `api` and `maintenance`, as

`deployment/README.md` lists them. Each is an ECS service of that

name in the cluster `acme-<env>`, with the log group

- `/acme/<env>/<process>`. The queue is `acme-<env>-webhooks` (the

- identity provider's deliveries, which the API checked at

- `/webhooks/identity` and queued), with a dead-letter queue named with

- `-dead` after it. The deliveries come in through `/acme/<env>/api`,

- and their handling is in `/acme/<env>/maintenance`.

+ `/acme/<env>/<process>`.

+ Before step 1, read every file in `../_shared/signals/`, a path from

+ this skill's folder. Its `README.md` says what each heading holds, and

+ each other file is one namespace or one process, the scaffold's and a

+ copy's alike: a copy adds `<ns>.md` there for each namespace of its

+ own. Step 3 reads their Alarms, step 5 their Outcomes, Gauges, and

+ Queues, step 7 their Log lines, Kinds, and Commands, step 8 their

+ Search, and the report their Report.

1. Verify the credential as Role and credential states. Compute the

window: `--since` back from now, as epoch seconds

for the cloud and as a Prometheus range for local.

2. Read the system's size first:

```bash

uv run acme-ops size --env <env>

```

When it refuses the env file for holding the provisioner's token,

stop, and give the person the line it printed: no later step runs,

and none sources the file. It prints tenants, users, and the change events

produced in the last day,

through `GET /v1/admin/size`

with the env file's operator token (`uv run acme-ops size --env <env>`,

which leaves the traffic generator's own tenants out). The size is

the maintenance worker's latest count, made every five minutes, and

the command prints how long ago it counted. A count older than ten

minutes says the worker's sweep has stopped counting, which is a

finding of its own; a `404` says no worker has counted in this

environment yet. A system of one tenant and one user is the

developer. Every finding below is read against this number and

against whose traffic it was.

3. Alarms. Cloud:

```bash

aws cloudwatch describe-alarms --alarm-name-prefix acme-<env>- \

--profile acme-<env>-investigate

```

Local has no alarm topic: run the alarm conditions as Prometheus

queries against `$ACME_PROMETHEUS_URL/api/v1/query`, over the

window `[<since>]`, each `curl` sourcing the env file in the same

command:

- 5xx ratio: `sum(rate(acme_http_requests_total{status=~"5.."}[<since>])) / sum(rate(acme_http_requests_total[<since>]))`, alarm above 0.01

- targets up: `up{job!="prometheus"}`, alarm on any 0 (a host process

and a container are two targets of one job; one of them is down

by design)

- p95 latency: `histogram_quantile(0.95, sum by (le) (rate(acme_http_request_seconds_bucket[<since>])))`, alarm above 1 s

- running processes: the `up` targets again, one per process

- the database's CPU and free storage: no local exporter; report

them as not read

- - work backlog: `max(acme_work_oldest_ready_seconds)`, alarm above

- 600 (the item ready longest waited ten minutes: no worker claims)

- - work dead letter: `max(acme_work_failed_recently)`, alarm at 1 or

- more (an item failed for good in the last fifteen minutes)

- - outbox lag: `max(acme_outbox_oldest_pending_seconds)`, alarm above

- 300 (the oldest row not yet relayed landed five minutes ago)

- - outbox dead letter: `max(acme_outbox_failed_recently)`, alarm at 1

- or more (a row failed for good in the last fifteen minutes)

+ - each signals file's Alarms, by the condition and the threshold it

+ gives

- The last four are the worker's sweep reads, one per pass; a local

- stack whose worker is not running has none, which is "not read".

+ A gauge is a read of the worker's sweep, one per pass; a local stack

+ whose worker is not running has none, which is "not read".

With `--alarm <name>`, start here and apply the first responder

rule of step 11 before reading anything else.

4. Request rate, error ratio, p95, by route. Cloud, the queries of the

dashboard `acme-<env>`, namespace `Acme`. The p95 is the load

balancer's, so read its ARN first:

```bash

aws elbv2 describe-load-balancers --names acme-<env> \

--query 'LoadBalancers[0].LoadBalancerArn' --output text \

--profile acme-<env>-investigate

```

`<load balancer>` below is what follows `loadbalancer/` in that

ARN, `app/acme-<env>/<id>`. One call then reads the three signals:

```bash

aws cloudwatch get-metric-data --profile acme-<env>-investigate \

--start-time <start> --end-time <end> --output text \

--query 'MetricDataResults[?length(Values) > `0`].[Id,Label,sum(Values),max(Values)]' \

--metric-data-queries '[

{"Id":"req","Period":60,"Expression":"SUM(SEARCH('"'"'{\"Acme\",OTelLib,environment,method,route,service,status} MetricName=\"acme_http_requests_total\" environment=\"<env>\"'"'"', '"'"'Sum'"'"', 60))"},

{"Id":"by","Period":60,"Label":"${PROP('"'"'Dim.status'"'"')} ${PROP('"'"'Dim.method'"'"')} ${PROP('"'"'Dim.route'"'"')}","Expression":"SEARCH('"'"'{\"Acme\",OTelLib,environment,method,route,service,status} MetricName=\"acme_http_requests_total\" environment=\"<env>\"'"'"', '"'"'Sum'"'"', 60)"},

{"Id":"p95","Label":"p95","MetricStat":{"Metric":{"Namespace":"AWS/ApplicationELB","MetricName":"TargetResponseTime","Dimensions":[{"Name":"LoadBalancer","Value":"<load balancer>"}]},"Period":60,"Stat":"p95"}}

```

It prints one line for each series with a datapoint in the window:

its id, its label, the sum of its minutes, and the highest of them.

- `req` is every request: its sum is the window's count, and the

rate is that count over the window's seconds.

- A `by` line is one status, method, and route. The error ratio is

the sum of the lines whose status starts with `5` over the sum of

`req`; one route's ratio is the same over that route's lines.

- `p95` is the load balancer's target response time over every

route, in seconds. The report's p95 is its highest minute times

1,000, in milliseconds, written once, as every route together;

with no `p95` line, no request crossed the load balancer, and the

report writes `p95 none`. The cloud has no p95 by route: the

app's histogram reaches CloudWatch as a statistic set, which

holds no percentile.

Every series carries all its labels as dimensions, and the exporter

adds `OTelLib`. CloudWatch matches dimensions exactly, so a

`MetricStat` naming only some of them finds nothing, and neither

does a `SEARCH` whose schema leaves one out: it answers with no

datapoint and no error. The schema is the dashboard's own

(`deployment/terraform/modules/dashboard/`). The load balancer's

health checks are requests, so a call that prints no `req` line is

"not read", never "no traffic".

Local:

```bash

set -a; . ~/.config/acme/ops/<env>.env; set +a

curl -sG "$ACME_PROMETHEUS_URL/api/v1/query" \

- --data-urlencode 'query=sum by (route, status) (rate(acme_http_requests_total[5m]))'

+ --data-urlencode 'query=sum by (route, status) (rate(acme_http_requests_total[<since>]))'

curl -sG "$ACME_PROMETHEUS_URL/api/v1/query" \

- --data-urlencode 'query=histogram_quantile(0.95, sum by (le, route) (rate(acme_http_request_seconds_bucket[5m])))'

+ --data-urlencode 'query=histogram_quantile(0.95, sum by (le, route) (rate(acme_http_request_seconds_bucket[<since>])))'

```

- 5. Workers, queue, pool, cache: one counter carries every outcome,

- `acme_outcomes_total{subsystem, outcome}` (subsystems `worker`,

- `work`, `outbox`, `queue`, `deliveries`, `cache`, `rate_limit`,

- `admission`, `idempotency`, `orchestrations`, `leases`), read as `sum by (subsystem, outcome)

- (increase(acme_outcomes_total[<since>]))`. A work item that failed

- for good counts `work`/`dead_letter`: its attempts ran out

- (`worker`/`failed` beside it), or its handler was refused, a failure

- no retry changes (`worker`/`refused`, a provider refusing the call

- itself). Each one is an `ERROR` line in `/acme/<env>/maintenance`,

- `work item <id> (<kind>) in org <org> failed for good: <reason>`,

- which step 7 finds; `work`/`requeued` counts the ones an operator

- sent back. The long-running records

- are `orchestrations`, one outcome per kind and state,

- `<kind>_<state>`: `_running` when a record starts, `_parked` when a

- step waits on a provider that did not answer, `_resumed`,

- `_succeeded`, `_failed`, and `_cancelled`, a person's end. The

- mechanism's own kind is `noop` (`noop_succeeded`); a product adds

- its kinds beside it. A parked

- record is woken by the event that clears its reason (`WAKE_PARKED`)

- or by a person; one that stays parked is read beside the provider's

- lines of step 8. A `_failed` is `defect`, a step that still failed

- on its item's last attempt, which the maintenance log names at level

- `ERROR` with the record's id and the org's (`<kind> <id> in org

- <org> failed: defect ...`), so step 7 finds it. A running record

- whose `_succeeded` never follows is a step the queue keeps retrying:

- read the worker's `failed` outcomes beside it. A lease on a leasable

- resource counts `leases`/`granted`, then `released`, `expired`, or

- `revoked`; a request that left its line without one counts

- `request_expired` or `request_<why>` (`asked`, `waiter_gone`,

- `refused`). The work queue and the outbox are Postgres tables, and

- each sweep pass reads four numbers of them across every tenant:

- `acme_work_oldest_ready_seconds` (how long the item ready longest has

- waited), `acme_work_failed_recently` (items failed in the last

- fifteen minutes and still failed),

- `acme_outbox_oldest_pending_seconds` (how long ago the oldest row not

- yet relayed landed), and `acme_outbox_failed_recently` (rows failed

- for good in the last fifteen minutes). Locally they are gauges on

- Prometheus. In the cloud they are metrics of the same names in `Acme`

- with no dimension, which the alarms `acme-<env>-work-backlog`,

- `-work-dead-letter`, `-outbox-lag`, and `-outbox-dead-letter` read,

- filtered from the worker's sweep line:

+ 5. Outcomes, gauges, queues, the pool: one counter carries every

+ outcome, `acme_outcomes_total{subsystem, outcome}`, read as

+ `sum by (subsystem, outcome) (increase(acme_outcomes_total[<since>]))`.

+ A signals file's Outcomes says what each outcome of its subsystems

+ means. Two subsystems are the infrastructure's, the report's Pool

+ and cache: `storage` counts the pool's `checkout_timeout` and

+ `statement_timeout`, and `cache` its `hit`, `miss`, and

+ `unreachable`.

+ The gauges are every signals file's Gauges. Locally they are gauges

+ on Prometheus, read as `max(<gauge>)`. In the cloud they are metrics

+ of the same names in `Acme` with no dimension, filtered from the

+ worker's sweep line, which the alarms read. One call reads them all,

+ with one query per gauge:

```bash

aws cloudwatch get-metric-data --profile acme-<env>-investigate \

--start-time <start> --end-time <end> \

- --metric-data-queries '[{"Id":"ready","MetricStat":{"Metric":{"Namespace":"Acme","MetricName":"acme_work_oldest_ready_seconds"},"Period":60,"Stat":"Maximum"}},{"Id":"failed","MetricStat":{"Metric":{"Namespace":"Acme","MetricName":"acme_work_failed_recently"},"Period":60,"Stat":"Maximum"}},{"Id":"outbox","MetricStat":{"Metric":{"Namespace":"Acme","MetricName":"acme_outbox_oldest_pending_seconds"},"Period":60,"Stat":"Maximum"}},{"Id":"dead","MetricStat":{"Metric":{"Namespace":"Acme","MetricName":"acme_outbox_failed_recently"},"Period":60,"Stat":"Maximum"}}]'

+ --metric-data-queries '[{"Id":"<id>","MetricStat":{"Metric":{"Namespace":"Acme","MetricName":"<gauge>"},"Period":60,"Stat":"Maximum"}}]'

```

+ The list holds one such query for each gauge, each with an `Id` of

+ its own: lowercase letters, digits, and underscores, starting with

+ a letter.

The outcomes and the pool are one more call in the cloud: the

counter by the dashboard's schema, the twin of the `sum by` above,

and the database's connection count:

```bash

aws cloudwatch get-metric-data --profile acme-<env>-investigate \

--start-time <start> --end-time <end> --output text \

--query 'MetricDataResults[?length(Values) > `0`].[Id,Label,sum(Values),max(Values)]' \

--metric-data-queries '[

{"Id":"out","Period":60,"Label":"${PROP('"'"'Dim.service'"'"')} ${PROP('"'"'Dim.subsystem'"'"')} ${PROP('"'"'Dim.outcome'"'"')}","Expression":"SEARCH('"'"'{\"Acme\",OTelLib,environment,outcome,service,subsystem} MetricName=\"acme_outcomes_total\" environment=\"<env>\"'"'"', '"'"'Sum'"'"', 60)"},

{"Id":"pool","Label":"connections","MetricStat":{"Metric":{"Namespace":"AWS/RDS","MetricName":"DatabaseConnections","Dimensions":[{"Name":"DBInstanceIdentifier","Value":"acme-<env>"}]},"Period":60,"Stat":"Maximum"}}

```

An `out` line is one process, subsystem, and outcome, and its sum

is the window's count. `pool` is read by its highest minute: the

most connections the database held.

- A backlog with the maintenance service at its desired count is a

- worker that claims nothing: read its log for `claim failed`. An

- outbox lag is a relay that keeps failing: its log names each attempt,

- `outbox relay of <row id> (<kind>) failed on attempt <n>: <error>`.

- An outbox dead letter is the last of those attempts: the log says

- `outbox row <row id> (<kind>) failed for good after <n> attempts:

- <error>`, and the org's diary holds an `outbox.row.failed` event that

- names the row's kind and target.

- A work item failed for good is step 7's line. The inbound queue's

- depth and oldest age, and pool checkouts, have no metric of the

- worker's; the cloud reads the pool from the database's connection

- count, as above, and the queue and its dead letter from SQS:

+ An outcome, a gauge, or an alarm is read as its signals file says,

+ with the log line it names as the cause, which step 7 or step 8

+ finds. A queue's depth and oldest age, and pool checkouts, have no

+ metric of the worker's; the cloud reads the pool from the database's

+ connection count, as above, and each queue of every signals file's

+ Queues, with its dead letter, from SQS:

```bash

- for q in webhooks webhooks-dead; do

+ for q in <queue> <queue>-dead; do

url="$(aws sqs get-queue-url --queue-name acme-<env>-$q \

--query QueueUrl --output text --profile acme-<env>-investigate)"

aws sqs get-queue-attributes --queue-url "$url" --profile acme-<env>-investigate \

--attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible

done

```

- A message in a `-dead` queue is a delivery the worker could not

- handle after its retries: a finding, with the queue's name. The

- worker counts each delivery it handles as `deliveries`/`<outcome>`

- (`applied`, `duplicate`, `unowned`, `malformed`, `failed`); a

- `failed` stays on the queue and comes back. Cloud

- also reads the running count against the desired count:

+ with every queue the signals files name in the list, each followed

+ by its `-dead`. A message in a `-dead` queue is one its consumer

+ could not handle after its retries: a finding, with the queue's

+ name. Cloud also reads the running count against the desired count:

```bash

aws ecs describe-services --cluster acme-<env> \

--services api maintenance \

--profile acme-<env>-investigate

```

6. Errors. There is one tracker project for the product, and every

environment reports into it, so the read names that project and

filters on the environment. The error tracker's REST API at

`$ACME_ERROR_TRACKER_URL` with the token as a bearer, the issues

last seen in the window in this environment, newest first, with

`local` as the environment locally:

```bash

set -a; . ~/.config/acme/ops/<env>.env; set +a

case "$ACME_ERROR_TRACKER_URL" in *sentry.io*) window=" lastSeen:-<since>" ;; *) window="" ;; esac

curl -s -H "Authorization: Bearer $ACME_ERROR_TRACKER_TOKEN" --get \

--data-urlencode "query=environment:<env>$window" \

"$ACME_ERROR_TRACKER_URL/api/0/projects/$ACME_ERROR_TRACKER_ORG/$ACME_ERROR_TRACKER_PROJECT/issues/"

```

The tracker decides how the window is asked, not `--env`. Sentry

takes it as the search term `lastSeen:-<since>`, in minutes, hours,

days, or weeks (`90m`, `1h`, `2d`, `1w`). GlitchTip, the local

stack's tracker or a deployed one, reads `lastSeen:` as a tag no

issue carries and would answer with no issue. So only Sentry's own

host gets the term, and any other tracker gets the query without

it. Either way, keep the issues whose `lastSeen` field is inside

the window. The window is never `statsPeriod`: that parameter sizes

each issue's graph, takes only `24h` and `14d`, and answers `400`

to any other value.

An issue in that project can hold events of more than one

environment, so an issue the query returned is not by itself this

environment's: read its events

(`/api/0/issues/<issue id>/events/`) and keep the ones whose

`environment` tag is `<env>`. Dropping the filter reports every

environment's errors as this one's, which is wrong, not wider.

The org's slug is what `GET /api/0/organizations/` lists (locally

`acme`):

```bash

set -a; . ~/.config/acme/ops/<env>.env; set +a

curl -s -H "Authorization: Bearer $ACME_ERROR_TRACKER_TOKEN" "$ACME_ERROR_TRACKER_URL/api/0/organizations/"

```

When the env file names no tracker, this step reads nothing and the

report says "not read", never "no errors"; the other steps stand on

their own, and an investigation is not stopped by it.

7. Logs. Cloud, one log group per process, `/acme/<env>/<process>`:

```bash

aws logs start-query --profile acme-<env>-investigate \

--log-group-names /acme/<env>/api /acme/<env>/maintenance \

--start-time <start> --end-time <end> \

--query-string 'fields @timestamp, level, request_id, @message | filter level = "ERROR" | sort @timestamp desc | limit 100'

aws logs get-query-results --query-id <id> --profile acme-<env>-investigate

```

Poll `get-query-results` at most 10 times for one query, each poll

after `sleep 5` in the same command, so ten polls cover about a

minute. When the status is still `Scheduled` or `Running` after the

tenth, stop polling: the report reads that query's logs as "not

read: the query did not finish in 10 polls", with the query id, and

the next step starts. Step 8's query is polled the same way.

Local: `docker compose -f deployment/local/docker-compose.yml -f

deployment/local/docker-compose.full.yml logs --since <since> api

maintenance` from the repository root when the processes

run in containers. When they run on the host (`scripts/dev.sh`

writes no file; it logs to its terminal), `grep` the file the

process was started with, and say "not read" when there is none.

A local line carries the request id in brackets, `[<id>]`, or as

`"request_id"` when `ACME_LOG_JSON` is on.

With `--request-id`, filter every source on it: `filter request_id

= "<id>"` in the cloud, `grep` locally.

- A failed work item is a finding with its id, its kind, its org, and

- its reason, all off the `failed for good` line. The reason says

- whether it is the provider's or a person's to fix; a

- `DELETE_ACCOUNT` one leaves a deleted person's personal org in

- place until it runs. Once the cause is fixed, a person sends it back

- with `uv run acme-ops work requeue --env <env> --org <org> <item>`,

- in their own terminal: it signs them in with the second factor and

- mints a `write` token for that one call. This skill names the

- command in the report and never runs it.

- 8. The identity provider. It says in its own log when it is not

- configured, and the skill reads that, never a secret: the secrets

- are denied to the role, and their names are enough. Cloud, over the

- same window:

+ A line a signals file's Log lines names is read as that file says:

+ the finding it is, with the ids it carries, and a kind on it as the

+ Kinds of the file that names the kind say. When the file's Commands

+ give a person's command for that finding, the report names it for a

+ person to run in their own terminal; this skill never runs it.

+ 8. The lines a read of `ERROR` misses: a start line, a warning, an

+ access line. Each signals file's Search is one pattern, and one

+ query joins them all with `|`. Cloud, over the same window:

```bash

aws logs start-query --profile acme-<env>-investigate \

--log-group-names /acme/<env>/api /acme/<env>/maintenance \

--start-time <start> --end-time <end> \

+ --query-string 'fields @timestamp, @log, @message | filter @message like /<every Search, joined with |>/ | sort @timestamp desc | limit 50'

```

- What each line means, and the secret it points to:

- - `identity provider: none (ACME_WORKOS_API_KEY is not set)` and

- `every sign-in through one answers 503`: nobody can sign in

- through WorkOS. `acme/<env>/workos_api_key` is `off`, or the

- API's tasks started before it was written.

- - `ACME_WORKOS_API_KEY is not the API key of the WorkOS application`,

- and the API's tasks stop at start: the secret holds a key WorkOS

- refuses as the Acme App's (the environment's API key, another

- application's, or the other WorkOS environment's). The key belongs

- on the Acme App's own API keys tab. `the WorkOS credential check

- did not finish` or `answered` is a warning only: WorkOS could not

- say, and the API started.

- - `ACME_WORKOS_WEBHOOK_SECRET is not set` on `POST

- /webhooks/identity`, a `503` there in the access lines of

- `/acme/<env>/api`: `acme/<env>/workos_webhook_secret` is `off`, or

- the API's tasks started before it was written. Every delivery is

- refused, and WorkOS sends it again later.

- - A `400` (`webhook_signature_invalid`) on `/webhooks/identity` in

- the access lines of `/acme/<env>/api`: the secret does not match

- the signing secret of the endpoint in the WorkOS dashboard, so

- every delivery is refused.

- - `identity delivery <id> names no org` or `names org <id>, which

- is gone` in `/acme/<env>/maintenance`: one delivery about an org

- this environment does not hold, dropped. Not a finding about a

- secret.

- The start lines are written once, when a task starts, so a window

- after the last rollout holds none: report "not in the window",

- never "configured". A finding here names the secret and

- `docs/runbooks/providers/workos.md`; writing the value is a

- person's step under their own sign-in, never this skill's. Locally,

- `grep` the same lines in each process's own output, as step 7 reads

- it; a laptop that sets no `ACME_WORKOS_API_KEY` signs people in by

- the local sign-in on purpose, so `identity provider: none` there is

- not a finding.

+ What each line means, and the secret it points to, is its signals

+ file's Log lines. The skill reads a secret's name, never its value:

+ the secrets are denied to the role, and their names are enough. A

+ line written once, when a task starts, is not in a window after the

+ last rollout: report "not in the window", never "configured".

+ Writing a secret's value is a person's step under their own

+ sign-in, never this skill's. Locally, `grep` the same patterns in

+ each process's own output, as step 7 reads it.

9. Traces. Cloud:

```bash

aws xray get-trace-summaries --profile acme-<env>-investigate \

--start-time <start> --end-time <end> \

--filter-expression 'service("acme-api") AND responsetime > 1'

```

Local, Jaeger's v3 API (the service is the process name, `api`, and

the request id is the span attribute `acme.request_id`):

```bash

set -a; . ~/.config/acme/ops/<env>.env; set +a

curl -sG "$ACME_JAEGER_URL/api/v3/traces" \

--data-urlencode query.service_name=api \

--data-urlencode "query.start_time_min=<start, RFC 3339>" \

--data-urlencode "query.start_time_max=<end, RFC 3339>" \

--data-urlencode query.duration_min=1s

```

and filter the spans on the attribute client-side; the query API

ignores attribute filters. An empty answer means the process ran

with no `ACME_OTEL_ENDPOINT`, which is a finding, not an error.

With `--request-id`, filter on the `acme_request_id` annotation in the

cloud and the `acme.request_id` attribute locally instead.

10. Cost, cloud only. The month to date against the budget:

```bash

aws ce get-cost-and-usage --profile acme-<env>-investigate \

--time-period Start=<first of month>,End=<today> \

--granularity MONTHLY --metrics UnblendedCost \

--filter '{"Tags":{"Key":"environment","Values":["<env>"]}}'

aws budgets describe-budgets --account-id <account> \

--profile acme-<env>-investigate

```

11. The first responder rule. In production nothing is suppressed:

a new production's one tenant is its first customer, so every

alarm and finding there is reported as a finding, with the request

ids that prove it. Outside production, an alarm or a finding is

read against the size of step 2, and it is suppressed only when

the traffic behind it is the team's own: one tenant and one user,

and that user is the team's. A suppression is never silent: it is

reported as suppressed, with the reason, the size, and the request

ids it rests on.

12. Write the report. Name the next skill: `ops-root-cause` with an

org id when one tenant's rows explain it, `ops-watch` when the

signal is still moving, `ops-infra-as-code` when the fix is a

- resource, and, for a failed work item whose cause is fixed, the

- requeue command of step 7 for a person to run. A session follows at

- most 2 hops of Next. The skill it starts with is hop zero; the

- report of the second hop still names its next skill, and the

- session stops there and reports.

+ resource, and, for a finding whose cause is fixed, the command its

+ signals file gives a person to run, as step 7 names it. A session

+ follows at most 2 hops of Next. The skill it starts with is hop

+ zero; the report of the second hop still names its next skill, and

+ the session stops there and reports.

## What it never does

- No write to the cloud: no `aws` verb that is not `get`, `describe`,

`list`, `start-query`, `get-query-results`, or `tail`.

- No secret value read or printed: the tokens stay in the env file,

which is sourced and never read, and

`aws secretsmanager get-secret-value` is denied to the role and

never attempted.

- No tenant data: the signals carry no tenant id, and this skill reads

no tenant's rows. That is `ops-root-cause`, for one named tenant.

- No `terraform apply`, no console clicks, no scaling by hand.

- No re-reading a wider window than asked; a longer look is a second

run with a longer `--since`.

- No unbounded poll: never more than 10 polls of a query.

## Output

```markdown

# Investigation: <env>, last <since>

**Credential.** <profile and the Arn it resolved to, or local>

**Size.** <tenants> tenants, <users> users, <n> change events in the last day, counted <age> ago

## Alarms

- <alarm name>: <state since when>, <suppressed: reason | escalated>

## Signals

- Requests: <rate>, error ratio <ratio>, p95 <ms> by route (cloud: one p95 <ms, or none>, every route together)

- - Workers: <outcomes per kind>, oldest ready item <age>, failed in the last fifteen minutes <n>, oldest pending outbox row <age>

- - Failed work items: <item id, kind, org id, reason; or none>

- - Orchestrations: <kind> <started, parked, succeeded, failed, cancelled> per kind, defects <record and org ids, or none>

- - Queue: webhooks <n> (dead <n>), deliveries <outcomes>; services api, maintenance <running>/<desired>

- - Pool and cache: <checkouts, timeouts, hits, misses>

+ - Services: api, maintenance <running>/<desired> (cloud only)

+ - Pool and cache: <connections, checkout and statement timeouts, hits, misses, unreachable>

+ - <each Report line of every signals file, in the order of the files' names>

- Errors: <count>, top issue <title> (<request id, or none>), or "not

read: the environment names no error tracker"

- Cost: <month to date> of <budget> USD (cloud only)

## Findings

- <what is wrong, one sentence, with the request id that proves it>

## Next

```