setup-local-stack · diff
git:20260906.9319867 to git:20260907.4a987fd
116 added, 161 removed. Audit A to A.
---
name: setup-local-stack
description: Configure gcx against the local oddyssey Grafana stack and query its four signals (metrics, traces, logs, profiles) without touching the user's own gcx contexts. Owns the global configuration's "local" stack value. Use when the configured stack is "local", when querying the local stack (Grafana host port from the global configuration, default 3000), when configuring gcx locally, when a command needs the Tempo, Prometheus, Loki, or Pyroscope datasource UID. gcx is the mandatory query CLI for the stack - install it if missing (brew install gcx, or the official install script from github.com/grafana/gcx).
---
# gcx on the local oddyssey stack
- The local stack is a single otel-lgtm container: Grafana, OTLP (gRPC and
- HTTP), Pyroscope's ingest endpoint, and four datasources — Tempo,
- Prometheus, Loki, Pyroscope — behind the Grafana datasource proxy. The
- host ports come from the **global configuration** (defaults `3000` /
- `4317` / `4318` / `4040`): read the effective URLs from `odd_stack_up`'s
- result (`grafana_url`, `otlp_endpoint`) or `odd_config_get` — never
- assume the defaults, and point an application's
- `OTEL_EXPORTER_OTLP_ENDPOINT` at those values, never at a hardcoded
- port. Profiles are the one signal that does not arrive over OTLP:
- pyroscope-io-style SDKs push over Pyroscope's own HTTP protocol,
- straight to `http://localhost:<local.pyroscope_port>` (issue #224) —
- discover that port from `odd_config_get` the same way. Grafana serves its API **anonymously** here: no credentials are
- required, and the
- `admin`/`admin` entries in the context below are accepted but inert (kept
- only so the template also fits an auth-enabled Grafana).
-
- The stack holds no volume **by design** — a reset wipes everything, and
- the observation report is the only durable artifact. To configure the
- container, pass `env` to `odd_stack_up`/`odd_stack_reset`; env (like the
- embedded defaults, e.g. delta-metric ingestion) applies at container
- creation only, so a container predating the current oddyssey version
- keeps its old definition until its next reset. An applied env is
- persisted into the global configuration's `stack_config.local` and
- reapplied on every later recreation, so it survives resets without being
- repeated — except credential-named variables (headers, tokens, ...),
- which are applied but never persisted (the result's `env_not_persisted`
- names them): pass those again on every recreation. The catalog of the
- image's variables — per-component debug logs, backend tuning, OTLP
- forwarding to a remote backend, OBI, `GF_*` — is
- [`references/otel-lgtm-env.md`](references/otel-lgtm-env.md), aligned on
- the pinned image tag. For anything env cannot
- express (volumes, networks), the supported escape hatch is a manual
- `docker run` reusing the same name and ports — `status`/`up`/`down` keep
- working against it, but a **reset recreates the container from the
- embedded definition plus env**: hand-mounted volumes and networks do not
- survive it. Bring it up with the oddyssey MCP tools (`odd_stack_status`,
+ One otel-lgtm container: Grafana, OTLP (gRPC and HTTP), Pyroscope's
+ ingest port, and four datasources behind the Grafana datasource proxy.
+ Bring it up with the oddyssey MCP tools (`odd_stack_status`,
`odd_stack_up`) before configuring anything here.
- **Read by section.** The caller's preflight (`backend-configuration`'s `## Check`)
- writes the isolated context below and proves it, and hands its path
- over in the mission block; an agent holding that handoff skips
- `## Configure an isolated context` (regenerate the file only when
- `gcx config check` fails on it) and reads `## Datasources` and
- `## This stack is push-based` — the two sections a query needs.
-
## Configure an isolated context
- Never edit the user's own gcx configuration. Point `GCX_CONFIG` at a
- **stable per-session path** — stable so the file is written once and
- reused while the configured ports stay the same, isolated so the
- user's contexts stay untouched. Read the effective `grafana_url` from
- `odd_config_get` (or `odd_stack_up`'s result) **before** writing the
- file, and put that URL — not the default — on the `server:` line:
+ ```bash
+ python3 <this skill's directory>/scripts/gcx_local.py
+ ```
+ It takes no arguments but `--json`. It reads the host ports from the
+ global configuration — never assumed, never hardcoded — writes an **isolated** context at a stable path so the
+ user's own gcx contexts stay untouched, and proves it with `gcx config
+ check`. It prints the `export GCX_CONFIG=...` line to put in front of
+ every later gcx call, the four datasource UIDs, and the OTLP and
+ Pyroscope endpoints an instrumented service should target. `--json` for
+ the same, parseable.
+
+ Exit 0 means gcx reached the stack. Exit 1 says which step failed: gcx
+ missing (install it — `brew install gcx`, or the official script from
+ github.com/grafana/gcx), or the stack unreachable (`odd_stack_up`).
+
+ Re-run it after any port change. It rewrites the file whole on purpose:
+ gcx binds a stored credential to its destination, so patching the
+ `server:` line in place leaves the binding stale and gcx refuses it
+ before any network use.
+
+ ## Inventory the services, in one command
+
```bash
- export GCX_CONFIG="${TMPDIR:-/tmp}/oddyssey/gcx-local.yaml"
- mkdir -p "$(dirname "$GCX_CONFIG")"
- cat > "$GCX_CONFIG" << 'EOF'
- current-context: local
- contexts:
- local:
- grafana:
- server: http://localhost:3000 # the configured grafana_url (odd_config_get) - never assume this default
- user: admin
- password: admin
- org-id: 1
- default-prometheus-datasource: prometheus
- default-loki-datasource: loki
- default-tempo-datasource: tempo
- default-pyroscope-datasource: pyroscope
- EOF
- gcx config check
+ python3 <this skill's directory>/scripts/probe_services.py <svc> [<svc> ...] --since 30m
```
- **A port change invalidates the whole file — regenerate it, never
- edit it in place.** gcx binds each stored credential to a keychain
- entry keyed by the config source and destination: after the
- configured ports change (`odd_config_set`), patching the `server:`
- line leaves the credential bound to the old destination and gcx
- rejects it before any network use:
- `Configured credential "stack:local" field "grafana-password" was
- rejected before network use: the keychain reference does not match
- this config source, owner, field, and destination` — and the
- suggested re-authenticate is a dead end on this anonymous stack.
- Delete the file and rewrite it whole from the block above with the
- new `grafana_url`: the fresh inline credential creates a new binding,
- and `gcx config check` passes on the new port (verified in both
- directions, 3000 → 3001 → 3000).
+ One call answers what every observation asks before it drives anything:
+ per service, which of the four signals carries it and under which
+ identity (instances, environment, version, SDK), the operations its
+ traces name, the metrics it publishes, and the current reading of every
+ cumulative series — the baseline a later query subtracts from. Absence is
+ reported as loudly as presence: a signal that is genuinely missing is a
+ result, not a failed probe.
- The four `default-*-datasource` entries make the `-d` flag unnecessary; the
- commands below keep it for explicitness, but it can be dropped.
+ **All the services in one call** — the script probes them concurrently,
+ so one call for three is one call, and three calls are three:
- Each shell invocation starts fresh, so `export GCX_CONFIG=...` again in every
- command block (or prefix the command with it) — the file itself persists, so
- the write and `gcx config check` happen only once per session — unless
- the configured ports change, which invalidates the file (above) and
- forces a full rewrite. If gcx
- itself is missing, install it (`brew install gcx`, or the official install
- script from https://github.com/grafana/gcx); for this stack the block above is
- the whole setup.
+ ```bash
+ GCX_CONFIG=<the path gcx_local.py printed> \
+ python3 <this skill's directory>/scripts/probe_services.py svc-a svc-b svc-c --since 30m
+ ```
+ That is the whole surface, so `--help` has nothing to add and the file
+ has nothing to read: service names are positional and the only required
+ argument, `--since <window>` sets the lookback (default `30m`), `--json`
+ prints the same report parseable, `--no-baseline` skips the counter
+ readings. Nothing else.
+
+ **Do not write these queries by hand.** Service names and a window
+ determine every one of them, so there is no judgment to exercise: the
+ script runs them concurrently and returns a synthesis in about a second,
+ where the same work spelled out call by call costs minutes and buries
+ the answer in raw JSON.
+
+ Exit 0 means every probe ran; exit 2 means at least one failed, and the
+ line marked `!` names it.
+
## Datasources
| Signal | Backend | UID | Query with | Language |
| --- | --- | --- | --- | --- |
| Traces | Tempo | `tempo` | `gcx traces labels/query/get -d tempo` | TraceQL |
| Metrics | Prometheus | `prometheus` | `gcx metrics labels/series/metadata/query` | PromQL |
| Logs | Loki | `loki` | `gcx logs labels/series/query` | LogQL |
| Profiles | Pyroscope | `pyroscope` | `gcx profiles list-profile-types/labels/query -d pyroscope` | profile selector |
- This table holds for **gcx v1.0.0 or newer** (verified against gcx
- v1.0.0 with Grafana 13.1.3, then gcx v1.2.0 with Grafana 13.2.0, all
- four signals round-tripped) — package managers may ship older builds,
- so check `gcx --version` before trusting it. The gcx command surface
- moves between versions, so when a documented command errors, trust
- `gcx <group> --help` over this table.
+ Verified against gcx v1.0.0 and v1.2.0, Grafana 13.1.3 and 13.2.0, all
+ four signals round-tripped. The gcx command surface moves between
+ versions: when a documented command errors, trust `gcx <group> --help`
+ over this table.
- gcx is the stack's mandatory **query** CLI — explore and measure
- through it. Recording raw datasource-proxy HTTP in a report
- (`/api/datasources/proxy/uid/<uid>/...` URLs) is
- the right form for **replayable evidence**: a proxy URL replays
- verbatim with curl alone, on a machine with no gcx context. Query with
- gcx; record a proof query in whichever form the verify run can replay
- exactly — both are contract-conform, and the report says which was
- used.
+ Discovery across the four signals is what `probe_services.py` above
+ already ran — read its output rather than re-deriving it. Past it, when a
+ finding needs a query the probe does not cover (`gcx metrics labels` /
+ `metadata`, `gcx traces labels`, a batch of `gcx traces get` after a
+ search), those calls are independent of each other: **run them
+ concurrently in one shell call**, each backgrounded with its own output
+ file and one `wait` per job, never serially and never one tool call each.
+ `gcx metrics series` and `gcx logs series` are not discovery commands —
+ bare, they error; both need at least one selector.
- Profiles carry no instance identity: a pyroscope-io profile's labels
- are `service_name`, `deployment_environment`, `otel.scope.name`,
- `otel.scope.version`, `process.runtime.name`, `process.runtime.version`
- — no `service.instance.id` equivalent (verified 2026-09-03, gcx 1.2.0:
- `gcx profiles exemplars profile '{service_name="<svc>"}' -d pyroscope
- --profile-type process_cpu:cpu:nanoseconds:cpu:nanoseconds --since 24h
- --jq '.exemplars[0].labels'`), and the store-wide `profiles labels`
- list mixes in Pyroscope's own self-profile labels (`hostname`,
- `pyroscope_spy`, `service_git_ref`, `service_repository`, `target`).
- Launch a driven service with a per-run profiler tag mirroring the
- OTel attribute (`service_instance_id=<run slug>`, through the profiler
- SDK's own tag mechanism — `tags` in the pyroscope-io SDKs), and
- qualify profile selectors by it; absent one, `process.runtime.version`
- plus application frames are the substitute, stated as such.
+ ## This stack is push-based
- Tempo's metrics-generator series carry no instance identity either.
- Their labels are the generator's intrinsic ones — `service`,
- `span_name`, `span_kind`, `status_code` on `traces_spanmetrics_*`,
- `client`, `server`, `connection_type` on `traces_service_graph_*`, plus
- `le` on each family's histogram — none of which is a
- `service.instance.id` equivalent, and `__metrics_gen_instance` names
- the generator itself, not the observed process (verified 2026-09-06,
- gcx 1.2.0: `gcx metrics labels --metric <metric>`, the same command per
- metric). Two processes sharing one `service.name` — the run under
- observation plus a stale listener, a compose container — therefore fold
- into a single series, whose counts are the stack's total and never one
- run's. Read these series for shape and topology; take a run's own
- counts and latencies from the service's own OTel histogram, qualified
- by `service_instance_id`
- (`http_server_request_duration_seconds_count{service_instance_id="<run slug>"}`,
- or whatever discovery returns). The generator's extra `dimensions` list
- is the lever that would add an instance label, and it is inert here:
- this image sets it to names no span carries, so it contributes none of
- the labels above.
+ **It is push-based.** Nothing is scraped, so `up{job="<service>"}` is
+ empty for every service, healthy or not — it proves nothing. Prove a
+ service is present with its own data: a Tempo search for
+ `{resource.service.name="<svc>"}`, a Prometheus series carrying it
+ (`target_info{service_name="<svc>"}`), a Loki stream selecting it.
- Discover before you query: `gcx metrics labels` / `gcx metrics metadata`,
- `gcx traces labels`, `gcx logs labels`, `gcx profiles list-profile-types`.
- The four signals' discoveries are independent — **run them
- concurrently inside one shell call** (each command backgrounded with
- its PID captured, its stdout to its own file and its stderr to
- another, one `wait "$pid"` per job), never serially and never
- one tool call each; the same holds for the batch of `gcx traces get`
- fetches once a search has returned its trace IDs. gcx is safe to run
- that way against one context (the Grafana reference's `## Query by
- signal` says so).
- `gcx metrics series` and `gcx logs series` are NOT discovery commands:
- bare, they error — both require at least one selector (e.g. `gcx metrics
- series 'target_info'`). Every service names its own telemetry — never
- assume a metric, label, or stream exists.
+ **Profiles carry no instance identity.** Their labels are
+ `service_name`, `deployment_environment`, `otel.scope.*`,
+ `process.runtime.*` — no `service.instance.id` equivalent, so two
+ processes sharing a service name merge into one flamegraph. Launch a
+ driven service with a per-run profiler tag mirroring the OTel attribute
+ (`service_instance_id=<run slug>`, through the SDK's own `tags`), and
+ qualify profile selectors by it; without one, say so rather than
+ implying the profile is the run's.
- ## This stack is push-based
+ **Tempo's metrics-generator series carry none either.** Their labels are
+ the generator's own (`service`, `span_name`, `span_kind`, `status_code`,
+ `client`, `server`, `le`); `__metrics_gen_instance` names the generator,
+ not the observed process. Read them for shape and topology; take a run's
+ counts and latencies from the service's own OTel histogram qualified by
+ `service_instance_id`.
- Apps push OTLP into the stack; Prometheus scrapes nothing here. So
- `up{job="<service>"}` is **empty for every service, healthy or not** — it
- proves nothing, and any workflow that gates on a scrape-style liveness
- check must skip it. Prove a service is present with its own data instead: a
- Tempo search for `{resource.service.name="<svc>"}`, a Prometheus series
- carrying it (`target_info{service_name="<svc>"}` or whatever discovery
- returns), a Loki stream selecting it.
+ **And it is not Grafana Cloud**: `gcx assistant` and investigations are
+ not served here.
- Also absent from this stack: `gcx assistant` and investigations — Grafana
- Cloud features the local anonymous instance does not serve.
+ ## Configuring the container
+
+ The image is configured exclusively through environment variables passed
+ to `odd_stack_up`/`odd_stack_reset`, and they apply **at container
+ creation only** — a container predating the current oddyssey version
+ keeps its definition until its next reset. An applied env is persisted
+ into `stack_config.local` and reapplied on every later recreation, so it
+ survives resets without being repeated; credential-named variables are
+ applied but never persisted, and must be passed again each time. The
+ image's full variable surface is
+ [`references/otel-lgtm-env.md`](references/otel-lgtm-env.md), aligned on
+ the pinned tag.
+
+ The stack holds **no volume by design**: a reset wipes everything, and
+ the observation report is the only durable artifact. For what env cannot
+ express (volumes, networks), the escape hatch is a manual `docker run`
+ reusing the same name and ports — `status`/`up`/`down` keep working
+ against it, but a **reset recreates the container from the embedded
+ definition** and hand-mounted volumes do not survive it.
+
+ ## Recording a query as evidence
+
+ Query through gcx. For a proof query a verification run must replay
+ verbatim, the raw datasource-proxy URL
+ (`/api/datasources/proxy/uid/<uid>/...`) is the right form — it replays
+ with curl alone, on a machine with no gcx context. Both are
+ contract-conform; the report says which was used.