v1.0 to git:20260812.7c6fef6

43 added, 38 removed. Audit A to A.

---

name: elasticsearch

- description: Query Elasticsearch logs, APM traces, and errors via curl — index patterns, field names, auth setup, and time-range syntax

+ description: Load when investigating production errors, latency, or trace data — triage order, index patterns, field names, and time-range syntax. Use before hand-rolling a query DSL call or guessing field names.

license: MIT

- metadata:

- author: athal7

- version: "1.0"

- provides: query-logs query-apm-traces query-errors

---

- Query application logs, APM traces, and errors using the Elasticsearch REST API directly.

+ Query application logs, APM traces, and errors via the Elasticsearch REST API. Endpoint `$ES_URL` varies per environment.

- Requires environment variables:

- - `ES_URL` — base URL (e.g. `https://elasticsearch.example.com`)

- - `ES_API_KEY` — API key for the `Authorization: ApiKey` header

+ ## Triage order

+ State the hypothesis, bound the time window, and identify the scope (one service / one endpoint / one user / system-wide) before querying anything. Write down what a "confirmed" answer looks like — otherwise absence of evidence reads as evidence of absence.

+ **Don't start with logs.** Work coarse to fine: error/rate spike (is it real?) → APM traces (which transaction) → APM errors (root-cause code path) → logs (raw context). `trace.id` links all three indices — pull it from an error or slow trace, then `{"term": {"trace.id": "<id>"}}` for the surrounding logs.

+ | Symptom | Look first |

+ |---|---|

+ | Slow page loads | Traces, sort `transaction.duration.us` desc |

+ | 500s spiking | Errors, group by `error.grouping_key` |

+ | One user affected | Logs, filter by user/session id |

+ | Periodic | Logs, time pattern in `@timestamp` |

+ | After a deploy | Errors, filter `@timestamp` after deploy |

+ **Escalate** rather than keep digging when: error rate stays above baseline >15min with no identified cause · a trace shows an external dependency timing out · errors reference a migration or schema change · you need production access you don't have.

## Time range syntax

Pass `time_range` as a string like `15m`, `1h`, `24h`, `7d`. Translates to `now-{value}{unit}` in ES range filters.

## Query logs

Search application logs. Index: `logs-*`. Sorted by `@timestamp` desc.

- ```bash

- ES_QUERY='{"query":{"bool":{"must":[{"query_string":{"query":"YOUR LUCENE QUERY HERE"}},{"range":{"@timestamp":{"gte":"now-1h"}}}]}},"_source":["@timestamp","message","log.level","service.name","trace.id"],"sort":[{"@timestamp":"desc"}],"size":100}'

- curl -s -X POST "$ES_URL/logs-*/_search" \

- -H "Authorization: ApiKey $ES_API_KEY" \

- -H "Content-Type: application/json" \

- -d "$ES_QUERY" \

- | jq '.hits.hits[]._source | {ts: .["@timestamp"], level: .["log.level"], svc: .["service.name"], msg: .message}'

```

+ POST logs-*/_search

+ ```

+ ```json

+ {"query":{"bool":{"must":[{"query_string":{"query":"YOUR LUCENE QUERY HERE"}},{"range":{"@timestamp":{"gte":"now-1h"}}}]}},"_source":["@timestamp","message","log.level","service.name","trace.id"],"sort":[{"@timestamp":"desc"}],"size":100}

+ ```

+ Response fields: `.hits.hits[]._source` — extract `@timestamp`, `log.level`, `service.name`, `message`.

Add a service filter by inserting a `term` clause into the `must` array:

```json

{"term": {"service.name": "my-service"}}

```

## Query APM traces

Find slow transactions. Index: `traces-apm*`. Sorted by duration desc.

- ```bash

- # min_duration_ms converts to microseconds: 500ms → 500000us

- MIN_US=500000

- ES_QUERY="{\"query\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":\"now-1h\"}}},{\"range\":{\"transaction.duration.us\":{\"gte\":$MIN_US}}}]}},\"_source\":[\"@timestamp\",\"service.name\",\"transaction.name\",\"transaction.duration.us\",\"transaction.result\",\"trace.id\"],\"sort\":[{\"transaction.duration.us\":\"desc\"}],\"size\":50}"

- curl -s -X POST "$ES_URL/traces-apm*/_search" \

- -H "Authorization: ApiKey $ES_API_KEY" \

- -H "Content-Type: application/json" \

- -d "$ES_QUERY" \

- | jq '.hits.hits[]._source | {ts: .["@timestamp"], svc: .["service.name"], tx: .["transaction.name"], ms: (.["transaction.duration.us"] / 1000 | round), result: .["transaction.result"]}'

```

+ POST traces-apm*/_search

+ ```

+ ```json

+ {"query":{"bool":{"must":[{"range":{"@timestamp":{"gte":"now-1h"}}},{"range":{"transaction.duration.us":{"gte":500000}}}]}},"_source":["@timestamp","service.name","transaction.name","transaction.duration.us","transaction.result","trace.id"],"sort":[{"transaction.duration.us":"desc"}],"size":50}

+ ```

+ Response fields: `.hits.hits[]._source` — extract `@timestamp`, `service.name`, `transaction.name`, `transaction.duration.us` (microseconds), `transaction.result`.

## Query APM errors

Find exceptions and error groups. Index: `logs-apm.error-*`. Sorted by `@timestamp` desc.

- ```bash

- ES_QUERY='{"query":{"bool":{"must":[{"exists":{"field":"error.exception"}},{"range":{"@timestamp":{"gte":"now-1h"}}}]}},"_source":["@timestamp","error.exception.type","error.exception.message","error.grouping_key","service.name","transaction.name"],"sort":[{"@timestamp":"desc"}],"size":50}'

- curl -s -X POST "$ES_URL/logs-apm.error-*/_search" \

- -H "Authorization: ApiKey $ES_API_KEY" \

- -H "Content-Type: application/json" \

- -d "$ES_QUERY" \

- | jq '.hits.hits[]._source | {ts: .["@timestamp"], svc: .["service.name"], type: .["error.exception.type"], msg: .["error.exception.message"]}'

```

+ POST logs-apm.error-*/_search

+ ```

+ ```json

+ {"query":{"bool":{"must":[{"exists":{"field":"error.exception"}},{"range":{"@timestamp":{"gte":"now-1h"}}}]}},"_source":["@timestamp","error.exception.type","error.exception.message","error.grouping_key","service.name","transaction.name"],"sort":[{"@timestamp":"desc"}],"size":50}

+ ```

+ Response fields: `.hits.hits[]._source` — extract `@timestamp`, `error.exception.type`, `error.exception.message`, `service.name`.

## Tips

- `query_string` uses Lucene syntax: `error AND timeout`, `level:ERROR`, `message:"connection refused"`

- - To count by service: append `,"aggs":{"by_svc":{"terms":{"field":"service.name","size":10}}}` and read `.aggregations.by_svc.buckets`

+ - To count by service: append `,"aggs":{"by_svc":{"terms":{"field":"service.name","size":10}}}` to the query JSON and read `.aggregations.by_svc.buckets`

- `trace.id` links logs ↔ traces ↔ errors across indices

- - If `$ES_API_KEY` is missing, ensure the variable is set in your environment (open a new shell)

+ ## Kibana Dashboard API Gotchas

+ - **`PUT /api/saved_objects/dashboard/:id` replaces ALL attributes.** Read the full object first, modify only `panelsJSON`, and write everything back including `controlGroupInput`, `optionsJSON`, etc. Omitting any attribute silently breaks panels.

+ - **`PUT /api/saved_objects/index-pattern/:id` wipes the `fields` attribute** if you only set `title`/`timeFieldName`. To recreate safely, delete and use `POST /api/data_views/data_view` which auto-discovers fields.

+ - **Inline Lens panels referencing an `index-pattern` saved object render blank if that object is corrupted.** The resilient pattern is `adHocDataViews` + `internalReferences` inside `embeddableConfig.attributes.state` — self-contained, no external saved-object dependency.

+ - **ES transform `_update` cannot change `pivot`.** Must stop, delete, recreate. If the dest index has historical data from rolled-over source indices, check `_snapshot` first.