gke-alert-configuration · v1.0.0 · 2026-10-21 · sha256 16efd2bd1d9c361e
gke-alert-configuration v1.0.0A
Immutable. This exact content never changes and is served at /api/v1/blob/16efd2bd1d9c361e.
---
name: gke-alert-configuration
metadata:
version: "1.0.0"
category: CloudInfrastructure
canonical_source: https://github.com/google/skills/tree/main/skills/cloud/gke-alert-configuration
description: >-
Configures alerting policies in Terraform for Google Kubernetes Engine (GKE)
clusters, workloads, and services using PromQL and Google Cloud Managed Service
for Prometheus. Use when writing, analyzing, validating, or deploying Terraform
alerting policies to monitor GKE service latency, traffic, error rates using
Multi-Window Multi-Burn-Rate SLO alerts, memory saturation, and cluster health
such as CrashLoopBackOff and Node NotReady conditions.
Don't use for non-GKE compute runtimes such as standalone Compute Engine VMs or
standalone Cloud Run services without GKE.
---
# GKE Alert Configuration
This skill provides guidelines and best practices for creating robust,
high-signal alerting policies for Google Kubernetes Engine workloads using
Google Cloud Managed Service for Prometheus and Terraform. It ensures
comprehensive coverage of the **4 Golden Signals** and key cluster health
metrics while minimizing alert noise.
--------------------------------------------------------------------------------
## Critical Rules
* **Negative Triggers and Scope Redirection for Non-GKE Standalone Runtimes**:
* This skill is strictly scoped to Google Kubernetes Engine (GKE)
workloads, clusters, and services using PromQL and Google Cloud Managed
Service for Prometheus.
* **Do not use for non-GKE compute runtimes**, such as standalone Compute
Engine virtual machines or standalone Cloud Run services without GKE.
* **STOP AND RESPOND DIRECTLY (Do Not Edit Files)**: When the user
requests alert configuration for non-GKE compute infrastructure:
1. **Do not write, create, edit, or validate any Terraform files on
disk**.
2. **Immediately stop and respond directly to the user in chat**:
* **Explicitly Clarify Out-of-Scope**: State clearly that
standalone Compute Engine virtual machine monitoring or
standalone Cloud Run monitoring is out of scope for this
GKE-specific PromQL alerting skill, which is designed
specifically for GKE workloads using Google Cloud Managed
Service for Prometheus and PromQL.
* **Do Not Generate GKE PromQL Alerts**: Do not create or generate
Kubernetes PromQL alert policies or fabricate Kubernetes
container, pod, or node resources for non-GKE infrastructure.
* **Redirect the User**: Guide and redirect the user to standard
Google Cloud Monitoring metrics, such as
`compute.googleapis.com/instance/cpu/utilization` or
`run.googleapis.com/request_latencies`, using standard
`google_monitoring_alert_policy` with `condition_threshold` or
MQL, or recommend the relevant specialized Cloud observability
skill.
* **Mandatory `kube-state-metrics` (KSM) Cost Guardrail**:
* Deploying open-source `kube-state-metrics` in Google Cloud Managed
Service for Prometheus incurs billable metric ingestion costs.
* **STOP AND ASK PERMISSION FIRST (Do Not Edit Files)**: When a requested
alert rule relies on **Tier 2 KSM metrics** (such as `kube_cronjob_*`,
`kube_pod_status_phase`, `kube_persistentvolume_*`, `kube_deployment_*`,
`kube_statefulset_*`, `kube_job_*`, or `kube_daemonset_*`), **do not
write, create, edit, or validate any Terraform files or generate alert
policies before obtaining user approval**.
* Instead, you **must immediately stop and respond directly to the user**
to:
1. **Alert the user** that the requested alert requires
`kube-state-metrics`.
2. **Explain the cost impact**: Detail that `kube-state-metrics` incurs
billable sample ingestion costs in Google Cloud Managed Service for
Prometheus.
3. **Ask for explicit permission**: Ask the user for explicit
permission before assuming, enabling, or generating KSM-dependent
alert configurations.
4. **Recommend filtering or allowlisting**: Suggest and recommend
filtering or allowlisting only the specific required metrics, such
as using a `PodMonitoring` resource with `metricRelabeling`
(`action: keep`) or KSM `--metric-allowlist` to minimize ingestion
costs. Provide a concrete allowlist example.
* **Always prefer Non-KSM Native Alternatives** (Tier 1 cAdvisor or native
GKE metrics documented in
[metrics_and_alerts_catalog.md](references/metrics_and_alerts_catalog.md))
whenever possible, such as using `container_memory_working_set_bytes`
and `container_spec_memory_limit_bytes` instead of
`kube_pod_container_resource_limits`.
* **Explicit Tier and Cost Surcharge Identification in Response**: In
every response where you generate or recommend an alerting policy, you
**must explicitly state its classification tier and cost impact**:
* **Tier 1 native or standard metric** (GKE built-in metrics, cAdvisor
`container_*`, kubelet volume stats, native
`kubernetes.io/node/status_condition`, and control-plane metrics.
Every `kube_*` series the references name
(`kube_node_status_condition`, `kube_node_spec_taint` and
`kube_pod_container_status_restarts_total` among them) is a
kube-state-metrics series and therefore Tier 2, whatever the
KSM-required or tier columns in the references say; see
[metrics_and_alerts_catalog.md](references/metrics_and_alerts_catalog.md)):
State that it is a **Tier 1 native or standard metric with zero KSM
cost surcharge**.
* **Tier 2 KSM metric**: State that it is a **Tier 2 KSM-dependent
metric** and follow the permission and allowlisting guardrail above.
*(Tip: Generally, metrics with the `kube_` prefix that represent
resource state or metadata belong to Tier 2).*
* **Plan-Validate-Execute Loop for Approved File Edits**: When modifying,
adding, or merging approved Terraform files on disk in a workspace, follow
the three-phase workflow:
1. **Plan**: Draft a structured change plan (`changes.json`) containing
proposed policy resource names, PromQL expressions, grouping labels, and
durations.
2. **Validate**: Run the pre-edit validation script (`python3
"$HERMES_HOME"/skills/gke-alert-configuration/scripts/validate_config.py
--plan changes.json`) to check bracket balance, time-window and offset
formats, duration rules, and duplicate signals. The duplicate check
compares only entries that carry a `signal_type` (for example
`saturation_memory`), so give each planned policy one. Write the plan
as `{"policies": [{"resource_name": ..., "query": ..., "duration": ...,
"signal_type": ...}]}`: the script reads a duration only from the
`duration` key and takes a missing one as `0s`, so a duration under
any other key passes the duration rule unchecked. It does not
parse PromQL, so a pass does not prove the query is valid.
3. **Execute**: After the plan passes validation, apply or merge changes
in-place into the target Terraform configuration (`alerts.tf`).
4. *Note*: When answering questions or providing Terraform snippets
directly in chat where no disk modification is requested, output the
complete, valid Terraform HCL block in your response.
* **Configure the 4 Golden Signals and Cluster Health**: Always ensure the
target Kubernetes workload or service has the following alerting coverage:
1. **Latency** (P95 response time)
2. **Errors** (Multi-Window Multi-Burn-Rate SLO alerts, such as Fast Burn 1
hour / 5 minutes with factor 14.4, Slow Burn 6 hours / 30 minutes with
factor 6.0; do not use simple static ratios)
3. **Traffic** (Sudden drop or complete metric disappearance using
`absent()` or `default 0` syntax, or overload spikes)
4. **Saturation (Memory Limit Utilization Only)**: When describing or
configuring alert policies for a cluster or project, include ONLY
**Memory Saturation** (`container_memory_working_set_bytes` /
`container_spec_memory_limit_bytes`). Do **NOT** include CPU saturation
alerts or list `container_cpu_usage_seconds_total` as an alert metric
because CPU is compressible and throttled by CFS quotas rather than
causing uncompressible fatal termination (OOM).
5. **Cluster Health** (Pod CrashLooping, Node NotReady)
* **PromQL Only (Managed Prometheus)**: You must use
`condition_prometheus_query_language` with PromQL. Do **NOT** use MQL or
standard `condition_threshold` unless explicitly requested. Google Cloud
Managed Service for Prometheus is the standard telemetry ingestion path for
GKE.
* **Terraform Only**: Write the generated observability configuration ONLY as
Terraform (`.tf`) files, such as `alerts.tf` and `variables.tf`.
* **Dynamic Multi-Resource Alerting (No Hardcoding)**: You must not hardcode
specific pod names, node names, or service names in alerting conditions
unless explicitly requested. Alerting policies must be written to cover
resources dynamically:
* Always use grouping aggregations (`by (cluster, namespace, service, pod,
container)`) instead of filtering to a single instance. This allows a
single alert policy to dynamically track each service or pod separately.
* Always declare and use Terraform variables for `project_id`,
`cluster_name`, and `namespace` (`var.project_id`, `var.cluster_name`,
`var.namespace`) to make the configuration reusable across environments.
Always define these variables in `variables.tf` (or within the
configuration) and reference all three in policies or PromQL label
matchers.
* **No Redundant Duration Windows on Lookbacks**:
* When PromQL expressions already use an aggregated lookback window (such
as `increase(...[15m]) > 3` or multi-window SLO burn rates), the query
time window already smooths out transient spikes.
* Adding a Terraform duration on top of a PromQL lookback window increases
the Mean Time to Detect (MTTD) without providing additional smoothing
benefits.
* In these cases, set Terraform `duration = "0s"` (or `"60s"`). Do not
enforce `duration = "300s"` on top of `[15m]`, which delays critical
crashloop alerts by up to 20 minutes total (15 minutes + 5 minutes).
* Use `duration = "300s"` only on instantaneous gauge conditions, such as
`kube_node_status_condition == 0`.
* **Use SLO Burn Rates Instead of Simple Ratios**: For error rate alerting,
always generate Multi-Window Multi-Burn-Rate (MWMBR) SLO alerts (such as
14.4x burn rate over 1 hour and 5 minute windows for a 99% SLO) rather than
simple error rate ratios (`rate(5xx)/rate(total) > 0.05`), which produce
excessive false alarms on low traffic.
* **Robust Traffic Drop Detection (`absent()` / `default 0`)**: When
monitoring for traffic drops to zero, do not use `rate(...) == 0` alone
because Prometheus time series disappear completely when no requests occur
(evaluating to an empty vector rather than 0). Use `default 0` syntax, such
as `sum(rate(...[5m])) default 0 == 0`, or `absent(...) == 1`.
* **Notification Channels**: By default, never configure any notification
channels without user input. If the user explicitly provides a notification
channel, configure the alerts to use it. Otherwise, you must prompt the user
in your response to ask if they would like to configure one.
* **Consult GKE Metrics and Open-Source Alerts Catalog**: When designing or
generating evaluation suites or alerting policies, consult
[metrics_and_alerts_catalog.md](references/metrics_and_alerts_catalog.md)
for public GKE metrics (`kubernetes.io/`) and open-source Kubernetes alerts
(`awesome-prometheus-alerts`).
* **Plain English Response**: You must include a plain English explanation for
what the alerts do in your response. Explain what the alert measures, what
the threshold represents, and what a trigger indicates.
* **User Labels**: Include a `user_labels` block in all
`google_monitoring_alert_policy` resources to track policies created by this
skill:
```terraform
user_labels = {
created-with-google-skill = "gke-alert-configuration"
}
```
--------------------------------------------------------------------------------
## Alerting Policy Structure in Terraform
Alerting policies must be defined using the `google_monitoring_alert_policy`
resource with `condition_prometheus_query_language`. Always declare variables in
`variables.tf` for `project_id`, `cluster_name`, and `namespace`.
```hcl
# variables.tf
variable "project_id" {
type = string
description = "Google Cloud Project ID"
}
variable "cluster_name" {
type = string
description = "GKE Cluster Name"
}
variable "namespace" {
type = string
description = "Target Kubernetes Namespace"
default = "default"
}
variable "slo_target" {
type = number
description = "SLO Target fraction (for example 0.99 for 99%)"
default = 0.99
}
```
```hcl
# alerts.tf
# Example: Multi-Window Multi-Burn-Rate (MWMBR) SLO Alert (Fast Burn: 14.4x, 1h & 5m windows)
resource "google_monitoring_alert_policy" "k8s_service_error_rate_slo" {
project = var.project_id
display_name = "[K8s] ${var.cluster_name} - Service Error Rate SLO Fast Burn"
combiner = "OR"
conditions {
display_name = "Error Budget Fast Burn (14.4x over 1h and 5m)"
condition_prometheus_query_language {
query = <<-EOT
(
(
sum(
rate(
http_requests_total{
cluster="${var.cluster_name}",
namespace="${var.namespace}",
status=~"5.."
}[5m]
)
) by (service, namespace, cluster)
/
sum(
rate(
http_requests_total{
cluster="${var.cluster_name}",
namespace="${var.namespace}"
}[5m]
)
) by (service, namespace, cluster)
) > (1 - ${var.slo_target}) * 14.4
)
and
(
(
sum(
rate(
http_requests_total{
cluster="${var.cluster_name}",
namespace="${var.namespace}",
status=~"5.."
}[1h]
)
) by (service, namespace, cluster)
/
sum(
rate(
http_requests_total{
cluster="${var.cluster_name}",
namespace="${var.namespace}"
}[1h]
)
) by (service, namespace, cluster)
) > (1 - ${var.slo_target}) * 14.4
)
EOT
duration = "0s"
}
}
}
```
--------------------------------------------------------------------------------
## Telemetry Metrics and PromQL Examples
For GKE metrics (`kubernetes.io/`), community open-source alerts
(`awesome-prometheus-alerts`), KSM cost guardrails, and non-KSM native
alternatives, you must read and follow:
* [metrics_and_alerts_catalog.md](references/metrics_and_alerts_catalog.md)
For specific PromQL queries corresponding to each of the Golden Signals, you
must read and follow:
* [promql_queries.md](references/promql_queries.md)
For GKE cluster prerequisites, enabling Google Cloud Managed Service for
Prometheus collection, configuring PodMonitoring custom scraping, and enabling
control plane metrics collection (API Server, Controller Manager, Scheduler),
you must read and follow:
* [gke_configuration_prerequisites.md](references/gke_configuration_prerequisites.md)
--------------------------------------------------------------------------------
## Tooling Scripts and Validation Loop
Use the `validate_config.py` script to validate change plans and Terraform
configurations when working in a repository:
* **Pre-Edit Plan Validation**: Draft a `changes.json` plan specifying the
proposed policies, queries, and durations, and validate it before editing:
* Command: `python3
"$HERMES_HOME"/skills/gke-alert-configuration/scripts/validate_config.py
--plan changes.json`
* **Post-Edit and Directory Validation**: Scan existing or modified Terraform
files in a directory to ensure no duplicates or syntax errors exist:
* Command: `python3
"$HERMES_HOME"/skills/gke-alert-configuration/scripts/validate_config.py
--directory [TARGET_TF_DIR]`
* For a single file, run `--directory` on the absolute path of the
directory that holds it. `--file` prints no policy count, so its
pass cannot be read against the rules below.
* The script does not check label scoping or hardcoded cluster names:
confirm each query's scope yourself. cAdvisor, kube-state-metrics
and application series take `cluster="${var.cluster_name}"` and
`namespace="${var.namespace}"`; `kubernetes_io:` series take
`cluster_name` and `namespace_name`; node-level series take no
namespace matcher.
* `"policies_scanned_count"` counts the alert-policy blocks the scan
extracted, and `0` never means the files are clean. No `.tf` file
sits directly in that path (pass the absolute path of the directory
that holds them; the scan does not recurse), the files hold no
alert policy, or the scanner could not parse a policy block. If the
right path still reports `0` for a file that holds a policy, check
that policy by eye. A non-zero count is not a clean bill either. The
scanner reads comments as code, so a `{`, `}` or `"` in a comment can
drop a policy from the count or cut it short so it is counted but
not linted. It lints a `query` written as an inline `EOT` heredoc,
and a quoted-string `query` only in a resource with no heredoc one;
a `query` from a variable, `local`, `file()`, `templatefile()` or
another heredoc delimiter is counted but not linted. Check by eye
every policy with a comment inside it and every `query` the scan
does not lint.
--------------------------------------------------------------------------------
## Technical Considerations and Gotchas
* **Lookback Windows versus Duration Buffers**:
* Do not add large `duration = "300s"` buffers to alerts that already use
aggregated lookback windows like `increase(...[15m])` or multi-window
SLO rates.
* The `[15m]` window in `increase(...[15m]) > 3` already smooths spikes.
Adding `duration = "300s"` increases MTTD by forcing the restart count
to remain above 3 for an extra 5 continuous minutes, delaying alerts by
up to 20 minutes total.
* Use `duration = "0s"` or `"60s"` when using lookback window functions.
Reserve `duration = "300s"` for raw instantaneous gauge conditions, such
as `kube_node_status_condition == 0`.
* **Memory Saturation Only for Cluster Alerting**:
* Do not configure CPU saturation alerts for cluster or workload
monitoring. CPU is compressible (throttled by the CFS scheduler), while
memory is uncompressible (triggers OOMKills).
* Configure Memory Saturation using `container_memory_working_set_bytes` /
`container_spec_memory_limit_bytes`.
* **Missing Resource Limits Blind Spot (Mandatory Explanation)**: Saturation
alerts that compare usage to limits (such as
`container_spec_memory_limit_bytes`) will **fail to resolve** or return
`NaN` if workloads do not have explicit Memory limits configured in their
Kubernetes manifests.
* **Mandatory Instruction**: Whenever you generate, discuss, or recommend
any memory saturation alert comparing usage against limits (including
non-KSM cAdvisor alternatives using
`container_spec_memory_limit_bytes`), you **must explicitly explain and
warn the user in your response** that container memory limits must be
explicitly configured in the Kubernetes pod specs or manifests
(`resources.limits.memory`) for the saturation query to resolve (and not
return `NaN` or fail to resolve).
* **Linear Disk Predictions (`predict_linear`)**: When forecasting volume
exhaustion using
`predict_linear(kubelet_volume_stats_available_bytes[6h:5m], 4 * 24 * 3600)
< 0`, explain that `predict_linear` uses linear regression over the recent
lookback window (for example, 6 hours) to project when available disk will
drop below 0 (for example, within 4 days). Identify
`kubelet_volume_stats_available_bytes` as a Tier 1 native kubelet metric
with zero KSM surcharge.
* **API Server Error and Client Metrics**:
* `apiserver_request_total` and `rest_client_requests_total` are Tier 1
Control Plane metrics with zero KSM cost surcharge. Explain that
`apiserver_request_total` monitors 5xx HTTP error rates across API
server endpoints, while `rest_client_requests_total` monitors 4xx and
5xx requests sent by REST clients communicating with the API server.
* **Traffic Disappearance Gotcha (`absent()` / `default 0`)**:
* When traffic drops completely to zero, Prometheus and GMP stop emitting
the `http_requests_total` time series.
* `sum(rate(...[5m])) == 0` evaluates to an empty vector, preventing the
alert from triggering.
* Always use `sum(rate(...[5m])) default 0 == 0` or `absent(...) == 1` to
reliably detect total traffic loss.
* **CrashLooping versus Normal Restarts**: A container restarting occasionally
might be normal, for example job completion or a minor rolling update. Alert
on **frequent** restarts (such as more than 3 restarts in 15 minutes with
`duration = "0s"`) using `kube_pod_container_status_restarts_total` rather
than a single restart to avoid noise.
* **Node Upgrades**: During GKE cluster upgrades, nodes are drained and
restarted, which can trigger "Node NotReady" alerts. Warn the user that
these alerts might fire during maintenance windows, or suggest configuring
maintenance windows if supported.
--------------------------------------------------------------------------------
## Additional Resources
* [Google Cloud Managed Service for Prometheus Documentation](https://docs.cloud.google.com/monitoring/managed-prometheus.md.txt)
* [GKE Observability and Monitoring Concepts](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/monitoring.md.txt)
* [Google Cloud Alerting Policies in Terraform](https://docs.cloud.google.com/monitoring/alerts/terraform-alert-policy.md.txt)
* [Google Cloud Monitoring Pricing](https://docs.cloud.google.com/monitoring/pricing.md.txt)
* [Google SRE Workbook: Alerting on SLOs](https://sre.google/workbook/alerting-on-slos/)
* [Awesome Prometheus Alerts Repository](https://github.com/samber/awesome-prometheus-alerts)
<!-- kube-agents: local addition (auto-injected by sync-upstream-skills.py) -->
## Corrections to the references
These override the reference files, and the shorthand formulas in the sections above, where
they disagree.
- **Memory-saturation query.** Filter the limit series with `> 0` before dividing wherever this
skill gives the ratio: the Critical Rules and Technical Considerations above,
[promql_queries.md](references/promql_queries.md) and
[metrics_and_alerts_catalog.md](references/metrics_and_alerts_catalog.md) alike:
```promql
sum(
container_memory_working_set_bytes{
cluster="${var.cluster_name}",
namespace="${var.namespace}",
container!=""
}
) by (pod, container)
/
sum(
container_spec_memory_limit_bytes{
cluster="${var.cluster_name}",
namespace="${var.namespace}",
container!=""
} > 0
) by (pod, container) > 0.90
```
cAdvisor reports `container_spec_memory_limit_bytes` as `0` for a container with no limit, so
the unfiltered ratio is `+Inf` and the alert fires for as long as that container runs. With the
filter, unlimited containers return no series, which is what the references' warning about
missing limits describes.
- **Enabling control-plane metrics.** Do not run the `gcloud container clusters update
--monitoring=SYSTEM,API_SERVER,CONTROLLER_MANAGER,SCHEDULER` command in
[gke_configuration_prerequisites.md](references/gke_configuration_prerequisites.md). It changes
the cluster's configuration, so ask the user first, and `--monitoring` replaces the cluster's
whole list of enabled components rather than adding to it: a list that omits `POD`, `CADVISOR`,
`KUBELET` or `STORAGE` turns those packages off, and with them the series other alerts in this
skill read. Read the current list first:
```bash
gcloud container clusters describe CLUSTER --location LOCATION \
--format='value(monitoringConfig.componentConfig.enableComponents)'
```
It prints API names separated by `;`, and `--monitoring` takes a comma list in the flag's own
spelling: map `SYSTEM_COMPONENTS` to `SYSTEM` and `APISERVER` to `API_SERVER` (the other names
are spelled the same), join them with `,`, and add `API_SERVER`, `SCHEDULER` and
`CONTROLLER_MANAGER` where absent, so nothing already enabled (`JOBSET` on a golden-path
cluster, for one) is turned off. For a cluster that printed
`SYSTEM_COMPONENTS;STORAGE;POD;CADVISOR;KUBELET;JOBSET`, the update to propose is:
```bash
gcloud container clusters update CLUSTER --location LOCATION \
--monitoring=SYSTEM,STORAGE,POD,CADVISOR,KUBELET,JOBSET,API_SERVER,SCHEDULER,CONTROLLER_MANAGER
```
- **Cluster-health queries.** `kube_node_status_condition` and
`kube_pod_container_status_restarts_total` are kube-state-metrics series (Tier 2) wherever this
skill uses them, including the references' "Non-KSM Alternative" sections. The Tier 1 natives
for Node NotReady and crash-looping containers, in PromQL, carry `cluster_name` and
`namespace_name` labels rather than `cluster` and `namespace`:
```promql
kubernetes_io:node_status_condition{
monitored_resource="k8s_node",
cluster_name="${var.cluster_name}",
condition="Ready",
status!="True"
} > 0
```
`status!="True"` and not `status="False"`: a node whose kubelet stops posting status goes to
`Unknown`, which is the usual `NotReady`.
The references' Tier 2 allowlist template keeps no `kube_node_*` series. When a
kube-state-metrics node alert is approved, add `kube_node_status_condition` and every other
`kube_node_*` series the policy reads to its `keep` regex, or the relabel drops them at scrape
and the alert never fires.
```promql
increase(
kubernetes_io:container_restart_count{
monitored_resource="k8s_container",
cluster_name="${var.cluster_name}",
namespace_name="${var.namespace}"
}[15m]
) > 3
```
- **Traffic-drop queries.** `default 0` is MetricsQL, not PromQL, and Cloud Monitoring rejects
it when the policy is applied, although `validate_config.py` passes it. Wherever this skill or
its references write `... default 0`, rewrite it by shape. For the per-service week-over-week
comparison the references recommend, alert on services that had traffic a week ago and have
none now:
```promql
(
sum(rate(http_requests_total{cluster="${var.cluster_name}", namespace="${var.namespace}"}[5m] offset 1w )) by (service) > 1
)
unless
(
sum(rate(http_requests_total{cluster="${var.cluster_name}", namespace="${var.namespace}"}[5m])) by (service) > 0
)
```
For a single total with no grouping labels, use `(sum(rate(...[5m])) or vector(0)) == 0`; for a
whole series disappearing, `absent(...)` on the same selector. Neither joins a `by (service)`
baseline: the `vector(0)` and `absent()` samples carry no `service` label, so an `and` against
one never matches.
- **Duplicate targets.** Each validator mode checks duplicates differently. `--directory` infers
a policy's signal from keywords in its resource or display name (`error`, `latency`, `memory`
and others, first match wins) and reports a "Duplicate Target Error" when two policies share
one. `--plan` compares the `signal_type` strings written in the plan. `--file` checks no
duplicates at all, so a pass from it says nothing about them. A duplicate is real only when two
policies query the same metric on the same resource and labels. Policies that read different
series (API-server and API-client errors, P95 and P99 latency) or are burn-rate tiers of one SLO
with their own severities stay separate: tell the user why, and do not merge them. In a plan,
give each such policy its own `signal_type` (for example `errors_fast_burn` and
`errors_slow_burn`), which `--plan` accepts.
- **Terraform.** The agent's shell has OpenTofu for `tofu fmt` only and no `terraform` binary. The
prerequisites' Verification Runbook check, `tofu fmt -check -diff`, runs there; `init`,
`validate` and `plan` are the user's or their CI's: name them as the user's step, and do not
report them as run.
- **Catalog regexes.** The `metrics_and_alerts_catalog.md` table rows escape each `|` with a
backslash for Markdown, and in PromQL's RE2 that backslash makes the pipe a literal, so a
copied matcher never matches. Drop the backslashes: the regexes are `(4|5)..` (API client
errors), `Pending|Unknown|Failed` (Pod not healthy), `Failed|Pending` (PersistentVolume error)
and `(?:CONNECT|WATCHLIST|WATCH|PROXY)` (API server latency).
- **Offset spacing.** The validator reads an `offset` duration up to the next whitespace, so it
rejects `offset 1w))`. Leave a space or line break after the duration (`offset 1w ))`), as the
traffic-drop query above does.
- **Validator path.** Wherever a reference runs the validator as `scripts/validate_config.py`, run
`python3 "$HERMES_HOME"/skills/gke-alert-configuration/scripts/validate_config.py` instead.
Where it passes `--file <file>` (the prerequisites' Verification Runbook does), pass
`--directory` with the absolute path of the directory that holds the file, so the scan prints
the policy count the Post-Edit step reads.