kube-agents-observability git:20261022.61cbc0aB

Immutable. This exact content never changes and is served at /api/v1/blob/dbed6b0092272e40.

---
name: kube-agents-observability
description: Audit, monitor, and debug the logging, tracing, metrics, and API/dashboard observability of the Platform Agent.
---

# Task

Audit, verify, and troubleshoot the logging, metrics, and distributed tracing observability of the Platform Agent.

> [!TIP]
> The provided Python scripts in the `scripts/` subdirectory are parameterized reference implementations. When troubleshooting, you can run them directly, customize their parameters, or write custom just-in-time scripts/commands to query more specific metrics, endpoints, or time ranges as required by the task context.

> [!IMPORTANT]
> The helpers reach Google APIs through the credential broker and never hold a token: the Cloud Trace and Cloud Monitoring reads go through `credential_proxy_client.ApiSession` (see `scripts/google_api.py`), and the Cloud Logging read runs `gcloud logging read` through the broker's shim. Write a just-in-time script the same way. Do not fetch an access token and do not write an `Authorization` header yourself: the shim refuses a `gcloud` token fetch, the metadata server answers the sandbox with an identity IAM grants nothing, and whatever either returns lands in the transcript.

# Workflow

## Logging

### 1. Audit Agent Main Logs

- Verify that the main agent container is writing logs to `/opt/data/logs/*.log`.
- View the internal agent log files directly:
  ```bash
  kubectl exec <pod-name> -c <agent-container-name> -n kubeagents-system -- tail -n 100 /opt/data/logs/agent.log
  ```
- The tool-call audit trail is not in `agent.log`: it is one JSON object per line in `/opt/data/logs/audit.jsonl` for the front door and `/opt/data/profiles/<profile>/logs/audit.jsonl` for a named profile; in Logs Explorer, `jsonPayload.audit_event:*`, with `jsonPayload.file_path` naming the profile's file.
  ```bash
  kubectl exec <pod-name> -c <agent-container-name> -n kubeagents-system -- tail -n 20 /opt/data/logs/audit.jsonl
  ```

### 2. Inspect Sidecar Log Aggregator (Fluent-bit)

- Verify the `fluent-bit` sidecar container tails the log directory and streams to standard output:
  ```bash
  kubectl logs <pod-name> -c fluent-bit -n kubeagents-system --tail=100
  ```
- Retrieve and verify the configuration of the Fluent-bit sidecar:
  ```bash
  kubectl get configmap <agent-name>-fluent-bit-config -n kubeagents-system -o yaml
  ```
- Ensure the shared `/opt/data` volume is mounted to both the agent and Fluent-bit containers:
  ```bash
  kubectl get pod <pod-name> -n kubeagents-system -o jsonpath='{.spec.containers[*].volumeMounts}'
  ```

### 3. Identify Active Chat Users (Auditing Interactions)

To determine which users have interacted with the system via Google Chat in the last 24 hours (or a custom window):

- Run the packaged Python helper script to automatically query and parse the GKE container logs from Google Cloud Logging:

  ```bash
  python3 /opt/data/profiles/platform/skills/kube-agents-observability/scripts/get_chat_users.py --project-id <PROJECT_ID> [--hours <HOURS>] [--limit <N>]

  ```

- Alternatively, search Cloud Logging manually (via console or gcloud CLI) for the custom GChat event format emitted by the hermes session store:
  ```bash
  gcloud logging read 'resource.type="k8s_container" "Logging incoming GChat event"' --project=<PROJECT_ID> --limit=1000 --format="json"
  ```
  Look for log lines containing the format: `Logging incoming GChat event: User=<email>, Session=<session_id>`.

## Metrics

> [!NOTE]
> LLM token and operational metrics are conditional on the LLM proxy or inference server used.
>
> - **LiteLLM**: The scripts below query custom LiteLLM metrics. See the [LiteLLM Prometheus Documentation](https://docs.litellm.ai/docs/proxy/prometheus) for a complete list of metrics.

> - **vLLM**: Exposes different Prometheus metrics (e.g., `vllm:num_requests_waiting`). See the [vLLM Metrics Documentation](https://docs.vllm.ai/en/stable/usage/metrics/) for details.
> - **Other providers**: Query names will vary based on the specific provider's exporter.

### 1. Verify Cloud Monitoring & Prometheus State

- Check that Google Cloud Managed Service for Prometheus (GMP) is running in the cluster:
  ```bash
  kubectl get pods -n gmp-system
  ```
- Read the scrape configuration off the `PodMonitoring` resources, not off Deployment annotations: the operator renders none. The Helm chart renders four where the cluster serves the `PodMonitoring` API (`platformAgent.podMonitoring`, `null` by default, forces them on or off); the fourth, `<name>-a2a-gateway-monitoring` (port 9096), selects the A2A gateway pod and is empty on an install that runs none. The three that carry the watcher, the chat plugin and the broker:
  - `<name>-gateway-monitoring`: the event watcher, listening in the gateway pod's `agent-api-auth` container on its `event-metrics` port (9095), on every replica.
  - `<name>-gateway-chat-monitoring`: the `chat_metrics` plugin, listening in the gateway pod's `platform-agent` container on its `chat-metrics` port (9097) with its `kubeagents_chat_*` series (`kubeagents_chat_messages_inbound_total` and the rest). It selects the pod carrying `kubeagents.io/is-leader: "true"`: above one replica only the leader runs Hermes, so only the leader has the port; at one replica the operator puts the label on the one pod.
  - `<name>-credential-proxy-monitoring`: the credential broker, serving its `kubeagents_*` series on the broker pod's metrics-only `cred-metrics` port (8766).
  ```bash
  kubectl get podmonitoring <name>-gateway-monitoring -n kubeagents-system -o yaml
  kubectl get podmonitoring <name>-credential-proxy-monitoring -n kubeagents-system -o yaml
  ```
  An absent `PodMonitoring` means no scrape is configured, whatever the Deployment says.
- Prove the scrape in Cloud Monitoring, not in the cluster. Watcher: `up{job="<name>-gateway-monitoring"}` at `1`, and `k8s_event_watcher_cluster_up` present for each watched cluster; the chat plugin under its own job, `up{job="<name>-gateway-chat-monitoring"}` at `1` for the one leader pod: `kubeagents_chat_messages_inbound_total` present once a message has arrived. Broker: `up{job="<name>-credential-proxy-monitoring"}` and `kubeagents_credential_proxy_requests_total{endpoint="/healthz"}`, which the readiness probe keeps non-empty; `kubeagents_tool_invocations_total` appears only after the first brokered command, so its absence alone says nothing about the scrape. The collector's own `cluster` and `location` labels take precedence, so the watcher's arrive as `exported_cluster` and `exported_location`; query those. Send this PromQL through the credential broker's API relay (`credential_proxy_client.ApiSession`, route `gcp.api.monitoring.promql-read`); the relay carries the credential, so the query needs no token of its own.

### 2. Inspect CPU and Memory Metrics

- Query Kubernetes metrics API to verify resource usage of the agent pods:
  ```bash
  kubectl top pod -l app=<agent-name>-gateway -n kubeagents-system
  ```

### 3. Check Token Usage (Last 24h)

- Run the python script to fetch LiteLLM total token metrics from Cloud Monitoring:
  ```bash
  python3 /opt/data/profiles/platform/skills/kube-agents-observability/scripts/check_token_usage.py --project-id <project-id>
  ```

### 4. List LiteLLM Metric Descriptors

- Run the python script to list all available metric descriptors for LiteLLM:
  ```bash
  python3 /opt/data/profiles/platform/skills/kube-agents-observability/scripts/get_metric_descriptors.py --project-id <project-id>
  ```

## Traces

> [!NOTE]
> The system defaults to GKE Managed OpenTelemetry for distributed tracing, but the collector is configurable and may have been discovered rather than defaulted. **Never assume the `gke-managed-otel` endpoint** — read it off the resource before diagnosing anything.
>
> - **Harness Agents**: Emit traces natively via the `hermes_otel` plugin.
> - **LiteLLM**: Emits trace spans via its OTLP callback system.
> - **Visualization**: Exported traces are stored in Google Cloud Trace and can be searched/analyzed in the **Trace Explorer** console.

### 1. Verify OpenTelemetry (OTel) Configuration

- Find the collector this agent is actually exporting to. The operator reports what it resolved and where the answer came from (`DeploymentEnv`, `Spec`, `OperatorEnv`, `Discovered`, `None`, or `Default`):

  ```bash
  kubectl get platformagent <name> -n kubeagents-system -o jsonpath='{.status.telemetry}'
  ```

  A source of `None` means discovery ran and this cluster has no collector: the agent carries `OTEL_SDK_DISABLED=true`, no endpoint, and telemetry is off by design. **Stop here** — the remaining checks in this section and all of section 2 assume an endpoint is set, and on a `None` agent they report a permanent, expected mismatch as if it were a fault. Point it somewhere with `spec.telemetry.otlpEndpoint`, or install a collector; the operator re-probes every 15 minutes and picks it up without a restart.

  A source of `Default` on a cluster without GKE Managed OTel is the one to treat as a fault: it means nobody established what is there, so discovery is switched off (`OTEL_COLLECTOR_DISCOVERY=false`) or the probe cannot complete — most often the operator's cluster-wide RBAC on `services` has been narrowed. Spans are going nowhere and the endpoint on the pod does not resolve.

- Ensure the `hermes_otel` plugin is enabled in the profile's own config — `/opt/data/config.yaml` for the Chat Agent, `/opt/data/profiles/<profile>/config.yaml` for the Platform and Cluster Agents.
- Verify the plugin's exporter backend matches that endpoint (on a `None` cluster, `enabled: false` and `backends: []` is expected in the plugin config). It is rewritten at container start from `OTEL_EXPORTER_OTLP_ENDPOINT`, so a mismatch means the pod predates the current setting and needs a restart:
  ```bash
  kubectl exec <pod-name> -c <agent-container-name> -n kubeagents-system -- \
    sh -c 'echo "$OTEL_EXPORTER_OTLP_ENDPOINT"; grep -r endpoint /opt/data/plugins/hermes_otel/config.yaml /opt/data/profiles/*/plugins/hermes_otel/config.yaml'
  ```

### 2. Diagnose Trace Collector Connectivity

- Test network reachability from the agent container to the OpenTelemetry collector, using the endpoint from the container's own environment rather than a hardcoded one:
  ```bash
  kubectl exec <pod-name> -c <agent-container-name> -n kubeagents-system -- \
    sh -c 'curl -i -s -o /dev/null -w "%{http_code}\n" -X POST "$OTEL_EXPORTER_OTLP_ENDPOINT/v1/traces"'
  ```
- Check the agent logs for OTLP connection warnings or trace export failures:
  ```bash
  kubectl logs <pod-name> -c <agent-container-name> -n kubeagents-system --tail=500 | grep -iE "(otel|trace|exporter|export)"
  ```

### 3. Fetch and Analyze Traces (Locating Performance Bottlenecks)

To list recent traces or analyze span latency distributions to locate performance bottlenecks (such as slow tool executions or model calls):

- Run the trace latency analyzer script:

  ```bash
  python3 /opt/data/profiles/platform/skills/kube-agents-observability/scripts/analyze_trace_latency.py --project-id <project-id> [--hours <hours>] [--limit <limit>]
  ```

  **Example Output:**

  ```text
  Retrieving the last 3 traces...
  ======================================================================
  Trace ID: 0006344377aac15d1baede1a41e88a2c
  Total Duration: 0.647 seconds | Total Spans: 3
  Breakdown of spans:
    - POST /v1/chat/completions                          :  0.646s (99.9%)
    - chat model-default                                 :  0.627s (97.0%)
    - auth /v1/chat/completions                          :  0.001s ( 0.1%)
  ```

- Alternatively, run the raw trace list script:
  ```bash
  python3 /opt/data/profiles/platform/skills/kube-agents-observability/scripts/fetch_traces.py --project-id <project-id> --hours 24
  ```

## Agent Status and Health

### 1. Diagnose Agent API and Dashboard Exposure

- Verify pod running status and details:
  ```bash
  kubectl get pods -n kubeagents-system -l app=<agent-name>-gateway -o wide
  ```
- Inspect Service configurations for the API port (`8642`, which targets the credential proxy on `8643`) and Dashboard port (`9119`):
  ```bash
  kubectl get service <agent-name> -n kubeagents-system -o yaml
  ```
- Probe the dashboard listener from inside the pod, not over `kubectl port-forward`. The dashboard
  binds `127.0.0.1`, and on a GKE Sandbox (gVisor) node pool — the install default — a port-forward
  lands in the host-side netns and never reaches the sandbox's listener, so it reports a refusal
  whatever the dashboard is doing:
  ```bash
  kubectl exec <pod-name> -c <agent-container-name> -n kubeagents-system -- \
    curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9119
  ```
- For a browser session against the dashboard, use `hermes-dashboard-tunnel.py` from the
  repository's own `scripts/` at its root — not this skill's `scripts/`, which does not carry it.
  It relays through `kubectl exec`, which does enter the sandbox.

### 2. Inspect Persistent Internal State & Memory

- Inspect the agent's active memory files and settings:
  ```bash
  kubectl exec <pod-name> -c <agent-container-name> -n kubeagents-system -- ls -la /opt/data/memory/
  kubectl exec <pod-name> -c <agent-container-name> -n kubeagents-system -- cat /opt/data/memory/heartbeat-state.json
  ```