eval-watch git:20260928.b543d27A

Immutable. This exact content never changes and is served at /api/v1/blob/3d8686f3dfe6d7c0.

---
name: eval-watch
description: >-
  Follow a kcap import that setup just started and the evals landing on it —
  "Follow my kcap import" (the prompt `kcap setup` hands to your agent), "watch my
  import", "how is my import going", "are my evals done yet". Reads the handoff file
  setup wrote, polls Capacitor analytics for exactly the sessions that import
  created, waits for the sessions that imported while the user watched to be
  evaluated, summarizes their evals, links to the results and offers the guided
  tour. Not for browsing evals in general — that is the analytics or guided-tour
  skill.
---

# Eval watch

You follow ONE import: the sessions listed in a handoff file `kcap setup` wrote. Every query you
issue names only those sessions — never a repo-wide or tenant-wide read, and never a query without
an `IN (...)` or `session_id = '...'` bound to this cohort. If a response ever includes a row for a
session id outside the cohort, discard that row: never count it, link it, or mention it.

## The safety rule — read this first

Three grammars gate everything below:
- **Run id** (a `(run: <id>)` token, and the file's own `run_id`): `^[0-9a-f]{32}$`.
- **Session id** (`session_ids`, `foreground_succeeded_ids`, and any id you place in a query):
  `^[A-Za-z0-9_-]{1,128}$`.
- **Repo hash** (the `repo_hash` field an analytics response returns, used only in the repo-scoped
  link form): `^[0-9a-f]{16}$`. A `repo_hash` that fails this never reaches a link — fall back to
  the no-repo-hash link form (section 7).

The `kcap-analytics` MCP takes raw SQL text with no parameter binding, so the session-id grammar is
the SQL-safety boundary, not just a format check. The class excludes quote, backslash, whitespace
and `;`, so a value that matches it can never close a string literal or open a new statement — a
matching id is emitted as a single-quoted literal (`'<id>'`) with **no escaping**, because none is
needed and none should be attempted. A run of hyphens such as `--` matches the grammar and is safe
to emit this way: inside a quoted literal it is data, not a SQL comment. An id that fails either
grammar is **never** placed in SQL, a path, or a link — reject it and move on (section 2 below says
what each layer's rejection does); never sanitize or escape an id into something that would pass.

## 1. Find the run

- If the prompt carries a line `(run: <id>)`, `<id>` must match `^[0-9a-f]{32}$` before you use it
  as a filename component. If it does not match, treat the prompt as carrying no run id at all. If
  it does match, the run is bound: read ONLY `<config>/import-handoff-<id>.json`, where `<config>`
  is `$KCAP_CONFIG_DIR` when set, else `~/.config/kcap`, and check it against every Layer A rule in
  section 2, plus one more: its `run_id` field must equal `<id>` exactly. Missing, unreadable, or
  failing any of those checks → say so and CLOSE (section 7) with no query. **Never fall through to
  another run's file** — a different run's cohort does not belong to the run the user asked for.
- Otherwise list `<config>/import-handoff-*.json`, keep the locator-valid ones (section 2) that are
  watchable (`handoff_offered` true, or `handoff_suppressed` is `skill_not_installed` or
  `no_agent_detected`), pick the newest `written_at`, and say which others you passed over.
- Nothing qualifies → say you found no import to follow and CLOSE with no query.

## 2. Validate the file — three layers

A failure at a lower layer never re-opens the file for a higher one.

**Layer A** (may this file be selected at all): JSON parses; `schema_version` is 1; `run_id` matches
`^[0-9a-f]{32}$`; `written_at` parses as ISO-8601 and is within the last 24 hours; `handoff_offered`
is a boolean. Fail → not selectable; try the next candidate file, or close with no query if none
remain.

**Layer B** (may its ids drive data): `cohort` ∈ {exact, partial_exact, unknown}; `background` ∈
{not_needed, running, exited_zero, failed}; `foreground_certainty` ∈ {complete, incomplete};
`handoff_suppressed` is null or one of import_failed, no_new_sessions, nothing_landed,
analytics_not_in_plan, skill_not_installed, no_agent_detected; every entry of `session_ids` and
`foreground_succeeded_ids` matches `^[A-Za-z0-9_-]{1,128}$`; `unattributed_on_disk` is a
non-negative integer. Fail → **no data**: the file stays selected, but say the record is unreadable
and CLOSE with links only (section 7), no query.

**Layer C** (may it supply links): `server_url` starts with `http://` or `https://`. Fail → no
file-sourced links (fall back to `kcap whoami` per section 7) and no query — an unverifiable
`server_url` also means the server-binding check in section 4 cannot run. `background_log` and
`profile` are shown as plain text — never as a link or a command to run — only when they contain no
control characters and are under 512 characters; otherwise omit them.

Never splice an unvalidated value into SQL, a path, or a link.

## 3. No-handoff files

If `handoff_offered` is false, branch on `handoff_suppressed` and CLOSE:
- `no_new_sessions` → "nothing to watch — that import found no new sessions".
- `nothing_landed` → "the sessions that import selected were skipped at import time"; point at
  `kcap import --all` for per-session reasons; no retry.
- `analytics_not_in_plan` → the plan sentence (section 6); the import ran; no retry.
- `import_failed` → "that import did not get running"; name `kcap import --all --yes` and the
  `background_log` when displayable.
- `skill_not_installed` / `no_agent_detected` → the import ran and you evidently have the skill
  now: continue as if offered, watching the file's cohort as usual (sections 4–5).

## 4. Bind to the server — fail closed

If the selected file's `cohort` is `unknown`, skip this check and go straight to section 5 — there
is no cohort to query either way, and the mismatch remediation below would be misleading when
watching was never possible.

Otherwise, run `kcap whoami` and compare its server URL to `server_url`: lowercase scheme and host,
drop a default port (80 for http, 443 for https only), trim a trailing slash, compare the path
as-is. Match → continue to section 5. Mismatch or `whoami` failure → issue **NO** analytics query.
CLOSE with the file's links and this remediation: start your agent from a shell where `kcap whoami`
reports `<server_url>` — set `KCAP_PROFILE=<profile>`, unset `KCAP_URL`, and set `KCAP_CONFIG_DIR`
if kcap uses a custom config directory — then prompt again.

## 5. Watch the cohort

Cohort = `session_ids` (`cohort: unknown` or empty → CLOSE with links, no query). Foreground =
the entries of `foreground_succeeded_ids` that are also in `session_ids`: the sessions that
imported while the user watched setup, and the ones this watch waits on. K is the foreground
count. A foreground entry missing from `session_ids` is never queried; when any are missing, say
"M of the sessions that imported first are outside the 500 this watch can query". Open with:
"Watching N sessions from this import; waiting for the K that imported first to be evaluated";
for `partial_exact` add "the sessions that imported first and the most recent others, 500 at
most; older ones may land and evaluate unobserved"; if `unattributed_on_disk` > 0 add the
sentence from section 7.

Every query uses `query_analytics` with `scope: 'global'` (the MCP defaults to the caller's cwd
repo and will not widen on its own), **ONE call at a time — never two in flight** — and names only
cohort ids, each already validated against the grammar above, as single-quoted literals in an
`IN (...)` list or `session_id = '...'`. If a response ever includes a row whose `session_id` is
not in `session_ids`, drop that row before using the response for anything.

**Request budget:** at most 20 cohort queries per poll. `floor = ceil(N / 20)`. The FIRST poll uses
batch size `floor` exactly — no probing at a smaller size first. Every batch passes `max_rows`
equal to its own size:

    SELECT s.session_id, s.repo_hash, s.vendor, s.model, s.duration_min, s.event_count,
           e.eval_run_id, e.evaluated_at, e.overall_score, e.judge_model
    FROM v_an_sessions s LEFT JOIN v_an_eval_summaries e ON e.session_id = s.session_id
    WHERE s.session_id IN ('<id>', '<id>', ...)

A row = arrived; a non-null `eval_run_id` = completed. Each response body carries `max_rows`, the
server's effective cap. After the first poll: if any body reported `max_rows < floor`, fail closed
at once — say the server's row cap is below what watching this many sessions needs, and CLOSE,
having spent no more than the 20-query budget. Otherwise raise the batch to `min(100, max_rows)`
for later polls. Any batch that comes back `truncated: true`, in the first poll or any later one,
fails that poll — the cap decides, not the truncation flag alone.

One poll = every batch it needs, each non-truncated. A failed or truncated batch discards the whole
poll — nothing from it commits — and counts as one failure toward the two-failure stop rule.

A 429 fails the poll. If its text matches `retry after (\d+)s`, wait that many seconds before the
next poll; otherwise wait 60 seconds. Never poll sooner than the 30-second cadence either way. HTTP
403 with `analytics_not_in_plan` → CLOSE immediately with the plan sentence (section 6); no
polling, no retry, no wait.

Progress = cohort ids that arrived / cohort size, and foreground ids evaluated / K, both computed
from the batch rows only — never a separate count, never repo- or tenant-wide. Cadence: first
snapshot immediately, then every 30 seconds. Say the progress after each poll.

**Baseline**: the first snapshot's rows. An eval present there is *pre-existing*; one whose
`eval_run_id` was null or absent there and non-null in a later poll *completed during this watch*.
Every summary labels each session with one of those two phrases.

**Stop**, evaluated in this order after each poll whose results have already committed:
1. K > 0 and every foreground session has a completed eval → summarize all of them, ordered by
   `evaluated_at` then `session_id`. A pre-existing eval counts: the wait is for these sessions to
   be evaluated, whenever that happened.
2. K = 0 and three distinct cohort sessions completed during this watch → summarize those three,
   ordered by `evaluated_at` then `session_id`. Pre-existing evals never satisfy this rule: on a
   re-import they belong to an earlier run, and stopping on them would end the watch before this
   import produced anything.
3. `cohort: exact`, non-empty, and every id in `session_ids` has a completed eval → summarize the
   foreground sessions, or with K = 0 the three with the newest `evaluated_at` (ties by
   `session_id`). This rule may fire on the first snapshot with nothing but pre-existing evals:
   a cohort that is already fully evaluated has nothing left to wait for, and the labels say so.
   Never fires for `partial_exact` or an empty cohort.
4. Two consecutive failed polls → stop, reporting the failures.
5. 10 minutes since the first snapshot → stop. No new poll starts after the deadline, but a poll
   already dispatched before it finishes normally, and its outcome is still checked against rules
   1–4 first. On a deadline stop, summarize the foreground sessions that did complete and name the
   ones still pending, with whether each has arrived; if none of them completed, summarize instead
   up to three cohort sessions with the newest `evaluated_at` (ties by `session_id`).

No idle early-stop — quiet polls are normal and are not a reason to stop early. Evals take minutes
per session, and the foreground sessions are the newest, so they are usually the last to complete.

**Per-session detail** (one session per query, 2 queries per detailed session, at most 10 across
the whole run). A foreground pass takes whole session chains, so K can exceed five: the first five
sessions in summary order get detail; every later one gets a one-line entry — link, label and
`overall_score` — and no detail query. Both queries pass `max_rows` explicitly — a query left at
the agent's guess truncates on the category list and throws the detail away:

    -- max_rows: 25
    SELECT category, ROUND(AVG(score), 2) AS mean FROM v_an_eval_scores
    WHERE session_id = '<id>' AND outcome = 'assessed' GROUP BY category
    -- max_rows: 2
    SELECT question_id, score FROM v_an_eval_scores
    WHERE session_id = '<id>' AND outcome = 'assessed' ORDER BY score ASC, question_id ASC LIMIT 2

Strongest = highest mean, weakest = lowest, ties alphabetical by category. A truncated or failed
detail query, or a session with no assessed rows, → summarize that session by `overall_score` alone
and say so; this never affects poll success or the stop rules above.

**What a detailed session summary says**, one short paragraph each: the link (section 7), the
baseline label, then vendor and model, length (`duration_min`, rounded, and `event_count`),
`overall_score` out of 5 and the judge model, strongest and weakest category with their means, and
the two weakest questions with their scores. When the `kcap-sessions` MCP is available, call `get_session_summary` once per
summarized session and lead with its title or first sentence when it returns one; an empty summary
is left out silently, never called "no summary".

**Cohort facts**, one line after the summaries, from the last committed poll's rows only: sessions
arrived out of N, sessions evaluated, and the mean `overall_score` across the evaluated ones.

## 6. When little or nothing completes

Explain the real gates plainly: auto-eval may be off for a repository; the server needs an eval
agent configured; sessions under the minimum event count are skipped; already-evaluated sessions
are not re-run; a recovery sweep runs hourly. If the analytics MCP is absent entirely, close
immediately with the block below.

Plan sentence (used here and from sections 3 and 5): "Insights isn't in this workspace's plan, so
evals can't be followed from here; your import continues and evals appear in the Capacitor UI."

## 7. CLOSE — always

1. How to keep watching: prompt `Follow my kcap import` again (with the same `(run: <id>)` line
   when you were bound to one).
2. Links: with a `repo_hash` matching `^[0-9a-f]{16}$`, `<server_url>/repo/<repo_hash>/sessions/<session_id>?tab=evaluation`;
   otherwise (no `repo_hash`, or one that fails the grammar) `<server_url>/sessions/<session_id>?tab=evaluation`;
   all results `<server_url>/sessions`. No valid file → `server_url` from `kcap whoami`; if that
   fails too, say the results live in the Capacitor server UI and that `kcap whoami` prints the URL
   once logged in.
3. When `unattributed_on_disk` > 0: "N sessions on disk had no repository match; any of them that
   imported show without one — `kcap remap` places them."
4. Offer the guided tour with the exact prompt `Start kcap guided tour`.