evaluate · diff

git:20261016.4915c25 to git:20261016.266342a

25 added, 137 removed. Audit A to A.

What this update changes in the checks

No check changed: every fixed check gives the same answer before and after this update.

A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.

---
name: evaluate
description: "Use when the user wants proof that finished work meets its acceptance criteria, not just that tests pass: 'verify it is actually done', 'check it against the spec', 'evaluate the run'. Runs mechanical checks, a semantic review per criterion, and multi-model consensus when uncertain. With execution.auto_evolve on (the default), a rejected result with a Seed continues into a Ralph fix loop; pass auto_evolve: false to verify only."
aliases: [eval]
---
# /ouroboros:evaluate
- Evaluate an execution session using the three-stage verification pipeline.
-
- ## Usage
+ Evaluate an execution session with the three-stage verification pipeline.
```
/ouroboros:evaluate <session_id> [artifact]
```
- **Trigger keywords:** "evaluate this", "3-stage check"
-
- ## How It Works
-
- The evaluation pipeline runs three progressive stages:
-
- 1. **Stage 1: Mechanical Verification** ($0 cost)
- - Lint checks, build validation, test execution
- - Static analysis, coverage measurement
- - Fails fast if mechanical checks don't pass
- - The only stage that can grant approval: at least one configured check must run and all must pass
-
- 2. **Stage 2: Semantic Evaluation** (Standard tier, advisory)
- - AC compliance assessment
- - Goal alignment scoring
- - Drift measurement
- - Reasoning explanation
- - Can withhold approval and supply feedback; cannot grant it
+ ## Stage authority
- 3. **Stage 3: Multi-Model Consensus** (Frontier tier, optional, advisory)
- - Multiple models vote on approval
- - Only triggered by uncertainty or manual request
- - A rejection withholds approval; an approval cannot grant it or lift a Stage 2 block
+ 1. **Mechanical** (lint, build, test, static analysis, coverage; no model cost): the only stage that can grant approval. At least one configured check must run and all must pass.
+ 2. **Semantic** (AC compliance, goal alignment, drift): advisory. Can withhold approval and give feedback; cannot grant it.
+ 3. **Consensus** (multi-model, only on uncertainty or request): advisory. A rejection withholds approval; an approval cannot grant it or lift a Stage 2 block.
- Without an executed Stage 1 check the outcome is `acceptance_state: unverified`
- (not approved), with the model review attached as feedback.
+ Without an executed Stage 1 check the outcome is `acceptance_state: unverified` (not approved), with the model review attached as feedback.
## Instructions
- When the user invokes this skill:
-
- ### Load MCP Tools (Required first)
-
- The Ouroboros MCP tools are often registered as **deferred tools** that must be explicitly loaded before use. **You MUST perform this step before proceeding.**
-
- 1. Use the active runtime's tool-discovery capability to find and load the evaluate MCP tools:
- ```
- tool discovery query: "+ouroboros evaluate"
- ```
- 2. The tool will typically be named `mcp__plugin_ouroboros_ouroboros__ouroboros_start_evaluate` (with a plugin prefix). After runtime tool discovery returns, the tool becomes callable.
- 3. If the tool is callable — already exposed, or loaded by discovery — proceed with the MCP-based evaluation below. An empty discovery result for an already-exposed tool is expected, not a failure. Skip to the **Fallback** section only if the tool is genuinely absent (no Ouroboros MCP server).
-
- **IMPORTANT**: Do NOT skip this step. Do NOT assume MCP tools are unavailable just because they don't appear in your immediate tool list. They are almost always available as deferred tools that need to be loaded first.
-
- **CRITICAL — deferred-schema guard (prevents "Invalid tool parameters"):**
- This skill can call `ouroboros_start_evaluate` after a fresh turn. A deferred tool's
- schema loaded on one turn is NOT guaranteed to still be loaded on the next. If
- you call it while its schema is not loaded in the **current** turn, the runtime
- rejects the call with **"Invalid tool parameters"** before it reaches the server.
- Therefore: **immediately before EVERY `ouroboros_start_evaluate` call in this skill,
- re-run `tool discovery query: "+ouroboros evaluate"`** (idempotent — a no-op when
- already loaded). If the load returns no matching tool (and the tool is not already callable — an empty load for an already-exposed tool is an expected no-op, not absence), switch to the documented
- fallback instead of retrying the failing call.
-
- ### Evaluation Steps
-
- 1. Determine what to evaluate:
- - If `session_id` provided: Use it directly
- - If no session_id: Check conversation for recent execution session IDs
+ ### Loading MCP tools (deferred-schema guard)
- 2. Gather the artifact to evaluate:
- - If user specifies a file: Read it with Read tool
- - If recent execution output exists in conversation: Use that
- - Ask user if unclear what to evaluate
+ Ouroboros tools may be deferred, and a schema loaded on one turn can unload by the next. Immediately before each `ouroboros_*` call, load it with the active runtime's tool-discovery capability: `"+ouroboros evaluate"` for `ouroboros_start_evaluate`; `"+ouroboros job"` for `ouroboros_job_wait`, `ouroboros_job_status`, and `ouroboros_job_result`. A call whose schema is not loaded fails with "Invalid tool parameters". If the tool is already callable, discovery is a no-op and an empty result is expected. Use the fallback only when the tool is genuinely absent.
- 2.5. Acting verification — reproduce and OBSERVE (do not skip for behaviour-bearing work):
- Stage 1 already runs mechanical checks (build/test). Go further when the runtime
- exposes acting tools — **computer-use / browser, `Bash`/shell, file reads**: don't
- just reason over the diff, **run the result and observe the real effect** (the
- command's output, the endpoint's response, the rendered UI via a screenshot). Do
- it via a dedicated verification sub-agent to keep the main session lean — or
- inline in the main session where the runtime restricts sub-agent spawning (the
- observation is what matters; the delegation is only an optimization). Probe the
- acceptance criteria against the ACTUAL observable behaviour and the adversarial
- classes (`misleading_output`, `hung_command`, `stale_state`, `dirty_worktree`, …).
- Feed the captured evidence (commands, outputs, artifact paths) into the evaluate
- call as part of the `artifact`. If acting tools are unavailable, note that
- behaviour was not observed and evaluate on the text alone.
+ ### Steps
- 3. Call the background `ouroboros_start_evaluate` MCP tool so rejected verdicts
- can continue through the configured Ralph convergence chain:
+ 1. **Session**: use the given `session_id`, else the most recent execution session id in the conversation.
+ 2. **Artifact**: a file the user names (read it with the active runtime's `inspect_code` capability), else recent execution output; ask with `ask_user` if unclear.
+ 2.5. **Acting verification** (behavior-bearing work): Stage 1 already runs build/test, so go further when the runtime can act. Run the result and observe the real effect (command output with `run_shell`, the endpoint's response, the rendered UI) instead of reasoning over the diff, ideally in one verification child via `orchestrate_subagents`, or inline when spawning is restricted. Probe the ACs against observed behavior and the adversarial classes (`misleading_output`, `hung_command`, `stale_state`, `dirty_worktree`, ...). Put the captured evidence (commands, outputs, artifact paths) into `artifact`. If the runtime cannot act, say behavior was not observed.
+ 3. **Start** the background evaluation so a rejected verdict can continue through the configured Ralph chain:
```
Tool: ouroboros_start_evaluate
Arguments:
session_id: <session ID>
artifact: <the code/output to evaluate, plus observed-behaviour evidence from 2.5>
seed_content: <original seed YAML, if available>
acceptance_criterion: <specific AC to check, optional>
artifact_type: "code" (or "docs", "config")
working_dir: <absolute project root, recommended>
trigger_consensus: false (true if user requests Stage 3)
auto_evolve: <optional override; omit to use execution.auto_evolve>
```
-
- `working_dir` controls both Stage 1 command execution and Stage 2 source-file visibility. Pass the absolute project root whenever available; if omitted, the MCP handler falls back to the registered brownfield default, seed project metadata, then the MCP server cwd.
-
- 4. Observe the returned evaluation job. If its terminal result contains
- `chained_ralph_job_id`, follow that Ralph job to terminal before presenting
- the convergence outcome. A missing Seed produces
- `chained_ralph_skipped: seed_unavailable`; preserve the rejected verdict and
- explain that automatic continuation was safely skipped. In OpenCode plugin
- mode, `auto_evolve=true` intentionally returns this pollable parent-owned job;
- with automatic evolution disabled, the plugin child remains the terminal
- surface and `job_id` is `None`.
-
- 5. Present results clearly:
- - Show each stage's pass/fail status
- - Highlight the final approval decision
- - If rejected, explain the failure reason
- - Suggest fixes if evaluation fails
- - Always end with a state breadcrumb based on the outcome:
- - **APPROVED**: `◆ Evaluation approved → next: accept, or ooo evolve to iteratively refine`
- - **REJECTED at Stage 1** (mechanical, `code_changes_detected: true`): `◆ Current state → next: Fix the build/test failures above, then ooo evaluate — or ooo ralph for automated fix loop`
- - **REJECTED at Stage 1** (mechanical, `code_changes_detected: false`): `◆ Current state → next: Run ooo run first to produce code, then ooo evaluate`
- - **REJECTED at Stage 2** (semantic): `◆ Current state → next: ooo run to re-execute with fixes — or ooo evolve for iterative refinement`
- - **REJECTED at Stage 3** (consensus): `◆ Current state → next: ooo interview to re-examine requirements — or ooo unstuck to challenge assumptions`
- - **NOT APPROVED (unverified)** (`acceptance_state: unverified`, no executed check): `◆ Current state → next: add executable checks to .ouroboros/mechanical.toml (or run ouroboros detect), then ooo evaluate; the semantic review above is feedback, not a verdict`
-
- ## Fallback (No MCP Server)
-
- If the MCP server is not available, use the `ouroboros:evaluator` agent to perform a prompt-based evaluation:
-
- 1. Delegate to `ouroboros:evaluator` agent
- 2. The agent performs qualitative evaluation based on the seed spec
- 3. Results are advisory (no numerical scoring without Python core)
-
- ## Example
-
- ```
- User: /ouroboros:evaluate sess-abc-123
-
- Evaluation Results
- ============================================================
- Final Approval: APPROVED
- Highest Stage Completed: 2
-
- Stage 1: Mechanical Verification
- [PASS] lint: No issues found
- [PASS] build: Build successful
- [PASS] test: 12/12 tests passing
+ `working_dir` sets both Stage 1 command execution and Stage 2 source visibility; pass the absolute project root whenever known.
+ 4. **Observe the job**: the start result carries the observer contract in `response.meta.job_observer` and in a final `<!-- ouroboros-job-observer-v1 base64 ... -->` content sentinel. If the structured meta is missing, recover the contract from the sentinel and fail closed unless there is exactly one sentinel at the end of the text, its bounded payload decodes to an object whose only key is `job_observer`, that contract passes canonical v1 validation (it equals, field for field, the v1 contract the server builds from its own `job_id`, `cursor`, `session_id`, `execution_id`, and `follow_result_job_keys`), its job identity matches the visible start receipt, and it does not differ from any structured contract; never rebuild it from visible IDs. With a valid contract, hand it unchanged to exactly one read-only observer child; its instructions already follow `chained_ralph_job_id` through `follow_result_job_keys`, and the main session must not poll the same job. Without a child session, catch up with `ouroboros_job_status` / `ouroboros_job_result` on the next turn or an explicit status request, and wait live with `ouroboros_job_wait` only when the user asked to watch. Present the convergence outcome only after any chained Ralph job is terminal. `chained_ralph_skipped: seed_unavailable` means no Seed was available: keep the rejected verdict and say continuation was safely skipped. In OpenCode plugin mode with automatic evolution disabled, the plugin child is the terminal surface and `job_id` is `None`.
+ 5. **Present** each stage's result, the final approval decision, and on rejection the reason and suggested fixes. Pick the next action from the outcome:
+ - approved: accept, or `ooo evolve` to refine further
+ - Stage 1 rejected with `code_changes_detected: true`: fix the build/test failures, then `ooo evaluate`, or `ooo ralph` for an automated fix loop
+ - Stage 1 rejected with `code_changes_detected: false`: `ooo run` first to produce code
+ - Stage 2 rejected: `ooo run` with fixes, or `ooo evolve`
+ - Stage 3 rejected: `ooo interview` to re-examine requirements, or `ooo unstuck`
+ - `acceptance_state: unverified`: add executable checks to `.ouroboros/mechanical.toml` (or run `ouroboros detect`), then `ooo evaluate`; the semantic review is feedback, not a verdict
- Stage 2: Semantic Evaluation
- Score: 0.85
- AC Compliance: YES
- Goal Alignment: 0.90
- Drift Score: 0.08
+ ### Fallback (no Ouroboros MCP server)
- ◆ Evaluation approved → next: accept, or `ooo evolve` to iteratively refine
- ```
+ Delegate to the `ouroboros:evaluator` agent for a qualitative, advisory evaluation against the Seed (no numerical scoring and no approval authority without the Python core).
## RFC #1392 State Breadcrumb Footer
- Your final response MUST end with exactly one breadcrumb footer line:
-
- ```
- ◆ <current state> → next: <recommended action>
- ```
-
- Derive `<current state>` from live session state via `ouroboros_session_status` when that MCP projection is available; otherwise derive it from this skill's actual outcome. Never use a linear `Step N of M` footer because Ouroboros is an evolutionary loop. When the next action is genuinely a choice, list 2-3 honest options in the `next:` clause. The breadcrumb line must be the last line of the response.
+ End the final response with exactly one line `◆ <current state> → next: <recommended action>`. Derive the state from live session state via `ouroboros_session_status` when available, otherwise from this skill's actual outcome. Never use a linear `Step N of M` footer; when the next step is a real choice, list 2-3 options.