meta-planning-ai-planning · diff
git:20260809.786f3c8 to git:20261016.0b768f9
192 added, 253 removed. Audit B to A.
---
name: meta-planning-ai-planning
- description: AI specification planning frameworks. Use when a spec touches model calls, prompts, retrieval, tool calling, agentic loops, or evals. Covers approach selection, model and provider choice, structured output contracts, loop guards, budgets, failure modes, and eval design.
+ description: Specification frameworks for features that call a model — approach tier, model pinning, output contracts, budgets, failure modes and eval design. Load when a spec touches model calls, prompts, retrieval, tool calling, agent loops or evals.
---
# AI Planning Frameworks
- > **Quick Guide:** Default to the simplest tier that satisfies the requirement — most features are one well-built model call. Pin the model id, define the output contract with a repair-vs-reject policy, budget in tokens and money rather than adjectives, enumerate the failure modes, and make quality measurable with an eval plan before implementation starts. Apply a framework only when the spec touches its artifact class — a feature with no retrieval needs no retrieval section.
+ > **Quick Guide:** Default to the lowest tier that meets the requirement — most features are one well-built call — and name the requirement behind every tier above it. Pin the model id, give every output a contract the code checks on receipt, state budgets in tokens and money, give each failure the feature can hit a row, and fix eval thresholds before the first run. Two provider facts shape the answers: a schema-constrained output mode guarantees shape only for a response that completed and was not refused, so the contract's remaining work is truncation, refusals and values; and reasoning tokens bill as output and count against the output ceiling, so a ceiling sized for the visible answer can run out before one appears.
- ---
+ **Detailed Resources:**
- <critical_requirements>
+ - [examples/core.md](examples/core.md) — templates for the sections every model-calling feature needs, and a worked specification
+ - [examples/retrieval-and-agents.md](examples/retrieval-and-agents.md) — retrieval design, tool contracts and loop guards
+ - [reference.md](reference.md) — output-mode guarantees, budget line items, default failure behaviour, retrieval starting points, grading methods, eval set sizing
- ## CRITICAL: Before Specifying AI Features
+ ---
- > **All specifications must be grounded in the codebase's real model clients, prompt modules, schemas, and eval fixtures** — reference specific files with line numbers
+ ## Which path applies
- **(You MUST justify the approach against the simpler tier — a fixed code-orchestrated chain beats an agentic loop whenever the step sequence is known)**
+ A section belongs in the spec only when the feature has its artifact; one filled in for completeness reads to the implementer as something to build.
- **(You MUST pin an explicit model id in configuration with a named fallback — never a floating alias, never inline in code)**
+ - **The feature calls a model** — approach, model, prompt, output contract, budgets, failure modes and eval plan; follow [examples/core.md](examples/core.md).
+ - **It retrieves from a corpus** — add the retrieval section from [examples/retrieval-and-agents.md](examples/retrieval-and-agents.md#pattern-8-retrieval-design).
+ - **The model calls tools, or orders its own steps** — add tool contracts and loop guards from [examples/retrieval-and-agents.md](examples/retrieval-and-agents.md#pattern-9-tool-contracts-and-agentic-loops).
- **(You MUST define the output contract completely: mechanism, schema, validation boundary, and a repair-vs-reject policy)**
+ ---
- **(You MUST state budgets as numbers — tokens per call, calls per request, cost per request, p95 latency — never as adjectives)**
+ <critical_requirements>
- **(You MUST identify where untrusted input enters every prompt, and require adversarial eval cases wherever it does)**
+ ## Before specifying an AI feature
- **(You MUST apply each framework only when the spec touches its artifact class — an unused section is omitted, never filled)**
+ **Name the requirement that rules out the simpler tier before specifying a chain, retrieval or a loop.** Each tier up adds cost, latency and failure rows, and a known step sequence run as a loop buys non-determinism and no capability.
- </critical_requirements>
+ **Pin the model id in configuration, name its fallback, and list what forces a re-eval.** A floating alias changes quality with no deploy, and a fallback nobody named becomes an unhandled exception on the first 5xx.
- ---
+ **Give every model output a contract — mechanism, schema, the validation that runs on receipt, and a repair-or-reject rule.** Provider-side enforcement covers shape for a completed, unrefused response and nothing more, so truncation, refusals and wrong values reach the consumer unless the spec stops them.
- **Auto-detection:** AI spec, LLM feature spec, prompt design spec, model selection, RAG spec, retrieval design, tool calling spec, agent loop spec, eval plan, token budget, structured output
+ **State budgets as numbers, counting reasoning tokens as output and cached input at its cached rate.** An adjective cannot be implemented or reviewed, and a cost line that leaves out either is wrong — low without the reasoning, high without the cache.
- **When to use:**
+ **Mark every untrusted input and bound what it can reach.** Delimiting lowers the odds that an injected instruction is followed without preventing it, so the prompt gets adversarial eval cases and its tools get only the access the task needs.
- - Specifying a feature that calls a model (classification, extraction, generation, summarization)
- - Choosing between a single call, a fixed chain, retrieval, and an agentic loop
- - Specifying prompt architecture, versioning, and untrusted-input boundaries
- - Specifying structured output contracts and validation policy
- - Specifying retrieval parameters, tool contracts, or loop guards
- - Specifying eval datasets, metrics, thresholds, and budgets
+ </critical_requirements>
- **When NOT to use:**
+ ---
- - When implementing AI code (use the relevant AI implementation skill)
- - For the API surface that exposes the capability (use the api planning skill)
- - For the planning PROCESS itself — research, scope fencing, success criteria — which the PM agent carries
+ **Auto-detection:** AI feature spec, LLM feature spec, model selection, pinned model id, model fallback, prompt version, system prompt, untrusted input, prompt injection, structured output contract, repair or reject, refusal handling, token budget, reasoning tokens, prompt caching, retrieval design, chunk size, re-ranking, tool contract, agent loop, step budget, eval plan, eval threshold, LLM-as-judge
- **Key patterns covered:**
+ **Applies to:**
- - Approach selection (single call → chain → retrieval → agentic loop)
- - Model and provider selection with pinned ids and fallback chains
- - Prompt architecture: role split, versioning, untrusted-input boundaries
- - Structured output strategy and repair-vs-reject policy
- - Retrieval design parameters
- - Agentic loop guards and tool side-effect classification
- - Eval design and adversarial cases
- - Failure-mode matrix
- - Token, cost, and latency budgets
+ - Choosing between a single call, a fixed chain, retrieval and an agentic loop, and recording why the simpler tier lost
+ - Model selection, prompt architecture and the output contract for each model call
+ - Token, cost and latency budgets, and the behaviour for each way a call fails
+ - Retrieval parameters, tool contracts and loop guards
+ - Eval sets, metrics and the thresholds that gate a merge
- **Detailed Resources:**
+ **Handled elsewhere:**
- - [examples/core.md](examples/core.md) - Per-artifact spec section templates and a worked example specification
+ - The spec's general shape — research, goal, scope fence, success criteria — belongs to the planning process; this skill adds the sections a model call brings
+ - Provider SDK calls — parameter names, client setup, streaming plumbing — are the implementation's; the spec names the mechanism, not the call
+ - The interface that exposes the feature — request and response shapes, error statuses — is settled with that interface; the failure matrix here says what the feature does, not how the interface reports it
---
<philosophy>
## Philosophy
- **Non-determinism is the material; contracts are what make it buildable.** A model's output cannot be trusted by construction, so every boundary — schema, budget, failure behavior, eval threshold — must be decided in the spec, or it gets decided implicitly in production.
-
- **When specifying AI work:**
-
- - Trace one existing model call end to end before specifying a new one — client, prompt module, validation, consumer
- - Name the requirement that forces each tier of complexity; if nothing does, use the simpler tier
- - Establish the baseline before proposing a change: current cost, latency, and quality numbers, or an explicit "unmeasured"
- - Decide product questions in the spec — may the model answer from parametric knowledge when retrieval is empty? — rather than leaving them to the implementer
-
- **When NOT to specify:**
-
- - Don't specify an agentic loop for a step sequence known in advance
- - Don't add retrieval infrastructure when the knowledge fits in the context window
- - Don't fill a tool-contract or retrieval section for a feature that has neither
- - Don't include implementation code — contracts, budgets, and behaviors only
-
- **Core principles:**
+ **Non-determinism is the material; contracts are what make it buildable.** A model's output cannot be trusted by construction, so every boundary — schema, budget, failure behaviour, eval threshold — is decided in the spec, or production decides it implicitly.
- - **Simplest sufficient tier**: most features are a single well-constructed call
- - **Validate on receipt, every branch**: provider-side enforcement reduces violations; it does not eliminate them
- - **Budgets are numbers**: "fast and cheap" is not implementable or reviewable
- - **Refusals are surfaced, never retried**: retrying with softened wording is a bypass attempt
- - **Eval thresholds gate the merge**: thresholds set after seeing results ratify instead of gating
+ - **Start from the existing call path.** Trace one model call end to end — client, prompt module, validation, consumer — and cite each as the pattern source with its file and lines; the new feature inherits its retries, timeout and logging, and a bare path leaves the implementer to guess which code was meant.
+ - **Record the baseline** — current cost, latency and quality, or an explicit "unmeasured" — so the change has something to beat.
+ - **Decide product questions in the spec**: whether the model may answer when retrieval finds nothing, whether its output acts or only advises.
+ - **Contracts, budgets and behaviours, not code.** Implementation code in a spec gets followed instead of the codebase's own patterns.
</philosophy>
---
- <patterns>
+ <decision_framework>
- ## Core Patterns
+ ## Which tier
- ### Pattern 1: Choosing the Approach
+ Two independent questions, both answered for every feature:
```
- Does the task need knowledge that is not in the model and not in the request?
- ├─ NO → Single model call with a well-built prompt. Stop here. Most features end here.
- └─ YES → Where does that knowledge live?
- ├─ A bounded set that fits the context window (< ~30% of it) → Pass it directly. No retrieval infrastructure.
- ├─ A large or growing corpus → Retrieval (RAG)
- └─ A live system of record (database, third-party API) → Tool calling, not retrieval
+ Where does the knowledge the task needs live?
+ ├─ In the model or the request → nothing to add
+ ├─ A bounded set that fits the window → inline it, cached where the provider caches
+ ├─ A large or growing corpus → retrieval
+ └─ A live system of record → a tool call, not retrieval
- Does the task need multiple dependent actions the model must sequence itself?
- ├─ NO → Single call, or a fixed chain of calls you orchestrate in code
- │ (a fixed chain is cheaper, more debuggable, and easier to eval than a loop)
- └─ YES → Agentic loop with an explicit step budget and termination conditions
+ Who decides the sequence of steps?
+ ├─ There is one step → single call
+ ├─ Several, known in advance → fixed chain orchestrated in code — cheaper, debuggable, evaluable
+ └─ Several, chosen by the model as it goes → agentic loop, with every guard in Pattern 9
```
- **Default to the simplest tier that satisfies the requirement.** Specify the more complex tier only when you can name the requirement that forces it, and record the rejected alternatives with the reason each was rejected.
-
- ---
-
- ### Pattern 2: Model and Provider Selection
+ The answers combine — retrieval feeding a single call, a tool call inside a fixed chain — and most features answer "in the request" and "one step". Specify a higher tier only by naming the requirement that forces it, and record each rejected alternative with its reason. Fitting the window is a limit, not evidence of use: quality falls as input grows well before the limit, so the eval set — not the token count — decides whether inlining holds.
- Score the candidates against the actual requirement and record the table in the spec.
+ ## Which output mechanism
- | Dimension | Question to answer |
- | -------------- | ------------------------------------------------------------------------- |
- | Capability | Does the smallest candidate pass the eval set? Test before assuming not. |
- | Context window | Does the worst-case assembled prompt fit with headroom for output? |
- | Latency | Does the p95 profile fit the surface (streamed UI vs background job)? |
- | Cost | What is the blended cost per request at projected volume? |
- | Structured out | Does the provider support the output mechanism you selected? |
- | Availability | What is the fallback when this provider returns 429 or 5xx? |
- | Versioning | Is the model id pinned? What is the deprecation and re-eval plan? |
- | Data handling | Does request content leave an acceptable boundary? Any retention concern? |
+ ```
+ Is the output an action against a tool or system, a record to consume, or actions followed by a record?
+ ├─ An action → tool calling, with strict (schema-constrained) arguments where the provider offers them
+ ├─ A record → the first the provider offers:
+ │ 1. a schema-constrained output mode
+ │ 2. a single submit tool with strict arguments
+ │ 3. JSON mode — valid JSON, schema unchecked
+ │ 4. delimited text + strict parser, with the widest eval coverage
+ └─ Actions, then a record → one tool per action; the final record through a terminal final-answer tool,
+ or the provider's schema-constrained output on the last turn
+ ```
- **Rules:**
+ **Every branch validates on receipt.** Constrained decoding enforces only the schema subset the provider supports — ranges, lengths and formats outside it move to the validator — on a response that completed and was not refused, and it never checks whether a value is true.
- - Always pin an explicit model id in configuration, never a floating alias, and never inline in code.
- - Always name a fallback, even if the fallback is "fail closed with error X" — an unnamed fallback becomes an unhandled exception.
- - Re-run the eval set when changing model id, prompt version, or retrieval parameters. Any of the three can move quality.
+ </decision_framework>
---
- ### Pattern 3: Prompt Architecture
-
- Every prompt the spec introduces is named, located, versioned, and trust-annotated.
+ <patterns>
- - **Location and version.** A prompt is a module with a version suffix in its filename (`prompts/summarize-ticket.v1.ts`), following the shape of an existing prompt module. The version is logged with every call so eval results attribute to a specific revision. Editing a shipped prompt in place is out of scope; ship the next version instead.
- - **Role split.** Instructions live exclusively in the system message. Task input is interpolated only into the user-role message. Few-shot examples, when used, are assistant-role and trusted.
- - **Untrusted-input boundary.** Name every variable, its source, and whether it is untrusted. Untrusted text is delimited with the codebase's existing fence helper, and a fenced instruction ("ignore your previous instructions") must not change behavior — the eval set includes this case. Retrieved document text is untrusted too.
- - **Variables.** List each variable with its type and source. A variable the template reads but the spec never mentions is a contract gap.
+ ## Core patterns
- ---
+ ### Pattern 1: Approach and Rationale
- ### Pattern 4: Structured Output Strategy
+ The proportionality record: the tier chosen, the requirement that rules out the one below, the alternatives rejected.
- ```
- Does the provider support tool calling / function calling?
- ├─ YES → Is the output a single well-defined record?
- │ ├─ YES → Tool calling with a single submit-style tool. Strongest schema adherence.
- │ └─ NO → Tool calling with one tool per action, plus a terminal final-answer tool
- └─ NO → Does it support a JSON/structured output mode?
- ├─ YES → JSON mode + schema validation on receipt
- └─ NO → Delimited free text + strict parser. Requires the widest eval coverage.
+ ```markdown
+ **Selected approach:** fixed code-orchestrated chain — classify, then summarize long tickets
+ **Why not simpler:** one call cannot skip summarizing the short majority without paying for it
+ **Why not a loop:** the step sequence is known in advance
+ **Why not retrieval:** the label taxonomy fits in the system prompt
```
- **Every branch validates on receipt.** Provider-side schema enforcement reduces violations; it does not eliminate them, and it never covers semantic correctness (a valid-shaped record with a hallucinated value passes schema validation).
-
- **Repair-vs-reject policy — state it explicitly:**
-
- | Situation | Policy |
- | -------------------------------------- | ------------------------------------------------------------------- |
- | Shape violation (missing/typo'd field) | One repair attempt with the validator error appended, then reject |
- | Semantic violation (impossible value) | Reject immediately; repair attempts tend to launder the bad value |
- | Streaming partial JSON | Buffer and validate only on completion; never act on partial output |
- | Repeated failure on the same input | Reject and record the input for eval-set inclusion |
-
- ---
+ Template: [examples/core.md](examples/core.md#pattern-1-approach-and-rationale)
- ### Pattern 5: Retrieval Design
+ ### Pattern 2: Model Selection
- | Decision | Options | How to choose |
- | ------------------ | ------------------------------------------ | ---------------------------------------------------------------------------------- |
- | Chunk size | Small (200-400) vs large (800-1500) tokens | Small for fact lookup; large when answers need surrounding narrative |
- | Overlap | 0 vs 10-20% of chunk size | Overlap when facts straddle boundaries; costs index size |
- | Split boundary | Fixed tokens vs structural (heading, code) | Structural whenever the corpus has reliable structure — it preserves meaning |
- | Retrieval mode | Semantic, keyword, hybrid | Hybrid when queries contain exact identifiers (error codes, SKUs, symbol names) |
- | Top-k | Retrieve k, re-rank to n | Retrieve wide (15-30), re-rank narrow (3-8). Wide-only retrieval dilutes context. |
- | Re-ranking | None vs cross-encoder/model re-rank | Add when precision@5 is the bottleneck; it costs latency |
- | Metadata filtering | Pre-filter vs post-filter | Pre-filter on tenant, locale, or permission — never post-filter access control |
- | Freshness | Batch reindex vs incremental | Match the corpus change rate; state who triggers reindex and how staleness is seen |
+ Score candidates against the requirement — capability on the eval set with the smallest tried first, context, latency, cost, output mechanism, availability, data handling — and record the table.
- **Always specify:** what happens on empty retrieval, what happens on low-similarity retrieval, whether answers must cite sources, and whether the model may answer from parametric knowledge when retrieval returns nothing. That last one is a product decision, not an implementation detail — decide it in the spec.
+ ```markdown
+ **Primary:** [pinned id] — configured at [config file], never inline
+ **Fallback chain:** [id] on 429 / 5xx / timeout, then [documented failure response]
+ **Re-eval trigger:** model id, prompt version or retrieval parameters change; the snapshot retires
+ ```
- ---
+ Pinned snapshots retire on a published schedule, so the spec names who re-runs the eval at migration.
- ### Pattern 6: Agentic Loop Guards
+ Template: [examples/core.md](examples/core.md#pattern-2-model-selection)
- Every loop specification names all six:
+ ### Pattern 3: Prompt Architecture
- | Guard | Specification requirement |
- | --------------------- | ------------------------------------------------------------------------ |
- | Step budget | Hard maximum number of model turns |
- | Token budget | Cumulative input+output ceiling across the whole loop |
- | Wall-clock budget | Total elapsed limit, independent of step count |
- | Termination | Every condition that ends the loop, including the success condition |
- | Repetition guard | Behavior when the model repeats an identical tool call |
- | Partial-result policy | What is returned when a budget is exhausted before the success condition |
+ Every prompt is named, located, versioned and trust-annotated.
- **Tool side effects must be classified.** Every tool is read-only or mutating. Mutating tools need: idempotency strategy, whether they require confirmation, and whether they are permitted on a retry after an ambiguous failure.
+ ```markdown
+ **Module:** `prompts/triage-classify.v1.ts` — version logged with every call
- **State carried between steps** is named exactly — nothing else persists. On step-budget exhaustion, return the partial answer flagged as truncated; never silently return an unfinished result as complete.
+ | Variable | Source | Untrusted? |
+ | -------- | ------------------ | ---------- |
+ | body | inbound ticket | yes |
+ | taxonomy | `config/labels.ts` | no |
+ ```
- ---
+ - **Versioned, never edited in place.** A shipped prompt is superseded by the next version, so eval results attribute to a revision.
+ - **Instructions in the system-level message, task input in the user message.** Untrusted text is fenced with the codebase's existing delimiter helper, and retrieved documents count as untrusted.
+ - **Stable content first, per-request content last.** Provider caches match on an exact prefix, so one per-request value near the top — a timestamp, a user's name — makes every call a miss. When the per-request input is a long document, Claude's [long-context guidance](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices#long-context-prompting) puts it near the top, above the query and instructions, so the spec states which wins — cache hits (the document after the stable prefix) or long-context quality (the document above the query) — and keeps the stable part above both either way.
+ - **Bound what untrusted text can reach.** A feature combining untrusted input, private data and an outbound channel — a tool that sends, a link that renders — drops one of the three or puts a person in front of the action, because no injection detector is reliable enough to make the combination safe.
+ - **List every variable the template reads, with its type and source.** A variable the template reads but the spec never mentions is a contract gap, and the implementer decides its source and its trust.
- ### Pattern 7: Eval Design
+ Template: [examples/core.md](examples/core.md#pattern-3-prompt-architecture)
- | Component | Requirement |
- | ------------ | ------------------------------------------------------------------------------------------------ |
- | Dataset | Concrete location and case count. Include adversarial and empty-input cases, not just happy path |
- | Labels | Who produced the expected outputs, and how disagreements were resolved |
- | Metrics | Named and computable. "Accuracy" alone is not a metric — say accuracy of what, measured how |
- | Thresholds | A number per metric that gates the merge |
- | Grading | Deterministic assertion, rubric grading, or model-graded — state which, and its known weakness |
- | Regression | Which existing evals must continue to pass unchanged |
- | Cost/latency | Measured per run, recorded alongside quality metrics |
- | Provenance | Every eval run records model id, prompt version, and retrieval parameters |
+ ### Pattern 4: Output Contract
- **Include adversarial cases whenever untrusted text reaches the prompt.** At minimum: instruction override, delimiter escape, and exfiltration attempts against any tool the loop can call.
+ The mechanism from the decision framework, the schema with what the provider enforces and what the validator enforces, and the repair-or-reject rule.
- ---
+ ```markdown
+ | Situation | Policy |
+ | ------------------ | ---------------------------------------------------------------------- |
+ | Shape violation | one repair attempt with the validator's error appended, then reject |
+ | Semantic violation | reject at once — a repair attempt tends to launder the bad value |
+ | Truncated | reject, or retry once with a higher ceiling — never parse the fragment |
+ | Refusal | the refusal policy in Pattern 6 — never the repair loop |
+ ```
- ### Pattern 8: Failure-Mode Matrix
+ Streamed partial JSON is buffered and validated on completion; nothing acts on a fragment.
- Enumerate every applicable row for the feature. Absent rows become production incidents.
+ Template: [examples/core.md](examples/core.md#pattern-4-output-contract)
- | Failure | Detection signal | Required behavior |
- | --------------------- | ------------------------ | -------------------------------------------------------------- |
- | Rate limited | 429 / provider header | Backoff with jitter, capped attempts, then fallback provider |
- | Timeout | Elapsed > budget | Abort the call, return the documented timeout error |
- | Provider outage | 5xx / connection error | Fallback chain, then documented degraded response |
- | Malformed output | Schema validation fails | Repair-vs-reject policy from Pattern 4 |
- | Context overflow | Pre-call token count | Shed lowest-priority context; never truncate the system prompt |
- | Empty retrieval | Zero results above floor | Documented no-context behavior; decide answer-vs-abstain |
- | Model refusal | Refusal in response | Surface unchanged; never retry with softened wording |
- | Tool error | Tool throws or errors | Return the error to the model once; escalate on repeat |
- | Runaway loop | Repetition guard trips | Terminate with partial result flagged as truncated |
- | Cost ceiling exceeded | Running cost counter | Terminate and return partial; alert per observability section |
+ ### Pattern 5: Budgets and Observability
- ---
+ Per call and per user request, in tokens, money and milliseconds, with the overflow rule — what is shed first, never the system prompt.
- ### Pattern 9: Budgets and Observability
+ ```markdown
+ | Budget | Ceiling |
+ | --------------------------------- | ------------------------------------ |
+ | Output tokens per call | [reasoning + visible] |
+ | Model calls per user request | [n, including repairs and fallbacks] |
+ | Cost per request | [$, at an assumed cache-hit rate] |
+ | p95 latency / time to first token | [ms] |
+ ```
- **Budget in tokens and money, not adjectives.** Per call: input and output token ceilings. Per user request: maximum model calls, blended cost target at current pricing, p95 end-to-end latency, and time-to-first-token for streamed surfaces. State the overflow behavior: what is shed first when the input budget is exceeded — never the system prompt.
+ Observability names what is logged per call (model id, prompt version, token counts, latency, cost, outcome, correlation id), the trace spans, what is redacted field by field, and what alerts.
- **Observability names four things:** what is logged per call (model id, prompt version, token counts, latency, cost, outcome, correlation id), the trace span boundaries (retrieval, each model call, each tool call), what is redacted (prompt bodies with user data, retrieved content, keys, PII — named field by field), and what alerts (threshold breaches worth notifying).
+ Template: [examples/core.md](examples/core.md#pattern-5-budgets-and-observability)
- </patterns>
+ ### Pattern 6: Failure-Mode Matrix
- ---
+ A row per failure the feature can hit, with its detection signal and required behaviour. Each row becomes a success criterion with a mocked trigger.
- <decision_framework>
+ ```markdown
+ | Failure | Detection | Required behaviour |
+ | ------------ | -------------------------------- | ----------------------------------------------------------------------------- |
+ | Rate limited | 429 with a rate-limit error code | wait out retry-after if sent, else backoff with jitter; capped; then fallback |
+ | Quota spent | 429 with a quota or spend code | fail at once — waiting never clears it |
+ | Incomplete | output ceiling reached | reject; never parse the fragment |
+ | Refusal | the provider's refusal signal | surface it, or a fallback model the provider documents — the spec picks |
+ ```
- ## Decision Framework
+ Default behaviour for every row: [reference.md](reference.md#default-failure-behaviour). Template: [examples/core.md](examples/core.md#pattern-6-failure-mode-matrix)
- ### Which Spec Sections Does This Feature Need?
+ ### Pattern 7: Eval Plan
- Apply a framework only when the spec touches its artifact class. The per-artifact section templates live in [examples/core.md](examples/core.md).
+ A dataset with a location and case count, named metrics with thresholds fixed before the first run, a grading method per metric, the regression set, and the model id, prompt version and retrieval parameters recorded with every run.
- ```
- Does the feature call a model at all?
- ├─ YES → Approach rationale (Pattern 1) + Model Selection (Pattern 2) + Prompt Architecture (Pattern 3)
- │ + Output Contract (Pattern 4) + Budgets (Pattern 9) + Failure Modes (Pattern 8) + Eval Plan (Pattern 7)
- ├─ Does it retrieve from a corpus? → Retrieval Design section (Pattern 5)
- ├─ Does the model call tools? → Tool Contracts section (see examples/core.md)
- ├─ Does the model sequence steps? → Agentic Loop section (Pattern 6)
- └─ None of the above touched → the section is omitted, never filled
+ ```markdown
+ | Metric | Threshold | Grading |
+ | -------------------- | ----------------------------- | ------------------------------------------------ |
+ | Label accuracy | ≥ 0.90 on 400 cases | deterministic |
+ | Injection resistance | every case, in each of 5 runs | deterministic |
+ | Summary groundedness | ≥ 0.95 | model-graded, checked against human raters first |
```
- ### Common Specification Mistakes
-
- | Mistake | Consequence |
- | ----------------------------------------------- | ------------------------------------------------------------------------- |
- | Specifying an agentic loop for a fixed sequence | Non-deterministic, expensive, hard to eval — a code chain was sufficient |
- | Omitting the empty-retrieval case | The model answers from parametric knowledge and the answer looks cited |
- | "Validate the response" with no schema | The developer invents a schema; downstream consumers break on drift |
- | No prompt version in the spec | Eval results cannot attribute to a revision; regressions are untraceable |
- | Floating model alias instead of a pinned id | Silent quality shift when the provider rotates the alias |
- | Budgets stated as adjectives ("fast", "cheap") | No implementable target, no reviewable violation |
- | No adversarial eval cases with untrusted input | Prompt injection ships undetected |
- | Retry on refusals | Wastes budget and reads as an attempt to bypass a safety response |
- | Truncating the system prompt on overflow | Instructions silently disappear; behavior changes without an error |
- | Post-filtering retrieval for access control | Cross-tenant content reaches the model before it is filtered out |
- | Eval thresholds set after seeing results | The gate ratifies whatever shipped instead of gating it |
- | Including implementation code in the spec | The developer follows your sketch instead of the codebase's real patterns |
-
- </decision_framework>
+ - **Size the set so the threshold's interval excludes the failing value** — [reference.md](reference.md#eval-set-size) has the table.
+ - **Run must-pass cases more than once**; one pass of a non-deterministic system is one sample.
+ - **Cover empty and degenerate input, not only the happy path** — empty input is a common way a model call fails in production, and a set of happy-path cases never exercises it.
+ - **Require adversarial cases wherever untrusted text reaches the prompt**: instruction override, delimiter escape, and exfiltration through every tool or channel the call can reach.
- ---
+ Template: [examples/core.md](examples/core.md#pattern-7-eval-plan)
- <red_flags>
+ ### Pattern 8: Retrieval Design
- ## RED FLAGS
+ Chunking, retrieval mode, top-k and re-ranking, access filters and freshness — plus four behaviours the spec decides: empty retrieval, low-similarity retrieval, citations, and whether the model may answer from its own knowledge when retrieval finds nothing.
- **High Priority Issues (a spec with one of these is incomplete):**
+ ```markdown
+ | Parameter | Value |
+ | -------------- | ---------------------------------------------------- |
+ | Split boundary | structural (headings) where the corpus has structure |
+ | Retrieval mode | hybrid — queries carry error codes |
+ | Top-k | retrieve wide, re-rank to the n the eval set picks |
+ | Access filters | tenant and permission, inside the retrieval query |
+ ```
- - A model call with no output schema or validation boundary
- - A floating model alias, or a model id inline in code
- - An agentic loop missing any of the six guards
- - Untrusted input reaching the system message, or fenced input with no adversarial eval case
- - Budgets as adjectives, or no budgets at all
- - No behavior defined for a failure row the feature can hit
+ Access control lives in the query, against a permission-aware store or a partition. A filter applied after the search returns fewer than k results for a small tenant, and a filter asked of the model is not enforcement. Retrieved text is untrusted input, including what an upload or an insider put into the corpus.
- **Medium Priority Issues:**
+ Starting values: [reference.md](reference.md#retrieval-starting-points). Template: [examples/retrieval-and-agents.md](examples/retrieval-and-agents.md#pattern-8-retrieval-design)
- - A loop specified where the step sequence is known in advance
- - Retrieval infrastructure for knowledge that fits the context window
- - An unversioned prompt, or an in-place edit to a shipped prompt version
- - Empty-retrieval behavior left undecided
- - No regression eval set named
+ ### Pattern 9: Tool Contracts and Agentic Loops
- **Common Mistakes:**
+ Every tool is read-only or mutating; a mutating one names its idempotency strategy, whether it needs confirmation, and whether it may run again after an ambiguous failure. Every loop names six guards.
- - Treating provider-side schema enforcement as validation
- - Repair attempts on semantic violations (laundering the bad value)
- - Acting on streamed partial JSON before completion
- - Specifying "handle API errors gracefully" instead of a per-row failure matrix
+ ```markdown
+ | Guard | Value |
+ | --------------------- | ------------------------------------------------------------- |
+ | Step budget | [max model turns] |
+ | Token budget | [cumulative ceiling across the loop] |
+ | Wall-clock budget | [elapsed limit] |
+ | Termination | [every condition, including success] |
+ | Repetition guard | [behaviour on identical consecutive tool calls] |
+ | Partial-result policy | [what is returned, flagged as truncated, when a budget trips] |
+ ```
- **Gotchas & Edge Cases:**
+ State carried between steps is named exactly; nothing else persists.
- - A valid-shaped record with a hallucinated value passes schema validation — semantic checks are separate
- - Retried refusals read as bypass attempts; surface them unchanged
- - Changing any of model id, prompt version, or retrieval parameters moves quality — all three trigger re-eval
- - A mutating tool retried after an ambiguous failure can double-apply; idempotency is part of the tool contract
+ Template: [examples/retrieval-and-agents.md](examples/retrieval-and-agents.md#pattern-9-tool-contracts-and-agentic-loops)
- </red_flags>
+ </patterns>
---
- <critical_reminders>
-
- ## CRITICAL REMINDERS
-
- > **All specifications must be grounded in the codebase's real model clients, prompt modules, schemas, and eval fixtures**
-
- **(You MUST justify the approach against the simpler tier — a fixed code-orchestrated chain beats an agentic loop whenever the step sequence is known)**
-
- **(You MUST pin an explicit model id in configuration with a named fallback — never a floating alias, never inline in code)**
+ <red_flags>
- **(You MUST define the output contract completely: mechanism, schema, validation boundary, and a repair-vs-reject policy)**
+ ## Red flags
- **(You MUST state budgets as numbers — tokens per call, calls per request, cost per request, p95 latency)**
+ **Breaks in production:**
- **(You MUST identify where untrusted input enters every prompt, and require adversarial eval cases wherever it does)**
+ - An output ceiling sized for the visible answer on a model that reasons — reasoning spends it, the response returns incomplete with no answer, and the tokens are billed — size it for both and give "incomplete" a failure row.
+ - A refusal sent through the repair loop or the transport retry — the same request is refused and billed again — detect the refusal signal and apply the spec's refusal policy.
+ - Every 429 retried — a spent quota or spend cap also answers 429, and no wait clears it — tell them apart by error code.
+ - A retry policy specified on top of a client that already retries — the attempts multiply — state which layer owns retries.
+ - Empty-retrieval behaviour left undecided — the model answers from its own knowledge and the answer looks cited — decide answer-or-abstain in the spec.
+ - The system prompt truncated to fit — instructions vanish with no error — shed the lowest-priority context instead.
+ - A step cap with no partial-result policy — agent frameworks commonly end the run with an error when the cap trips, discarding a partial answer that was available — state what each tripped budget returns.
+ - A mutating tool allowed to run again after an ambiguous failure with no idempotency key — the action applies twice — put the key in the tool contract.
+ - Eval thresholds chosen after seeing results — the gate ratifies whatever shipped — fix them in the spec.
- **(You MUST apply each framework only when the spec touches its artifact class — an unused section is omitted, never filled)**
+ **Surprising behaviour:**
- **Failure to specify these contracts produces AI features whose outputs go unvalidated, whose costs are unbounded, whose injections ship undetected, and whose regressions cannot be traced to a prompt revision.**
+ - A valid-shaped record with an invented value passes schema validation; semantic checks are separate code.
+ - An id's shape does not say whether it is pinned: a dateless id is a fixed snapshot on one provider and a hot-swapped alias on another, so the spec cites the provider's versioning page.
+ - A threshold on a small eval set measures noise — at 0.90 on 40 cases, 0.85 and 0.95 are indistinguishable.
+ - A model judge is swayed by answer length and by the order a pair is presented in, and its scores mean nothing until they are checked against human labels.
- </critical_reminders>
+ </red_flags>