ai-prompt-regression-testing ยท diff
git:20261003.12d2318 to git:20261015.fac8b7e
88 added, 100 removed. Audit A to A.
---
name: ai-prompt-regression-testing
- description: 'Prompt engineering regression test matrix register: baseline outputs, semantic drift thresholds, judge evaluations, and golden dataset.'
+ description: 'Use when comparing prompt changes against a fixed evaluation set: record actual model settings, required checks, missing results, judge limitations and case-level regression evidence.'
category: engineering
risk: safe
source: self
source_type: self
date_added: "2026-10-01"
author: Ranjeet2063
tags: [ai, prompt-engineering, testing, evaluation, llm, benchmarking]
tools: []
source_repo: Ranjeet2063/agentic-awesome-skills
---
- # AI Prompt Regression Test Matrix
-
- **What it is:** Tracks regression baselines, evaluation rubrics, and automated judge verdicts to prevent output degradation across prompt revisions.
-
- ## Overview
-
- Provides a standardized, auditable framework and data model for **AI Prompt Regression Test Matrix** operations across distributed engineering and decentralized application systems.
-
- ## When to Use This Skill
-
- - When formalizing architectural contracts, security invariants, or operational limits for AI Prompt Regression Test Matrix.
- - When cross-functional review is required between protocol developers, smart contract auditors, and AI engineering agents.
- - When generating reproducible CSV, SQL DDL, JSON Schema, and Notion property registers for tracking compliance.
-
- ## How It Works
-
- 1. Define the parameters, thresholds, and identity bindings required for the target operational register.
- 2. Select appropriate boundary enforcement values from validated enum select sets.
- 3. Export standardized artifacts (CSV table, SQL DDL, JSON Schema) to integrate into validation CI pipelines.
-
- ## Field Reference
-
- | # | Field Name | Type | SQL Type | JSON Schema Type | Notion Property Type | Example Value |
- |---|------------|------|----------|------------------|----------------------|---------------|
- | 1 | Prompt TestCase ID | `id` | `SERIAL PRIMARY KEY` | `integer` | Text | `PTEST-001` |
- | 2 | Prompt Identifier | `text` | `VARCHAR(64)` | `string` | Text | `soroban_code_refactor_v2` |
- | 3 | Target LLM Model Family | `select` | `VARCHAR(64)` | `string` | Select | `Claude 3.5 Sonnet` |
- | 4 | Evaluation Metric | `select` | `VARCHAR(64)` | `string` | Select | `AST Code Correctness` |
- | 5 | Semantic Drift Threshold | `number` | `NUMERIC(5,2)` | `number` | Number | `0.05` |
- | 6 | Golden Baseline Match % | `number` | `NUMERIC(5,2)` | `number` | Number | `98.50` |
- | 7 | Judge Model Evaluator | `text` | `VARCHAR(64)` | `string` | Text | `Gemini 1.5 Pro` |
- | 8 | Zero-Shot Reasoning Verified | `select` | `VARCHAR(16)` | `string` | Select | `Yes` |
- | 9 | Latency Bound Seconds | `number` | `NUMERIC(6,2)` | `number` | Number | `3.20` |
- | 10 | Test Suite Verdict | `select` | `VARCHAR(32)` | `string` | Select | `Passed` |
- | 11 | Benchmarking Date | `date` | `DATE` | `string, format: date` | Date | `2026-10-01` |
-
- ## Select Options
-
- **Target LLM Model Family**
+ # AI Prompt Regression Testing
- ```
- Claude 3.5 Sonnet | GPT-4o | Gemini 1.5 Pro | DeepSeek Coder
- ```
+ ## When to Use
- **Evaluation Metric**
+ - Compare a prompt change with an observed baseline on a fixed evaluation set.
+ - Detect schema, factuality, refusal, tool-selection or task-completion regressions.
+ - Record a reviewable decision when results are noisy, missing or contradictory.
- ```
- AST Code Correctness | Semantic Embedding Cosine | Exact Match | Rubric Scoring
- ```
+ ## Workflow
- **Zero-Shot Reasoning Verified**
+ 1. Define the task, representative cases and forbidden outcomes. Use synthetic
+ or approved/redacted data; freeze case IDs and the dataset revision before runs.
+ 2. Record baseline/candidate prompt digests, exact available model/version,
+ inference settings, tool versions and evaluation environment. Do not hardcode
+ old model-family options or silently change models between comparisons.
+ 3. Predeclare checks and acceptance rules. Use deterministic schema/field checks
+ for structured output, behavioral checks for tools and a clear human/rubric
+ standard for open text. Similarity does not prove factual or code correctness.
+ 4. Run matched baseline/candidate cases under the same permissions and budget.
+ Record latency/cost and repeated runs when variability matters. A timeout,
+ missing output or judge error is incomplete, not a pass or a zero-cost result.
+ 5. Treat judge outputs as evidence with limitations. Blind/randomize presentation
+ where feasible, calibrate against human-labeled examples, and record judge
+ identity/rubric. Do not ask a model to verify private reasoning or accept a
+ judge's confidence as an objective correctness measurement.
+ 6. Review differences case by case and report the denominators, failures and
+ uncertainty. Keep mandatory safety/schema failures visible even if aggregate
+ similarity improves. Use the predeclared rule, not a threshold selected after results.
+ 7. Store a redacted report and exact artifacts at an authorized destination.
+ Decide retain/revise/reject for the stated dataset; deployment is separate.
- ```
- Yes | No
- ```
+ ## Regression register
- **Test Suite Verdict**
+ | Field | Type | Example / meaning |
+ |---|---|---|
+ | case_id | string | PTEST-001; a textual ID, not SERIAL |
+ | dataset_revision | immutable reference | exact case inventory, expected checks and labels |
+ | prompt_revision | string | baseline/candidate digest |
+ | model_runtime | string | exact available ID/version plus settings |
+ | check | string | required JSON fields or approved rubric criterion |
+ | result | enum | pass / fail / incomplete |
+ | evidence | reference | redacted actual output, tool trace or observed check |
+ | judge_configuration | optional object | exact judge ID/rubric/version and calibration limits |
+ | latency_ms | nonnegative number or null | null if not observed; never invented |
+ | cost | amount/currency or null | actual billing basis; unknown remains unknown |
+ | run_revision | immutable reference | binds the report to the actual run |
- ```
- Passed | Degraded | Failed Regression
- ```
+ IDs and statuses must have consistent SQL/JSON/CSV representations. Do not store
+ private prompts, user messages or tokens in a public spreadsheet or Notion register.
- ## Relations
+ ## Worked example
- - `Audit Reference` -> links to the formal review documentation or test repository.
- - `Target Architecture` -> links to the deployed contract or autonomous agent runtime component.
+ Two cases test a support-answer prompt. PTEST-001 requires an answer grounded in
+ the supplied policy; PTEST-002 requires a structured escalation when the policy
+ is absent. The candidate sounds more helpful but PTEST-002 has no observed result.
+ The report is incomplete, not a win. The following Python 3 helper aggregates
+ already reviewed case statuses; it does not call models or decide correctness.
- ## Examples
+ ```python
+ def summarize_regression(results, expected_case_ids):
+ if not isinstance(expected_case_ids, list) or not expected_case_ids:
+ return {"status": "incomplete", "reason": "missing case inventory"}
+ if any(not isinstance(case, str) or not case for case in expected_case_ids):
+ return {"status": "incomplete", "reason": "invalid case IDs"}
+ if len(set(expected_case_ids)) != len(expected_case_ids):
+ return {"status": "incomplete", "reason": "duplicate expected case IDs"}
+ if not isinstance(results, list) or any(not isinstance(row, dict) for row in results):
+ return {"status": "incomplete", "reason": "invalid results"}
+ by_id = {}
+ for row in results:
+ case = row.get("case_id")
+ if not isinstance(case, str) or case not in expected_case_ids or case in by_id:
+ return {"status": "incomplete", "reason": "unexpected or duplicate result"}
+ if row.get("result") not in ("pass", "fail", "incomplete"):
+ return {"status": "incomplete", "reason": "unknown result"}
+ by_id[case] = row["result"]
+ failed_cases = [case for case, result in by_id.items() if result == "fail"]
+ if set(by_id) != set(expected_case_ids) or "incomplete" in by_id.values():
+ return {"status": "incomplete", "reason": "missing or incomplete case",
+ "failed_cases": failed_cases, "evaluated": len(by_id)}
+ return {"status": "regression" if failed_cases else "checks-passed",
+ "failed_cases": failed_cases, "evaluated": len(by_id)}
- **Prompt**
- ```
- How do I configure and track AI Prompt Regression Test Matrix for our production environment?
+ print(summarize_regression(
+ [{"case_id": "PTEST-001", "result": "pass"}],
+ ["PTEST-001", "PTEST-002"],
+ ))
```
- **Recommended Next Step**
-
- > Generate the unified field schema, SQL DDL migration, and JSON validation schema to register into your system catalog.
- >
- > Workflow: Define criteria -> Run automated verification -> Record baseline -> Monitor invariants.
-
- ## Best Practices
-
- - Enforce strict typing on numerical bounds and currency amounts; avoid unstructured free-text fields for critical states.
- - Re-run validation test suites on every state-altering commit or parameter change.
- - Keep example data synthetic and isolated from production cryptographic keys or private endpoints.
+ Expected output: incomplete with reason missing or incomplete case. For all
+ observed pass statuses the output is checks-passed, meaning only those recorded
+ checks passed; it is not a release or general quality certificate.
## Limitations
- - Provides architectural specifications, data models, and verification schemas; does not execute direct transaction signing without authorized external tooling.
- - Requires network connectivity and valid RPC credentials when querying on-chain states.
-
- ## Security & Safety Notes
-
- - All parameters declare `risk: safe`. No unauthorized state modification or privileged credential access is performed.
- - Use synthetic dummy keys and mock addresses in test suites and local verification scripts.
-
- ## Common Pitfalls
-
- - **Problem:** Mismatched decimal precision between contract runtime and database register.
- **Solution:** Always verify decimals using the explicit field mapping in this reference.
- - **Problem:** Missing authorization checks prior to state update.
- **Solution:** Cross-validate against the Security Audit register before deployment.
-
- ## Related Skills
-
- - @ai-agent-tool-routing - covers tool schema registration and retry policy.
- - @cross-chain-relayer-audit - covers message hashes, nonces and quorum proofs.
- - @smart-contract-formal-verification - verifies state invariants mathematically.
-
- ## Reusable Prompt
-
- ```
- I want to establish a verified AI Prompt Regression Test Matrix register for our production protocol.
- Guide me through the required field parameters and output the corresponding SQL DDL and JSON Schema.
- ```
+ - No RPC, relayer, signing key or blockchain deployment is required for this
+ generic prompt workflow. Real integrations may have their own dependencies.
+ - Exact text match is unsuitable for many valid open-ended outputs; embeddings
+ and judges have blind spots and cannot replace mandatory behavioral checks.
+ - A golden set must be reviewed/versioned; it is not created merely by naming it.
+ - This entry does not include a provider adapter, judge, dataset or runnable suite.
+ Use the project's supplied materials and state what was actually executed.
+ - A finite test set cannot prove future prompt behavior or absence of injection.