analyze-failures · v0.5.0 · 2026-10-20 · sha256 39e3f67ec9794480

analyze-failures v0.5.0A

Immutable. This exact content never changes and is served at /api/v1/blob/39e3f67ec9794480.

No tool to install first
claude plugin marketplace add https://markdownregistry.com/mp/a/art_gbq7js4h5xzemgfk/39e3f67ec9794480fa4f23932419976079fb8c85493d3c769277da7b3db3c3e1.json

Then /plugin install analyze-failures@markdownregistry-gbq7js4h5xzemgfk in Claude Code. It installs this exact version, the one graded A here, pinned to the commit it was graded at, and leaves it there.

---
name: analyze-failures
description: Diagnoses one red Katalon True Platform (formerly TestOps) run. It classifies each failed result as a product defect, an automation defect, or an environment issue, clusters failures by signature, and files ALM-linked defects for confirmed product bugs. Use when a run or suite has failures and the question is whether the application broke or the test did. Not for the ship call, which is release-analyze, and not for suite repair over time, which is test-maintenance. For the manual tester and the automation tester.
license: MIT
compatibility: Requires the Katalon True Platform MCP server and network access.
metadata:
  version: "0.5.0"
  mcp-contract: "katalon-docs-2026-10"
---

# Analyze Failures

This skill turns a set of failed results into a diagnosis. When the user asks, it also files defects. The core value is classification: it separates real product bugs from automation and environment noise.

The reader is the manual tester or the automation tester who looks at a red run and needs to know whether the application broke or the test did.

Katalon True Platform (formerly TestOps) holds the results. Katalon Test Execution - Cloud (formerly TestCloud) holds the artifacts of cloud runs. Clients add a prefix to tool names (Claude Code uses `mcp__<server>__<tool>`). This skill uses bare names.

## What the MCP gives you

| Need | Tool |
|---|---|
| Find recent or specific results | `find_test_results`, `list_executions` |
| Read one execution or one result | `read_execution`, `read_execution_test_results`, `read_test_result` |
| Run history for flaky and trend signals | `analyze_test_case_stability`, `analyze_test_flakiness` |
| Existing defect context | `get_defect_insights` |
| Artifacts of a cloud execution | `list_execution_outputs` |
| Run with AI session detail | `read_manual_ai_session` |
| Rerun a completed execution | `rerun_execution` |
| Attach evidence | `prepare_artifact_upload`, `confirm_artifact_upload` |
| File a defect | `find_alm_integration_projects`, `create_defect` |

Read each tool schema before the first call. This skill does not repeat parameters.

Limits to state up front:

- AI root-cause summaries and automation error-pattern analytics are product features in the web app. They are not MCP calls. Do not claim to call them.
- `create_defect` needs a failed test result ID and the ALM integration details. Defect creation without an ID does not exist.
- Two ID spaces both use the word "execution". `read_execution` and `list_executions` use the numeric True Platform ID. `list_execution_outputs` uses the Test Execution - Cloud string ID. No tool converts one to the other. Never swap the two list tools.
- `rerun_execution` works only on a completed execution.
- MCP is stateless. Ask in chat for any missing input and carry IDs from call to call.

## Workflow

```text
+----------------+   +----------------+   +----------------+   +----------------+
| Collect        |-->| Add history    |-->| Classify and   |-->| Report, then   |
| failures       |   | and evidence   |   | cluster        |   | act (asked)    |
+----------------+   +----------------+   +----------------+   +----------------+
```

1. **Collect the failures.** Use `find_test_results` for recent or targeted results, or `read_execution_test_results` for one run. Use `read_test_result` for detail on each failed case. If the user gives no scope, ask for an execution ID, a suite, or a date range.
2. **Set aside results with no verdict.** Blocked, not executed, and Run with AI sessions stuck in TODO or IN_TESTING are not failures. List them apart. Do not force them into a bucket.
3. **Read run history first.** Call `analyze_test_case_stability` for the cases in scope. Use `analyze_test_flakiness` for a lookback window with a minimum run count. A case is flaky when the same test on the same build gave different results. Never label a case flaky from one read of its code. A model may explain a root-cause category after history shows the pattern.
4. **Read the evidence.** Call `list_execution_outputs` with the cloud execution ID for artifact links. For Playwright results, open the trace and the error context (see the reference). For a Run with AI result, read the session with `read_manual_ai_session` and check whether the agent misread the step.
5. **Classify each failure** into one bucket:
   - **Product defect.** The application behaved wrong and the failure repeats.
   - **Automation defect.** The test is wrong: locator, wait, stale data, or a step that no longer matches the UI.
   - **Environment or data.** Infrastructure, account, network, fixture, or AUT state.
   A case that passed 8 or more of its last 10 runs needs a rerun or stability evidence before you call its failure a product defect.
6. **Cluster by signature.** Group failures with the same step, error message, and object. One cluster is one candidate root cause and at most one defect.
7. **Rerun only to learn something.** `rerun_execution` costs Test Execution - Cloud minutes and can change AUT data. Ask the user first. A pass on an unchanged rerun is flaky evidence. It does not clear the failure. Do not loop reruns until green.
8. **Check for existing defects.** Call `get_defect_insights` for the iteration or date range. If it cannot show a defect for the signature, ask the user to search the ALM tracker before you file.
9. **File defects only when asked, and only for product defects.** Follow `references/write-policy.md`: preview each defect (summary, description, failed result ID), get one confirmation, call `find_alm_integration_projects` if the integration is unknown, call `create_defect`, then report the key or link the call returns. If the call returns no key, say so.
10. **Attach evidence only when asked.** Use `prepare_artifact_upload` and then `confirm_artifact_upload`. Files up to 10 MiB. See the tool schema for what the artifact attaches to.
11. **Report** per cluster: classification, likely cause, affected cases, evidence, and action (file, repair, or rerun after a fix).

Never invent a stack trace, a defect key, or an ALM project. If an input is missing, report blocked and name the input.

## Prompt recipes

- `Triage the failures in execution 8842: which are product bugs, flaky tests, or environment?`
- `Cluster today's failed results by root cause and tell me what to file.`
- `File defects for the confirmed product bugs in the checkout suite and link them to the failed results.`

## Hand-offs

- Automation defects and flaky cases go to `test-maintenance`.
- A ship decision from the failure picture goes to `release-analyze`.
- A coverage gap exposed by a failure goes to `test-plan`.
- The full capability boundary lives in `true-platform-testing`.

Read `references/failure-triage.md` before you classify. It holds the signal table, the flaky rules, the evidence reading steps, and the report template.