git:20260521.5aeddd8 to git:20260521.237dd37

---

name: momentic-result-classification

description:

Classify or explain Momentic test run results using Momentic MCP tools.

Use when the user asks to categorize a failure, understand why a run failed,

triage test results, or compare run results to past run results.

---

# Momentic result classification (MCP)

Momentic is an end-to-end testing framework where each test is composed of browser interaction steps. Each step combines Momentic-specific behavior (AI checks, natural-language locators, ai actions, etc.) with Playwright capabilities wrapped in our YAML step schema. When these tests are run, they produce results data that can be used to analyze the outcome of the test. The results data contains metadata about the run as well as any assets generated by the run (e.g. screenshots, logs, network requests, video recordings, etc.). Your job is to use these test results to classify failures that occurred in Momentic test runs.

## Instructions

1. Given a failing test run, start by understanding the current run from the data already provided, such as the run summary, the failing step's screenshots, and nearby step results. In many cases the current run is enough on its own to explain the failure.

2. Only look beyond the current run, including past runs of the same test, when the current run is not sufficient to determine what should have happened, such as when the test intent is ambiguous, when you are not sure whether something is an application change versus a test issue, or when distinguishing flakiness from a regression.

3. After analyzing why the run failed, bucket the failure into one of the below categories, explaining the reasoning for choosing the specific category.

4. Also determine the failure's recoverability:

- `RECOVERABLE` when the test itself can be updated so future runs pass.

- `ONE_TIME_RECOVERABLE` when this specific run could be recovered without persisting a test change.

- `NON_RECOVERABLE` when manual intervention is required.

## Helpful MCP tools

`momentic_get_run` — Returns some metadata about the run and a summary of the full run results. Use the metadata to help you parse through the run results (e.g. which attempt to look at, which step failed, etc.). If the current run details were already supplied in the initial context, do not call this again for that same run unless you explicitly need a different attempt.

`momentic_list_runs` — Recent runs for a test so you can compare the result of past runs over time. **Always pass `gitBranchName` when it exists on the run in question** so that it's more likely you're looking at the same version of the test. Pass `recovered=true` when you want to inspect recovered runs.

`momentic_get_step_result` — Returns the result of a specific step, with other information such as full step trace and before/after screenshots. Use `parentStepIdChain` for steps nested inside other steps. Only request `includeTrace=true` when you need it, because it can be very large.

`momentic_get_test_steps_for_run` — Returns the simplified test steps recorded on a run (`stepsSnapshot`, `beforeStepsSnapshot`, `afterStepsSnapshot`). You can use this to understand the intent of the test if you need more information than what you can glean from the test name and description.

## Background

### Test run result structure

When momentic tests are run via the CLI, the results are stored in a "run group". The data for this run group is stored in a single directory within the momentic project. By default, the directory is called `test-results`, but can be changed in momentic project settings or on a single run of a run group. The run group results folder has the following structure:

```

test-results/

├── metadata.json data about the run group, including git metadata and timing info.

└── runs/ On zip for each test run in the run group.

├── <runId_1>.zip a zipped run directory containing data about this specific test run. Follows the structure described below.

└── <runId_2>.zip

```

When unzipped, run directories have the following structure:

```

<runId>/

├── metadata.json run-level metadata.

└── attempts/<n>/ one folder per attempt (1-based n).

├── metadata.json attempt outcome and step results.

├── console.json optional browser console output.

└── assets/

├── <snapshotId>.jpeg before/after screenshot for each step (see attempt metadata.json for snapshot ID).

├── <snapshotId>.html before/after DOM snapshot for each step (see attempt metadata.json for snapshot ID).

├── har-pages.log HAR pages (ndjson).

├── har-entries.log HAR network entries (ndjson).

├── resource-usage.ndjson CPU/memory samples taken during the attempt.

├── <videoName> video recording (when video recording is enabled).

└── browser-crash.zip browser crash dump (only present on crash).

```

When getting run results via the momentic MCP, tools such as `momentic_get_run` will return links to the MCP working directory (default `.momentic-mcp`). This directory will contain unzipped run result folders, following the structure above, named `run-result-<runId>`.

### Element locators

Certain step types that interact with elements have a "target" property, or **locator**, that specifies which element the step should interact with.

#### Locator caches

Locators identify elements by sending the page state html/xml to an llm as well as a screenshot. The llm identifies which element on the page the user is referring to. Momentic will attempt to "cache" the answer from the llm so that future runs don't require AI calls. On future runs, the page state is checked against the cached element to determine whether the element is still usable, or the page has changed enough such that another AI call is required.

A locator cache can bust for a variety of reasons:

- the element description has changed, in which case we'll always bust the cache

- the cached element could not be located in the current page state

- the cached element was located in the page state, but fails certain checks specified on the cache entry, such as requiring a certain position, shape, or content.

You can find the `cacheBustReason` on the `trace` property in the results for a given step, but only when you explicitly request `includeTrace=true`. The `cache` property is also listed on the results, showing the full cache saved for that element.

#### Identifying bad caches

Sometimes the element that was cached is not the element that the user intended to target. This can cause failures or unexpected behaviors in tests. In these cases, it helps to verify exactly why the wrong cache was saved in the first place. Only request `includeTrace=true` for these cache-debugging cases or when you suspect incorrect Momentic execution data. Use the `runId` property of the `targetUpdateLoggerTags` on the incorrect cache to get the details of the original run, calling `momentic_get_run` with this runId. This will return the run where the cache target was updated.

### Module caching

Cached modules skip executing their steps when the module cache key and resolved inputs are unchanged, and reuse the cached return value from the module's last step.

Authentication modules can also save and restore browser auth state from the module cache, including cookies, localStorage, and IndexedDB. They may use a page-content check after restoring auth state to decide whether the cache is still valid.

### File uploads

A file upload step prepares one file for the next native file picker, so it must run before the action that opens the picker.

Sources can be remote URLs, `file://` references to earlier downloads, CLI-local paths, or uploaded user files. The step can also override the presented filename, and Momentic wires the prepared file into the browser's file chooser handling.

## Using past runs

Look at past runs of the same test only when the current run is not enough on its own to explain the failure. Past runs help answer questions like:

- When did this test start failing?

- What differed vs the last passing run?

- Did the same action behave differently on an earlier run?

- Did the application behavior change for this test?

- Is this actually flaky, or did the app or test change?

When you do need past runs, do not rely only on summaries from `momentic_get_run` or `momentic_list_runs` to understand what happened in a test run. You must look at the specific run details, including step results and screenshots, to determine the behavior of past runs.

When looking at past runs, use the following workflow:

1. Call the `momentic_list_runs` tool to identify the runs you want more detail on.

2. Call `momentic_get_run` for that specific run to get the run details.

3. Call `momentic_get_step_result` for step results that you want to see in more detail, especially for step screenshots. Do not request `includeTrace=true` unless screenshots and the normal step-result fields are insufficient.

Always look at screenshots when determining the behavior of test runs. Do not guess at what happened on a step without checking screenshots.

### Recovered runs

A recovered run is one that ultimately passed via Momentic's failure recovery mechanism after one or more failing attempts. Treat recovered runs as partial failures, not clean passes.

### Multi-attempt runs

When `momentic_list_runs` shows a passing run with `attempts > 1`, treat it as a partial failure worth investigating, not a clean passing run. Use the `attemptNumber` parameter to retrieve earlier failed attempt results for that run to understand what was going wrong before the retry succeeded.

### Flakiness and intermittent failures

- In order to consider a test flaky or failing intermittently, it must be intermittently failing for the same app and test behavior.

- Just because a test failed once does not mean that it is flaky. It could have failed because of an application change.

- You need to determine whether there was an application or test change between runs by analyzing the screenshots and other run data.

- You cannot make assumptions about flakiness or intermittent failures without verifying whether an application or test change caused the failure.

### Test temporality

- Any past results may not necessarily match today’s test file. The test may have changed, meaning the result was on a different version of the test.

- You can call `get_test_steps_for_run` to help you determine if the test itself changed between runs, although note that this tool returns a _summary_ of each test step. If you suspect that specific details on certain steps have changed between test runs, full step details are included in the response from `momentic_get_step_result`; only request `includeTrace=true` when those fields and screenshots still are not enough.

## Bug vs change

- Bug: something very clearly went wrong when it should not have, such as an error message appearing. It is obvious just by looking at a single step or two that this is a bug.

- Change: a clear change in the application behavior that you can prove through screenshots.

## Recoverability

Along with the category, determine one recoverability value:

- `RECOVERABLE` — The failure can be automatically fixed by updating the test itself so that future runs pass.

- Examples: an application change that requires a test update; vague locators or assertions that can be rewritten to pass stably.

- `ONE_TIME_RECOVERABLE` — The failure can be recovered for this specific run without persisting a test change.

- Examples: a random modal that can be dismissed without affecting test purpose; a temporary delay where waiting or retrying would likely succeed.

- `NON_RECOVERABLE` — The failure cannot be automatically addressed and requires manual intervention.

- Examples: missing credentials; missing local files required for upload; outages likely caused by third-party systems where test steps cannot fix the issue.

## Formal classification output

- Exactly one category id and one recoverability value. Do not invent new labels or use multiple categories.

- Ground your decision in data. Be sure that you have fully investigated the run before assigning the category.

- You can only confirm behavior through screenshots. Do not guess at step behavior without checking screenshots.

- When reasoning cites another run, use the full runId UUID exactly as returned by tools. Do not shorten it to a prefix.

```text

Reasoning: <a few sentences tied to summary, any relevant past runs, and test intent>

Category: <one id from the list>

Recoverable: <RECOVERABLE | ONE_TIME_RECOVERABLE | NON_RECOVERABLE>

Confidence: <high | medium | low>

```

Confidence levels:

- `high` — direct evidence, such as a clear screenshot of a label change or crash

- `medium` — strong inference from multiple signals but no single conclusive screenshot or data point

- `low` — ambiguous evidence; the classification required significant inference or the root cause is unclear

## Category ids

Use these strings verbatim:

- `NO_FAILURE` — The run had no failures; all attempts passed.

- Examples: unexpected behavior when viewing the run trace; the AI clearly misread or hallucinated data that is unambiguous in the screenshot, and no reasonable test alternative exists to avoid the AI step.