git:20260624.400a13c to git:20261105.5408638

103 added, 154 removed. Audit B to A.

---

name: testrail

- description: Interact with the TestRail Test Management API. Use when fetching test plans, test runs, test cases, submitting test results, or attaching screenshots and logs to test evidence. Supports the full execution lifecycle from locating tests to recording step-by-step outcomes.

- ---

- # TestRail Test Management

- ## Authentication

- **Env vars** (required by all scripts):

- ```

- TESTRAIL_URL=https://[account].testrail.io

- TESTRAIL_EMAIL=testrail@company.com

- TESTRAIL_PASSWORD=<api_key>

- ```

- Credentials stored in 1Password: **"TestRail"** in **SSN - QA** vault.

- **Install:** `pip3 install testrail-api`

+ description: Interact with TestRail through the testrail-arriba CLI. Use when fetching test plans, test runs, test cases, submitting test results, or attaching screenshots and logs to test evidence. Supports the full execution lifecycle from locating tests to recording step-by-step outcomes.

+ allowed-tools: Bash(testrail-arriba --version), Bash(testrail-arriba auth status)

---

- ## Object Hierarchy

- ```

- Project (e.g. Solsta = ID 2)

- └── Suite (test case library)

- └── Section (folder within a suite)

- └── Case (reusable test case template with steps)

- └── Milestone (release/sprint marker)

+ # TestRail with testrail-arriba

- └── Plan (groups multiple runs together)

- └── Entry (one per suite in the plan)

- └── Run (test execution instance)

- └── Test (live instance of a Case — auto-created)

- └── Result (outcome you submit)

- └── Attachment (screenshot, log)

- ```

+ [testrail-arriba](https://github.com/arriba-tools/testrail) is the CLI this project's TestRail

+ workflows use. It takes step outcomes in Markdown, uploads the screenshots they name to the run, and

+ sends the HTML TestRail renders, so step results are never written as HTML by hand.

- **Key relationships:**

- - A **Case** becomes a **Test** when included in a **Run**

- - You submit **Results** to a Test (multiple results = history)

- - **Runs inside a Plan** are in `entries[].runs[]` — NOT returned by `get_runs`

- - **Attachments** are uploaded separately after submitting a result

+ ## Prerequisites

- **Status IDs:** 1=Passed, 2=Blocked, 3=Untested, 4=Retest, 5=Failed

+ Installed version and login:

- ---

+ !`testrail-arriba --version 2>&1 || echo "testrail-arriba is not installed"`

- ## Scripts

+ !`testrail-arriba auth status 2>&1 || true`

- All scripts in `skills/testrail/scripts/`. Run with `python3 skills/testrail/scripts/<script>.py`.

+ If it is not installed, tell the user to download it from the

+ [releases](https://github.com/arriba-tools/testrail/releases) and put it on the `PATH`, or run

+ `cargo install --git https://github.com/arriba-tools/testrail`, and stop.

- ### Browsing & Discovery

+ If there is no login, tell the user to run `testrail-arriba auth login`, which stores the API key in

+ the keyring, and stop. `TESTRAIL_URL`, `TESTRAIL_EMAIL`, and `TESTRAIL_API_KEY` in the environment

+ work without a stored login, and `TESTRAIL_PASSWORD` is accepted in place of the key.

- | Script | Purpose | Key Flags |

- |--------|---------|-----------|

- | `get_projects.py` | List all projects | `--names-only`, `--active-only` |

- | `get_suites.py` | List suites in a project | `--project-id`, `--names-only` |

- | `get_sections.py` | List sections/folders | `--project-id`, `--suite-id`, `--tree` |

- | `get_cases.py` | List test cases | `--project-id`, `--section-id`, `--with-steps`, `--limit` |

- | `get_milestones.py` | List milestones | `--project-id`, `--id`, `--active-only` |

- | `get_plan.py` | Find plans by name/ID or list all | `--id`, `--project-id`, `--name`, `--list`, `--active-only` |

- | `get_runs.py` | Get runs inside a plan | `--plan-id`, `--names-only` |

- | `get_tests.py` | Get test instances in a run | `--run-id`, `--status`, `--names-only` |

- | `get_steps.py` | Get steps for a case or test | `--case-id`, `--test-id`, `--json` |

+ The QA credentials are in 1Password, item **"TestRail"** in the **SSN - QA** vault. Pipe the key

+ to `auth login --site <site> --email <email> --token` to store it without a prompt.

- ### Submitting Results

+ ## Conventions

- | Script | Purpose | Key Flags |

- |--------|---------|-----------|

- | `add_result.py` | Submit a result with optional step outcomes | `--run-id`, `--case-id`, `--status`, `--comment`, `--elapsed`, `--step-results-file` |

- | `add_attachment.py` | Attach a file to a result or run | `--result-id`, `--run-id`, `--file` |

- | `submit_test.py` | Submit result with step outcomes | `--run-id`, `--case-id`, `--status`, `--comment`, `--elapsed`, `--step-results-file` |

+ - Every ID takes TestRail's prefixed form, such as `C39`, `R201`, or `T1001`, or the page's URL.

+ - `--project` (`-p`) takes the project's ID or name, as in `-p Solsta` or `-p 2`.

+ - Add `--json` to any command whose output you need to parse.

+ - Lists stop at 50 unless `--limit` or `--all` says otherwise, and say so on stderr when more remain.

+ - `attachment delete` asks before acting, `--yes` skips the prompt.

+ - Pass `--site acme.testrail.io` when several logins are stored.

- All scripts support `--json` for raw JSON output. Run any script with `--help` for full options.

+ ## How TestRail Fits Together

- ---

+ ```

+ Project -p takes its ID or name

+ ├── Suite → Section → Case (C39) the test and its steps: case view

+ ├── Milestone

+ ├── Plan → Run (R201) a plan's runs: run list --plan

+ └── Run (R201) runs outside plans: run list -p

+ └── Test (T1001) one case in either kind of run: test view

+ └── Result added with result add, never edited

+ ```

- ## Common Workflows

+ A case becomes a test when a run includes it, and each result added to a test joins its history.

- ### Find and explore a test plan

+ ## Find a Plan and What Is Left to Test

```bash

- # List plans for Solsta

- python3 skills/testrail/scripts/get_plan.py --project-id 2 --list --active-only

- # Get full plan detail (entries + runs)

- python3 skills/testrail/scripts/get_plan.py --id 15

- # List runs in the plan

- python3 skills/testrail/scripts/get_runs.py --plan-id 15 --names-only

- # List tests in a run

- python3 skills/testrail/scripts/get_tests.py --run-id 201 --names-only

+ testrail-arriba plan list -p Solsta --active --name "2.1"

+ testrail-arriba plan view 15 # entries with every run

+ testrail-arriba run list --plan 15

+ testrail-arriba test list R201 --status untested,retest

+ testrail-arriba test list --plan 15 --case C52 # one case on each platform in a plan

```

- ### Execute a test case

+ ## Browse the Case Library

```bash

- # 1. Get the steps

- python3 skills/testrail/scripts/get_steps.py --case-id 39

+ testrail-arriba section list -p Solsta --tree

+ testrail-arriba case list -p Solsta --section 31

+ testrail-arriba case list -p Solsta --ref PROJ-123 # cases behind a Jira issue

+ testrail-arriba case list -p Solsta --grep sso # a word anywhere in a case's text

+ testrail-arriba case view C39

+ ```

- # 2. Execute steps (manual or automation)

+ `--suite` is needed only when the project has several suites. `milestone list`, `suite list`, and

+ `project list --active` cover the rest.

- # 3. Attach screenshots to the run as you go

- python3 skills/testrail/scripts/add_attachment.py --run-id 201 --file /tmp/step1.png

- # → attachment_id=55 (use this in step results JSON)

+ ## Execute a Test

- # 4. Submit result with inline screenshot refs in step actuals

- python3 skills/testrail/scripts/submit_test.py \

- --run-id 201 --case-id 39 --status passed \

- --comment "All steps verified" --elapsed "2m 15s" \

- --step-results-file /tmp/step_results.json

- ```

+ ```bash

+ # 1. Read the steps, the run's configuration, and the latest results.

+ testrail-arriba test view T1001

- ### Submit with per-step inline screenshots (preferred)

+ # 2. Run the steps, saving a screenshot per step.

- Screenshots render **inline** in TestRail's step results view. This requires a 3-step process

- because attachment IDs are only known after upload.

+ # 3. Write the outcomes to steps.json, then check what would be sent.

+ testrail-arriba result add --test T1001 -s failed --steps steps.json --dry-run

- #### ⚠️ CRITICAL: HTML Formatting Rules for Step Results

+ # 4. Add the result once.

+ testrail-arriba result add --test T1001 -s failed -e "4m" --version 2.1.0 \

+ --comment "Fails at step 2." --steps steps.json --attach console.log

+ ```

- TestRail step results (`custom_step_results`) require specific HTML formatting to render properly:

+ `--run R201 --case C39` works in place of `--test`. `--comment-file` reads a longer comment,

+ `--defect` links issues, and `--field NAME=VALUE` sets custom result fields.

- 1. **Wrap text in `` tags** — This triggers `markdown_editor_id: 1` which enables HTML rendering. Without `` tags, all HTML is escaped and shows as raw text.

- 2. **Use `<blockquote>` for expected results** — Renders with a colored left border, visually distinct.

- 3. **Use `<img>` tags for inline images** — Place after the `` block: `<img src="index.php?/attachments/get/<id>" />`

- 4. **Do NOT use markdown** — `![alt](url)` syntax does NOT render in step results.

- 5. **All three fields support HTML** — `content`, `expected`, and `actual` all render HTML when `` tags are present.

+ ### Steps file

- #### Step results JSON format

+ A JSON array with one object per step, in the case's order. Text is Markdown, and image paths

+ resolve against the file's folder.

```json

[

- {

- "content": "Navigate to login page",

- "expected": "<blockquote>Login page loads with username and password fields</blockquote>",

- "actual": "Page loaded correctly. Username and password fields visible.<img src=\"index.php?/attachments/get/55\" />",

- "status_id": 1

- },

- {

- "content": "Click Submit",

- "expected": "<blockquote>Dialog appears confirming login</blockquote>",

- "actual": "Nothing happened. No dialog appeared.<img src=\"index.php?/attachments/get/56\" />",

- "status_id": 5

- }

+ {"status": "passed", "actual": "The page loaded with both fields.\n\n![](shots/step1.png)"},

+ {"status": "failed", "actual": "Nothing happened after **Submit**.\n\n![](shots/step2.png)"}

]

```

- **Field breakdown:**

- - `content` → Shown under **Step** heading. Wrap in `` tags.

- - `expected` → Shown under **Expected Result** heading. Wrap in `<blockquote>` for visual distinction.

- - `actual` → Shown under **Actual Result** heading. Wrap observation in ``, append `<img>` for screenshot.

- - `status_id` → 1=Passed, 5=Failed, 2=Blocked, 4=Retest.

+ | Field | Sent as |

+ |-------|---------|

+ | `content` | HTML paragraphs, or the case's step when left out |

+ | `expected` | HTML in a `<blockquote>`, or the case's expected result when left out |

+ | `actual` | HTML paragraphs, with images lifted out after them as `<img>` tags |

+ | `status` or `status_id` | A status name, as `status list` shows them, or its ID |

- #### Inline screenshots workflow (single result, no duplicates)

+ Each image is uploaded to the run once, however often it appears, and its ID is written into the

+ `<img>` tag. `--step-image 2=shot.png` adds a screenshot to a step without editing the file. Text

+ that already starts with an HTML tag is sent as it is, and `--raw` sends every field as given.

- ```bash

- # During testing: attach each screenshot to the RUN as you capture it

- python3 skills/testrail/scripts/add_attachment.py --run-id 16 --file screenshots/step1.png

- # → attachment_id=55

- python3 skills/testrail/scripts/add_attachment.py --run-id 16 --file screenshots/step2.png

- # → attachment_id=56

+ ## Results and Attachments

- # After testing: submit result with HTML img refs baked into step actuals

- # In your step results JSON, use <img> tags (NOT markdown):

- # "actual": "Step passed.<img src=\"index.php?/attachments/get/55\" />"

- python3 skills/testrail/scripts/submit_test.py \

- --run-id 16 --case-id 78 --status passed \

- --comment "All steps verified" --elapsed "4m 00s" \

- --step-results-file /tmp/steps_with_images.json

- # → One result, inline screenshots, no duplicates

+ ```bash

+ testrail-arriba result list --test T1001

+ testrail-arriba result list --run R201 --case C39

+ testrail-arriba attachment add shots/*.png --run R201 # prints each ID and its inline src

+ testrail-arriba attachment list --test T1001

+ testrail-arriba attachment download --run R201 -o ./R201

+ testrail-arriba status list # names and IDs, custom ones included

```

- **Key insight:** Attaching to the **run** (`--run-id`) doesn't require a result ID,

- so you get attachment IDs first, bake them into the step results, and submit once.

- This avoids the duplicate result problem caused by TestRail having no "update result" API.

- ### Browse the test case library

+ Downloads go to a `testrail-arriba` folder in the temp directory unless `-o` names one, and each

+ file name starts with its attachment ID.

- ```bash

- # Section tree

- python3 skills/testrail/scripts/get_sections.py --project-id 2 --tree

+ ## Raw API Calls

- # Cases in a section

- python3 skills/testrail/scripts/get_cases.py --project-id 2 --section-id 31 --with-steps

+ For an endpoint testrail-arriba has no command for, `api` reuses its login:

- # All cases (careful — could be large)

- python3 skills/testrail/scripts/get_cases.py --project-id 2 --names-only

+ ```bash

+ testrail-arriba api get_case_fields

+ testrail-arriba api "get_cases/2&suite_id=1"

```

- ---

+ `-d` posts a JSON body, from the argument, `@FILE`, or `-` for stdin.

## Gotchas

- 1. **Runs in plans are hidden.** `get_runs` (standalone) won't return them. Use `get_plan` → `entries[].runs[]` or the `get_runs.py --plan-id` script.

- 2. **No "start test" API.** Just fetch steps, execute, then submit a result. Use `status_id: 4` (Retest) as an "in progress" marker if needed.

- 3. **Attachments are separate.** Submit result first, get `result_id`, then attach files. Or use `submit_test.py` which does both.

- 4. **Step results field name** is `custom_step_results` (not `step_results`).

- 5. **Single-suite projects** (like Solsta) don't need `--suite-id` for most queries.

- 6. **No "update result" API.** You can only add new results, never modify existing ones. To avoid duplicates when adding inline screenshots, attach to the **run** first (`add_attachment.py --run-id`), then bake the attachment IDs into step results before submitting.

- 7. **Inline image syntax:** Use `<img src="index.php?/attachments/get/<attachment_id>" />` — NOT markdown `![alt](url)`. The attachment must exist on the same run/test. Attachment IDs are global within a project.

- 8. **HTML rendering requires `` tags.** If you submit step results without `` tags wrapping the text, TestRail escapes all HTML (shows raw `<img>` as text). Always wrap text content in `` tags to trigger `markdown_editor_id: 1`.

- 9. **Use `<blockquote>` for expected results.** Renders with a colored left border for visual distinction from the actual result.

+ 1. **Results cannot be edited.** TestRail has no update result API, so a second `result add` adds

+ another result to the history. Run with `--dry-run` first.

+ 2. **Runs in plans are hidden** from `run list -p`. Use `run list --plan` or `plan view`.

+ 3. **There is no start test API.** Read the steps, execute them, then add a result. A `retest`

+ status works as an in-progress marker.

+ 4. **Inline images need the run.** `result add` uploads them there itself. When writing HTML by hand

+ with `--raw`, attach to the run first with `attachment add --run` and use the printed `src`.

+ 5. **HTML renders only inside `` tags.** Without them TestRail escapes the HTML and ignores

+ Markdown image syntax, which is why `--raw` text must carry its own `` tags.