git:20260512.488622b to git:20260625.710528e

- description: Write or update a SWTBot JSON probe script (e.g. from a test plan), then validate it locally by running the `ProbeRunner` Tycho test against the Copilot for Eclipse plugin. Use this whenever you need end-to-end UI validation against a real Eclipse workbench.

- # Authoring and Validating SWTBot Probe Scripts (Eclipse)

- Use this skill to write or update a JSON probe script and validate it by

- invoking the `com.microsoft.copilot.eclipse.swtbot.test` Tycho bundle locally.

- Probe scripts are JSON, so no Java is compiled per test case.

- 1. Drop a probe at `com.microsoft.copilot.eclipse.swtbot.test/probe-scripts/<plan-slug>-<tc-id>.json`

- (e.g. `chat-send-receive-001.json`). Start every probe with this preamble that

- settles the workbench and opens the Copilot Chat view:

- 2. Run it (cross-platform command; use `./mvnw` on macOS/Linux):

- ```powershell

- Root `clean verify` is the recommended default. Tycho prefers each

- bundle's freshly-packaged jar in `<module>/target/` and silently falls

- back to the stale Maven cache if the jar is missing — making `setData`

- markers and other source edits appear to be ignored.

-

- The narrower `./mvnw -pl com.microsoft.copilot.eclipse.swtbot.test -am

- -Dprobe.script=... verify` shortcut is **experimental**: in practice

- it can fail dependency resolution (observed: `Missing requirement:

- com.microsoft.copilot.eclipse.core 0.0.0`, with Tycho choosing the

- wrong `osgi.arch` on aarch64). Use only as a follow-up, after a green

- root `clean verify`.

-

- 3. Read results under `com.microsoft.copilot.eclipse.swtbot.test/target/probe-results/`:

-

- ```

- results.json # pass/fail summary

- workspace.log # sandbox Eclipse .metadata/.log

- screenshots/

- <id>.png # from `screenshot` steps

- FAILED-stepNN-<action>.png # auto-captured on step failure

- ui-dumps/<id>.xml # from `dumpUi` steps

- ```

-

- If `-Dprobe.script` is unset the test is skipped, so ordinary `./mvnw verify`

- runs are unaffected.

-

- > **Linux headless:** prefix with `xvfb-run -a`. **macOS:** the swtbot test

- > pom carries a `swtbot-osx` profile that injects `-XstartOnFirstThread`

- > into the test JVM via the `swtbot.platformArgLine` placeholder

- > interpolated into `<argLine>`. If you ever edit

- > `com.microsoft.copilot.eclipse.swtbot.test/pom.xml`, **keep the

- > `${swtbot.platformArgLine}` token in `<argLine>`** — replacing it with a

- > plain literal silently drops the macOS arg and the workbench fails with

- > `SWTException: Invalid thread access`. **Stale cache recovery:**

- > `Remove-Item -Recurse -Force $env:USERPROFILE\.m2\repository\com\microsoft\copilot\eclipse`

- > then re-run `clean verify`.

-

- ## How the runner works

-

- `ProbeRunner` is a JUnit 4 test Tycho launches under a real Eclipse workbench

- (`useUIHarness=true`, `useUIThread=false`). It:

-

- 1. Loads the JSON probe pointed at by `-Dprobe.script`.

- 2. Walks each step via `SWTWorkbenchBot`, wrapping every step in a uniform

- try/catch so one failure doesn't crash the run.

- 3. Writes `results.json`, screenshots (PNG — `SWTBotPreferences.SCREENSHOT_FORMAT`

- is pinned), and widget-tree dumps.

- 4. Fails the Maven build if any `failFast` step failed.

-

- Before the bot starts, the runner pre-populates configuration-scope

- preferences so Quick Start, What's New, Welcome, and "Terminal Support

- Unavailable" dialogs don't pop during a probe — probes see a clean workbench.

-

- `screenshot` and `dumpUi` capture from the **on-screen workbench shell**:

- `StepExecutor#captureWorkbenchShell` resolves the active workbench

- `Shell`'s bounds, then snapshots those screen pixels with

- `java.awt.Robot.createScreenCapture` (with SWTBot's full-display

- `bot.captureScreenshot` as a final fallback). SWT's own GC-based capture

- (`new GC(display).copyArea(...)` and `Shell#print(GC)`) returns blank

- images on macOS Cocoa for workbench shells — Robot is the workaround.

- Caveats:

-

- - **macOS:** the JVM running the tests needs **Screen Recording**

- permission (System Settings → Privacy & Security → Screen Recording).

- Without it Robot returns blank or wallpaper-only frames.

- - **Headless AWT:** `-Djava.awt.headless=true` makes `new Robot()` throw,

- and the SWTBot fallback is also blank on Cocoa. Don't set it.

- - **Windows HiDPI:** `Shell#getBounds()` returns DIP coordinates; Robot

- expects raw screen pixels. At display scaling > 100% the captured

- region is undersized / clipped. (At 100% — typical CI runners — it's

- a pixel-perfect match.)

-

- ## Authoring rules

-

- ### Drive the UI, don't reach into services

-

- Probes **must** interact with the workbench the way a user does — SWTBot

- clicks, typing, menu navigation, keyboard shortcuts, or `invokeCommand`

- against a registered Eclipse command id (the same `IHandlerService#executeCommand`

- path that menus and key bindings trigger — useful when an entry point is

- buried in a popup like the Copilot status-bar menu, which is awkward to

- drive with SWTBot). Do not add actions that reflectively invoke private

- methods, call OSGi services to flip state, or read private fields; those

- hide real UX bugs and break on every internal refactor. If a scenario needs

- a new primitive (e.g. a chat-mode dropdown picker), extend the

- action/locator vocabulary in `StepExecutor` with a UI-level primitive,

- not a service shortcut.

-

- ### Tag widgets for `widgetId` locators

-

- SWTBot's blessed identification convention is

- `widget.setData("org.eclipse.swtbot.widget.key", "<stable-id>")` at widget

- construction, located from tests with the `widgetId` locator. Zero runtime

- cost, far more robust than class-name or creation-order lookup. Prefer

- `widgetId` over `widgetClass` whenever you control the widget's construction;

- use `widgetClass` only for platform / third-party widgets you can't tag.

-

- Current tags: `user-turn` (`UserTurnWidget`), `copilot-turn`

- (`CopilotTurnWidget`), `model-picker` (chat-model `DropdownButton`).

-

- ### Screenshots are artifacts; assertions catch regressions

-

- `screenshot` never fails a step. Validation has two layers:

-

- - **Structural** (machine-checked): `assertExists`, `waitFor`, specific

- locators. These failures land in `results.json` and drive the build exit

- code — the agent judges pass/fail from here.

- - **Visual** (agent-checked): after the run, open the PNGs under

- `screenshots/` with your vision tool and describe what you see. No

- baseline-image comparison in JSON (cross-platform font rendering makes

- pixel diffs flaky).

-

- Pair screenshots with assertions around every non-trivial interaction:

-

- ```json

- { "action": "screenshot", "id": "before-send" },

- { "action": "click", "locator": { "by": "buttonWithTooltip", "tooltip": "Send" } },

- { "action": "waitForIdle" },

- { "action": "assertExists", "locator": { "by": "widgetId", "value": "user-turn" } },

- { "action": "screenshot", "id": "after-send" }

- ```

-

- When a locator is wrong, insert a `dumpUi` step and inspect

- `ui-dumps/*.xml` to find the real widget class / id.

-

- ### Authentication prerequisite

-

- The Copilot JS agent reads its GitHub token from the host's standard Copilot

- store (`%USERPROFILE%\AppData\Local\github-copilot\apps.json` on Windows;

- `~/.config/github-copilot/apps.json` elsewhere) — the Tycho JVM inherits it.

-

- - **Signed-in host required** for any probe that exercises chat. Sign in via

- any Copilot client (Eclipse plugin, VS Code, `gh auth login`).

- - **Probing unauthed state:** assert on the "Sign in to GitHub" button that

- replaces the chat input; don't poll internal status managers.

- - CI-side auth bootstrap is out of scope for this skill.

-

- ## Reading results & interpreting failures

-

- Maven exit code `0` **and** `results.json` `.failed == 0` → overall pass.

- Otherwise open the failed step's `message`, its `FAILED-stepNN-*.png`, the

- nearest `ui-dumps/*.xml`, **and** `workspace.log` — many "widget not found"

- failures are downstream of a Copilot LS that never started or authenticated.

- Look for `!ENTRY com.microsoft.copilot.eclipse` entries and NPEs on

- `CopilotLanguageServer.*`.

-

- | Symptom | What's wrong / fix |

- |---|---|

- | `IllegalArgumentException: Missing required field: …` | Step JSON missing a field required by that action (see action table). |

- | `AssertionError: waitFor timed out: locator …` | Widget not appearing. Add a `dumpUi` before the failing step, check class/id, or raise `timeoutSec`. |

- | `assertExists failed: … shouldExist=true` | Locator didn't match. Confirm `by` matches the widget family. |

- | `click not supported on <Type>` | Wrapper has no `click()`; use the right `by` (e.g. `button`, not `label`). |

- | Tests skipped: "No probe script specified" | Pass `-Dprobe.script=probe-scripts/<name>.json`. |

- | `model-info-label` wait times out | Usually auth; open `workspace.log`. |

- | All screenshots are blank / identical ~4KB PNGs across every step | Robot capture failed. macOS: grant Screen Recording permission to the JVM and re-run. Verify by file size — real workbench frames are 50KB+; a uniform 4KB family means the on-screen capture path didn't run or returned empty pixels. |

- | Widget-id markers missing from `dumpUi` | Stale Maven cache — run full `clean verify` (see Quick start). |

-

- Quick pass/fail check:

-

- ```powershell

- Get-Content com.microsoft.copilot.eclipse.swtbot.test/target/probe-results/results.json |

- ConvertFrom-Json | Select-Object passed, failed, durationMs

- ```

-

- ```bash

- jq '{passed, failed, failed_steps: [.steps[] | select(.status=="failed") | {index, action, message}]}' \

- com.microsoft.copilot.eclipse.swtbot.test/target/probe-results/results.json

- ```

-

- ## Action reference

-

- Set `"failFast": false` on a step to record a failure without aborting the probe.

-

- | `action` | Required | Optional | Notes |

- |---|---|---|---|

- | `screenshot` | — | `id` | PNG of the active workbench window written to `screenshots/`. |

- | `sleep` | — | `timeoutSec` (default 1) | Plain `Thread.sleep`. |

- | `waitForIdle` | — | — | Flushes the SWT event queue. |

- | `pressKey` | `key` | `locator` | Accepts `ENTER`/`CR`, `ESC`/`ESCAPE`, `TAB`, `SPACE`, `BS`/`BACKSPACE`, `DELETE`. Without `locator`, presses on the active shell; with a `locator` targeting a text / styledText widget, sends the key directly (needed for chat input ENTER-to-send). |

- | `showView` | `idRef` (Eclipse view id) | — | Opens via `IWorkbenchPage#showView`. |

- | `closeView` | `idRef` | — | Hides the view if present. |

- | `invokeCommand` | `idRef` (command id) | — | Runs via `IHandlerService#executeCommand`. |

- | `click` | `locator` | — | Reflective `click()` on matched widget. |

- | `typeIn` | `locator`, `text` | — | Works on text / styled-text widgets. |

- | `clearElement` | `locator` | — | Sets text to empty. |

- | `waitFor` | `locator` | `timeoutSec` (default 30) | Polls until locator resolves. |

- | `waitForMethod` | `locator`, `method` | `timeoutSec` (default 30), `expectedValue` | Polls until a no-arg getter on the located widget returns non-null (and non-empty for `String`), or — if `expectedValue` is set — until `toString()` equals it. Invoked reflectively via the class hierarchy. Use for UI-exposed state not reachable via finders, e.g. `DropdownButton.getSelectedItemId()`. |

- | `assertExists` | `locator` | `shouldExist` (default true) | Asserts presence / absence. |

- | `dumpUi` | — | `id` | Writes widget hierarchy XML. |

- | `newSession` | — | — | Copilot-specific: triggers `newChatSession` command. |

- ## Locator reference

- Locators are JSON objects with a `by` discriminator. SWTBot is **not

- XPath-based** — stick to this vocabulary; extend `Locator` / `StepExecutor`

- if you need a new finder.

- | `by` | Other fields | What it finds |

- |---|---|---|

- | `viewId` | `id` | Eclipse view by id (`bot.viewById`). |

- | `label` | `text` | First label with the given text. |

- | `button` | `text` | First button with the given text. |

- | `buttonWithTooltip` | `tooltip` | First button whose tooltip matches (use for icon-only buttons like **Send**). |

- | `text` | `index` (default 0) | Nth text field. |

- | `styledText` | — | First StyledText (editors / chat input). |

- | `tree` | `labels` (array) | Tree path under the active tree. |

- | `cssId` | `value` | Widget whose `CssConstants.CSS_ID_KEY` equals `value` (e.g. `chat-content-viewer`, `chat-action-bar`). Walks the full shell tree. |

- | `cssClass` | `value` | Widget whose `CssConstants.CSS_CLASS_NAME_KEY` contains `value` as a token. `model-info-label` is only set on a **completed** Copilot turn, so waiting on it is a reliable "response received" signal. |

- | `widgetId` | `value` | Widget tagged with `setData("org.eclipse.swtbot.widget.key", value)`. **Preferred** identifier for widgets you own. |

- | `widgetClass` | `value` | First widget whose `getClass().getSimpleName()` equals `value` (walks the full shell tree). Fallback for widgets you can't tag. |

- ## Canonical example: send a prompt and verify a response

- { "action": "clearElement", "locator": { "by": "styledText" } },

- { "action": "typeIn", "locator": { "by": "styledText" }, "text": "your prompt" },

- { "action": "screenshot", "id": "typed" },

-

- { "action": "waitForMethod",

- "locator": { "by": "widgetId", "value": "model-picker" },

- "method": "getSelectedItemId",

- "timeoutSec": 60 },

-

- { "action": "click", "locator": { "by": "buttonWithTooltip", "tooltip": "Send" } },

-

- { "action": "waitFor", "locator": { "by": "widgetId", "value": "user-turn" },

- "timeoutSec": 30 },

- { "action": "assertExists", "locator": { "by": "widgetId", "value": "user-turn" } },

-

- { "action": "waitFor", "locator": { "by": "cssClass", "value": "model-info-label" },

- "timeoutSec": 120 },

- { "action": "screenshot", "id": "agent-response" },

- { "action": "assertExists", "locator": { "by": "widgetId", "value": "copilot-turn" } }

- Key signals:

-

- - `model-picker.getSelectedItemId()` non-null — the workbench has resolved an

- active chat model. Sending before this NPEs in `onSendInternal` with

- "activeModel is null" because auth + model fetch complete asynchronously.

- - `user-turn` — the user turn has rendered (Send button dispatched the prompt).

- - `model-info-label` — only appears once a Copilot turn has **completed**

- (rendered in the turn's footer at end of streaming); the reliable "response

- fully received" signal.

- - `copilot-turn` — the assistant turn container; good secondary assertion.

-

- Without a signed-in host the language server can't complete a turn, so

- `model-info-label` never appears and the step times out — itself a useful

- diagnostic (check `workspace.log`).

-

- ## Extending the vocabulary

- Everything the probe understands lives in

- `com.microsoft.copilot.eclipse.swtbot.test/src/.../probe/`:

- - `ProbeStep.java` / `Locator.java` — JSON shape.

- - `StepExecutor.java` — action dispatch + locator resolution.

- - `ProbeRunner.java` — runner loop, reporting, screenshots.

- Add a case to the `switch` in `StepExecutor#execute` (or `resolve`) and the new

- action is usable from every probe.

- ### Keep one probe focused

- Each JSON script represents one test case. Split unrelated behaviours into

- separate probes so a single `FAILED-step…` screenshot tells you what broke.