dotagents-qa · diff
git:20260809.8745c0c to git:20260810.f18ce1a
68 added, 348 removed. Audit A to A.
---
name: dotagents-qa
- description: QA dotagents behavior changes in a Docker sandbox. Use when changes may affect dotagents install, sync, list, doctor, skill placement, agent symlinks, MCP or hook config generation, user scope, subagent runtime files, or package/runtime behavior.
+ description: QA dotagents changes and published releases in Docker, including CLI lifecycles, user/global scope, real plugins, and Claude, Codex, OpenCode, or Pi projections. Use when behavior, packaging, scopes, or harness integration needs runtime proof.
+ spec_hash: df3db16f11bb
---
# dotagents QA
- Do real QA for the change in front of you. Docker is the safety boundary, not
- the test plan: use it so dotagents cannot write to host agent config, host home
- directories, or host cache state while you build fixtures that prove the
- changed behavior.
-
- Answer the practical question: "With this local dotagents build, does the
- changed behavior still install, sync, and wire the expected files?"
-
- ## 1. Understand The Change
-
- Start from the diff and identify the behavior that could regress:
-
- ```bash
- git status --short
- git diff --stat
- git diff -- <paths>
- ```
-
- Write down the QA target before running commands:
- - Which command path changed: `install`, `sync`, `list`, `doctor`, `add`,
- `remove`, `mcp`, `trust`, `init`, package runtime, or scope resolution.
- - Which surfaces must be inspected: `.agents/skills`, agent skill symlinks,
- MCP config, hook config, subagent runtime files, lockfile, gitignore, CLI
- output, or user scope.
- - Which fixture shape proves it: checked-in example, local skills, nested
- skills, wildcard source, specific agents, MCP entries, hooks, existing
- broken state, user-scope state, or remote source.
+ Prove the requested behavior in Docker without touching host agent homes, caches, plugin registries, or credentials. Treat Docker as the safety boundary and the changed or released behavior as the test plan.
- Read the targeted reference before running runtime-specific QA:
- - Core install/sync example: [references/core-agentic-qa.md](references/core-agentic-qa.md)
- - Plugin runtime verification matrix: [references/plugin-runtime.md](references/plugin-runtime.md)
- - Codex custom agents and plugin marketplace/install proof: [references/codex.md](references/codex.md)
- - Claude Code files, plugin validation, and runtime caveats: [references/claude.md](references/claude.md)
- - Cursor files/runtime caveats: [references/cursor.md](references/cursor.md)
- - Grok Build files/runtime caveats: [references/grok.md](references/grok.md)
- - OpenCode files/runtime caveats: [references/opencode.md](references/opencode.md)
+ ## 1. Define the contract
- Run focused Vitest coverage for logic bugs. Use this skill for end-to-end QA
- evidence, not as a substitute for regression tests.
+ Before commands, state:
- ## 2. Enter A Docker Sandbox
+ - the exact subject: local checkout, packed local build, or published version;
+ - the commands and semantics at risk;
+ - project scope, user scope, or both;
+ - the harnesses involved;
+ - the fixture and evidence that constitute a pass.
- Build the repo-local QA image when it is missing, when this Dockerfile changes,
- or when the repo `packageManager` pnpm version changes:
+ For a published release, verify its registry metadata and install that exact version. Never silently test the local checkout instead. If you fix a discovered defect, report the release failure and packed-local-build pass separately.
- ```bash
- docker build \
- -f skills/dotagents-qa/Dockerfile \
- -t dotagents-qa:local \
- skills/dotagents-qa
- ```
+ Read the relevant references before acting:
- The image installs the latest npm-published Codex, Claude Code, and OpenCode
- CLIs (`codex`, `claude`, `opencode`). Use them for version checks, help-output
- checks, and optional isolated runtime probes. Their presence does not prove
- runtime discovery by itself; for plugins, prefer native dry-run/management
- commands where available: `claude plugin validate` for Claude plugin and
- marketplace shape, and `codex plugin marketplace add/list` plus
- `codex plugin add/list` for Codex marketplace installation. Authenticated
- model-backed checks are still explicit opt-ins. Keep runtime credentials out of
- fixtures and retained `/qa-out` artifacts.
+ - Docker, non-root setup, exact release installation, and baseline commands: [references/docker-sandbox.md](references/docker-sandbox.md)
+ - Ordinary install/sync behavior: [references/core-agentic-qa.md](references/core-agentic-qa.md)
+ - Real plugins, full lifecycle, native clients, and user/global scope: [references/release-plugin-matrix.md](references/release-plugin-matrix.md)
+ - Plugin adapters and automated proof: [references/plugin-runtime.md](references/plugin-runtime.md)
+ - Harness details: [Claude](references/claude.md), [Codex](references/codex.md), [OpenCode](references/opencode.md), [Pi](references/pi.md), [Cursor](references/cursor.md), [Grok](references/grok.md)
- Use an interactive container so the QA steps stay change-specific:
+ Planning is part of acting: read the relevant references before proposing a command sequence, not only before executing it.
- ```bash
- REPO="$(pwd)"
- OUT="$(mktemp -d "${TMPDIR:-/tmp}/dotagents-qa.XXXXXX")"
- docker run --rm -it \
- -v "$REPO:/host-repo:ro" \
- -v "$OUT:/qa-out" \
- dotagents-qa:local
- ```
+ ## 2. Establish isolation
- If your tool environment is not attached to a TTY, use `-i` instead of `-it`
- and feed the same commands with a here-doc. Keep `-i`; without stdin attached,
- the container shell will receive no script.
+ Use the repo QA image. Rebuild it when the Dockerfile, pnpm version, or required current harness versions changed.
- Inside the container:
+ Run package and runtime work as a non-root user. Keep these inside Docker or disposable directories:
```bash
- set -euo pipefail
- export CI=1
export HOME=/sandbox/home
export DOTAGENTS_STATE_DIR=/sandbox/state
export DOTAGENTS_HOME=/sandbox/user-agents
-
- mkdir -p "$HOME" "$DOTAGENTS_STATE_DIR" "$DOTAGENTS_HOME" /sandbox/repo
- tar -C /host-repo \
- --exclude=.git \
- --exclude=node_modules \
- --exclude=.turbo \
- --exclude=coverage \
- --exclude=core \
- --exclude='*.tsbuildinfo' \
- --exclude='packages/*/dist' \
- -cf - . | tar -C /sandbox/repo -xf -
-
- cd /sandbox/repo
- pnpm install --frozen-lockfile
- pnpm build
+ export CODEX_HOME=/sandbox/codex-home
+ export CLAUDE_CONFIG_DIR=/sandbox/claude-home
```
- Run package commands as the non-root `node` user. Root can bypass chmod-based
- permission checks, which can mask or invert filesystem regression tests.
-
- For non-interactive QA, copy as root, then hand the repo to `node` before
- running package scripts:
-
- ```bash
- chown -R node:node /sandbox
- su -s /bin/bash node -c '
- set -euo pipefail
- export CI=1
- export HOME=/sandbox/home
- export DOTAGENTS_STATE_DIR=/sandbox/state
- export DOTAGENTS_HOME=/sandbox/user-agents
- cd /sandbox/repo
- pnpm install --frozen-lockfile
- pnpm build
- pnpm check
- pnpm qa:example
- '
- ```
+ Mount the host checkout read-only and copy it into the container without `.git`, `node_modules`, `dist`, coverage, or `*.tsbuildinfo`. Never point user-scope commands or client CLIs at the host's real home.
- Run `pnpm check` inside Docker unless the change requires a narrower target or
- the check is already known to be unrelated. If `build` or `check` fails, treat
- that as a QA finding and stop before fixture work unless you are explicitly
- isolating the playbook mechanics. If skipped or bypassed, report why.
+ Do not forward auth for file, validation, marketplace, or resource-discovery checks. Copy credentials only for a separately authorized model-backed proof, scrub them afterward, and never retain them in `/qa-out`.
- ## 3. Prefer The Checked-In Agentic QA
+ ## 3. Run baseline validation
- Use the checked-in example QA for ordinary install/sync QA:
+ For a local plugin/runtime change, normally run:
```bash
+ pnpm install --frozen-lockfile
+ pnpm check
pnpm qa:example
- ```
-
- The QA runner builds the local CLI, copies `examples/full/` to a temp project, and
- asserts:
- - `install`, `list`, `doctor --fix`, and `doctor` complete successfully
- - managed skills under `.agents/skills/`
- - Claude/Cursor skill symlink behavior
- - MCP files for Claude, Cursor, Codex, and OpenCode
- - hook files for Claude and Cursor
- - canonical installed subagent under `.agents/agents/`
- - generated subagent runtime files for Claude, Cursor, Codex, and OpenCode
- - canonical installed plugin bundle under `.agents/plugins/`
- - Codex repo-scoped plugin marketplace under `.agents/plugins/marketplace.json`
- - generated plugin runtime files for Claude, Cursor, Codex, Grok, and OpenCode component projections
- - `sync` repair after deleting representative generated files
-
- Use `node skills/dotagents-qa/scripts/qa-example.mjs all --keep` when you need
- to inspect the temp project; the script prints the retained path.
-
- For automatic real-client plugin proof, run:
-
- ```bash
pnpm qa:plugins
```
- This runs every installed no-auth client proof in a fresh single-target fixture:
- Claude validates, adds, installs, and lists the plugin, then verifies one skill,
- zero leaked client-extension agents, and both portable MCP servers. Codex adds
- the generated marketplace and installs/lists the plugin. Grok Build runs
- `plugin list` and `plugin details` when available. Missing client CLIs are
- reported and skipped; the command fails if none are installed.
-
- Cursor currently has no plugin validation or marketplace command in its desktop
- CLI, so Cursor remains covered by the isolated per-harness integration contract rather
- than being mislabeled as client-level E2E. The OpenCode proof remains available
- as an explicit task but is not part of the automatic client set.
-
- For paid Codex runtime proof of generated custom agents, run the runtime proof
- outside Docker only when the branch affects Codex custom agents or when
- reporting that Codex itself works:
-
- ```bash
- node skills/dotagents-qa/scripts/qa-example.mjs codex-runtime --keep
- ```
-
- That mode copies Codex auth/config into a temp `CODEX_HOME`, marks only the
- temp example project trusted, and asserts that Codex can spawn the generated
- `.codex/agents/code-reviewer.toml` agent. See
- [references/codex.md](references/codex.md) before changing or running this
- path.
-
- ## 4. Build A Manual Fixture Only When Needed
-
- Create a temp project manually only when the checked-in example does not cover
- the changed behavior. Start from this shape and add only what the diff needs.
-
- ```bash
- fixture=/sandbox/fixture
- mkdir -p "$fixture/local-skills" "$fixture/local-agents/agents"
- for skill in review commit; do
- mkdir -p "$fixture/local-skills/$skill"
- printf -- "---\nname: %s\ndescription: Fixture %s skill.\n---\n\n%s fixture.\n" \
- "$skill" "$skill" "$skill" > "$fixture/local-skills/$skill/SKILL.md"
- done
-
- cat > "$fixture/local-agents/agents/code-reviewer.md" <<'EOF'
- ---
- name: code-reviewer
- description: Review code for correctness.
- ---
-
- Review the current diff and return findings with file references.
- EOF
-
- cat > "$fixture/agents.toml" <<'EOF'
- version = 1
- agents = ["claude", "cursor", "codex", "opencode"]
-
- [[skills]]
- name = "review"
- source = "path:./local-skills/review"
-
- [[skills]]
- name = "commit"
- source = "path:./local-skills/commit"
-
- [[mcp]]
- name = "fixture"
- command = "node"
- args = ["-e", "process.exit(0)"]
-
- [[hooks]]
- event = "Stop"
- command = "echo fixture"
-
- [[subagents]]
- name = "code-reviewer"
- source = "path:./local-agents"
- EOF
- ```
-
- Useful fixture changes:
- - Skill resolution: nested `skills/` layouts, wildcard sources, duplicate
- names, local paths outside `.agents`, or the exact `path:` shape touched.
- - Agent placement: include only affected agents, or include all supported
- agents when shared config or registry behavior changed.
- - MCP and hooks: use the exact command, URL, headers, env refs, hook event, or
- matcher affected by the diff.
- - Subagents: include a portable Markdown fixture under `agents/`, assert the
- installed canonical file in `.agents/agents/`, assert generated runtime files
- for Claude/Cursor/Codex/OpenCode, and inspect `agents.lock`.
- - Plugins: include a local plugin fixture with representative components,
- assert the installed canonical bundle in `.agents/plugins/`, assert generated
- runtime files for Claude/Cursor/Codex/Grok/OpenCode, and inspect `agents.lock`.
- - Sync and doctor: pre-create broken or legacy state, then prove repair and
- diagnostics.
- - User scope: set both `HOME` and `DOTAGENTS_HOME`, pass `--user`, and inspect
- generated files under those temp directories; never use the host home
- directory.
- - Package/runtime: run the built CLI as above, or pack/install the package only
- when the packaging path itself changed.
- - Remote sources: use `getsentry/skills`; avoid remotes for ordinary
- install-location checks.
+ Add focused regression tests for every confirmed logic bug. A failing baseline is a finding; do not bypass it silently. If a check is irrelevant or unavailable, record why.
- ## 5. Exercise The CLI
+ ## 4. Exercise complete lifecycles
- Run the built local CLI from the fixture. Capture output and inspect generated
- files, not just exit codes.
+ Use the CLI from a fresh fixture and inspect output plus files:
```bash
- cli=(node /sandbox/repo/packages/dotagents/dist/cli/index.js)
- cd "$fixture"
-
- "${cli[@]}" install | tee /qa-out/install.out
- "${cli[@]}" list | tee /qa-out/list.out
- "${cli[@]}" doctor --fix | tee /qa-out/doctor-fix.out
- "${cli[@]}" doctor | tee /qa-out/doctor.out
+ dotagents init --agents claude,codex,opencode,pi
+ dotagents add <source> [name]
+ dotagents list --json
+ dotagents doctor
+ dotagents install
```
- Assert what matters for this change. Examples:
+ Inspect `agents.toml`, `agents.lock`, canonical skills/plugins, ownership markers, marketplaces, manifests, harness projections, and warnings. Delete representative managed artifacts, run `dotagents sync`, and prove exact repair. Run `dotagents remove <name>` and prove canonical and generated cleanup.
- ```bash
- grep -q "review" /qa-out/list.out
- grep -q "commit" /qa-out/list.out
- test -f .agents/skills/review/SKILL.md
- test -f .agents/skills/commit/SKILL.md
- test -L .claude/skills
- test -f .mcp.json
- test -f .cursor/mcp.json
- test -f .codex/config.toml
- test -f .opencode/opencode.jsonc
- test -f .claude/settings.json
- test -f .cursor/hooks.json
- test -f .agents/agents/code-reviewer.md
- test -f .claude/agents/code-reviewer.md
- test -f .cursor/agents/code-reviewer.md
- test -f .codex/agents/code-reviewer.toml
- test -f .opencode/agents/code-reviewer.md
- test -f .agents/plugins/qa-tools/plugin.json
- test -f .agents/plugins/marketplace.json
- test -f .claude-plugin/marketplace.json
- test -f .cursor-plugin/marketplace.json
- test -f .agents/plugins/qa-tools/.claude-plugin/plugin.json
- test -f .agents/plugins/qa-tools/.cursor-plugin/plugin.json
- test -f .agents/plugins/qa-tools/.codex-plugin/plugin.json
- test -f .grok/plugins/qa-tools/.dotagents-managed
- test -L .opencode/skills/plugin-qa
- test -f .opencode/skills/.dotagents-managed/plugin-qa
- test ! -e .opencode/agents/plugin-reviewer.md
- grep -q "code-reviewer" agents.lock
- grep -q "qa-tools" agents.lock
- grep -q "Generated by dotagents" .claude/agents/code-reviewer.md
- grep -q "Generated by dotagents" .codex/agents/code-reviewer.toml
- ```
+ Invoke every lifecycle command named by the contract. In particular, run `dotagents install` explicitly even though `add` also installs, and assert the config and lockfile paths directly rather than inferring them from `list` output.
- For `sync` or subagent writer changes, break the generated state in the way the
- diff claims to repair, then verify the repair:
+ Cover an unambiguous skill source as well as plugins so plugin-first discovery still falls back correctly. Use `--all` only with a controlled small catalog.
- ```bash
- rm .mcp.json .claude/skills .claude/agents/code-reviewer.md .codex/agents/code-reviewer.toml
- rm .agents/plugins/marketplace.json .claude-plugin/marketplace.json .agents/plugins/qa-tools/.claude-plugin/plugin.json
- rm .cursor-plugin/marketplace.json .agents/plugins/qa-tools/.cursor-plugin/plugin.json
- rm .agents/plugins/qa-tools/.codex-plugin/plugin.json
- rm -rf .grok/plugins/qa-tools
- rm .opencode/skills/plugin-qa
- "${cli[@]}" sync | tee /qa-out/sync.out
- test -f .mcp.json
- test -L .claude/skills
- test -f .claude/agents/code-reviewer.md
- test -f .codex/agents/code-reviewer.toml
- test -f .agents/plugins/marketplace.json
- test -f .claude-plugin/marketplace.json
- test -f .cursor-plugin/marketplace.json
- test -f .agents/plugins/qa-tools/.claude-plugin/plugin.json
- test -f .agents/plugins/qa-tools/.cursor-plugin/plugin.json
- test -f .agents/plugins/qa-tools/.codex-plugin/plugin.json
- test -f .grok/plugins/qa-tools/.dotagents-managed
- test -L .opencode/skills/plugin-qa
- test -f .opencode/skills/.dotagents-managed/plugin-qa
- test ! -e .opencode/agents/plugin-reviewer.md
- ```
+ For user plugins, test both `--user` and `--global`. Isolate `HOME` and `DOTAGENTS_HOME`; inspect global Claude, Codex, OpenCode, and Pi paths; repair with one flag spelling and remove with the other.
- For user-scope changes:
+ ## 5. Test representative plugins
- ```bash
- cd /sandbox
- "${cli[@]}" --user install | tee /qa-out/user-install.out
- test -f "$DOTAGENTS_HOME/agents.toml"
- test -d "$DOTAGENTS_HOME/skills"
- ```
+ When plugin compatibility is in scope, inspect current manifests and test:
- Copy useful evidence before leaving the container:
+ - `getsentry/agent-plugin`;
+ - `vercel/vercel-plugin`;
+ - a selected plugin such as `frontend-design` from `anthropics/claude-plugins-official`.
- ```bash
- cp -a "$fixture" /qa-out/fixture
- cp -a "$DOTAGENTS_HOME" /qa-out/user-agents 2>/dev/null || true
- ```
+ In a proposed compatibility plan, name all three repositories and explain that selectors, versions, commits, and component counts come from manifest inspection at execution time.
- ## 6. Real Agent Clients
+ Use a fresh project per source. Record source commits. Do not guess repository names, selectors, versions, component counts, or layouts.
- Use real clients only when discovery or registration in Claude, Cursor, Codex,
- VS Code, or OpenCode changed. Docker proves generated files and symlinks; it
- does not prove an installed host client notices them.
+ ## 6. Prove harnesses independently
- Keep host-client checks isolated with explicit temp homes/config dirs where
- the client supports it. If a client cannot run without reading host state, say
- so and report the Docker-generated files you inspected instead.
+ Keep per-harness fixtures isolated. In particular, do not enable Pi in the OpenCode proof: OpenCode can read Pi's shared `.agents/skills` links and create a false pass.
- Inside the QA image, start with cheap client availability checks:
+ - Claude: validate generated plugin and marketplace manifests, then marketplace add, install, list, and details.
+ - Codex: add the project or user marketplace root, list available plugins, install, and list enabled plugins.
+ - OpenCode: run the exact command `opencode debug skill` and assert exact projected skill names and locations.
+ - Pi: in a separate Pi-only fixture, verify expected skill links, resolved targets, and `.dotagents-managed/<skill>` ownership markers.
- ```bash
- codex --version
- claude --version
- opencode --version
- ```
+ These no-auth checks prove validation, registration, installation, or resource discovery. They do not prove model-backed invocation. Never claim a plugin skill or subagent executed without a model interaction and visible evidence.
- Codex subagents need real runtime proof before claiming Codex loaded them. Use
- `node skills/dotagents-qa/scripts/qa-example.mjs codex-runtime --keep`;
- `codex debug prompt-input` is not enough unless it visibly includes the
- generated agent name or instructions. Project-scoped `.codex/agents/` load only
- when Codex trusts the project. See [references/codex.md](references/codex.md).
+ ## 7. Report
- Claude has no cheap dry-run skill list. If auth/network/model cost is
- acceptable, run a minimal non-interactive prompt from the temp project;
- otherwise report it as skipped.
+ Include:
- Provider and gateway checks are runtime-specific. Before claiming model-backed
- runtime proof, run the check inside Docker and report the exact provider config
- used with secret values redacted.
+ - exact dotagents and harness versions;
+ - Docker image/setup and non-root user;
+ - fixture shapes and source commits;
+ - commands and inspected paths;
+ - assertions that passed;
+ - confirmed defects and their regression tests;
+ - release results versus local-fix results;
+ - skipped authenticated checks and residual risk;
+ - retained `/qa-out` location, if any.
- ## 7. Report Evidence
+ Reporting is lossless. Enumerate every supplied command, version, source commit, and inspected path, even when repetitive. A category summary such as “state and marketplaces” is insufficient; do not invent missing values.
- Report:
- - the changed behavior you targeted
- - Docker image and setup used
- - fixture shape and why it matched the diff
- - commands run
- - generated files or command output inspected
- - assertions that passed
- - `/qa-out` host path if retained for debugging
- - skipped checks and residual risk
+ Keep the report candid: filesystem proof, native management proof, resource-discovery proof, and model-backed execution are different confidence levels.