git:20260902.405b5c5 to git:20260907.f0de30d

- apra-fleet # installs for Claude Code (default)

- apra-fleet --llm agy # or OpenCode/Codex/Copilot

- cd ~/.apra-fleet/bin && apra-fleet start # start the apra-fleet

- ```

-

- **2. Connect your agent.** Load the fleet server in Claude Code with

- `/mcp` (or restart your provider CLI). Your agent now has a fleet.

-

- **3. Register members -- in plain language.** apra-fleet is driven

- conversationally through any MCP-capable agent:

-

- > "Register a local member called `doer`. Register another called

- > `reviewer`. Pair them."

-

- > "Register 192.168.1.10 as `build-server`. Username akhil, work folder

- > `/home/akhil/projects/myapp`."

-

- Remote passwords are collected out-of-band -- typed into a separate

- terminal, never the chat -- used once to set up SSH keys, then forgotten.

-

- **4. Run your first workflow:**

-

- ```bash

- apra-fleet workflow hello-world

- ```

-

- Then point the fleet at real work:

-

- ```bash

- apra-fleet workflow fleet-sprint \

- --issue my-project-epic --members doer \

- --branch fleet-sprint/first-run --base main

- ```

-

- Open the dashboard, watch your fleet PLAN->BUILD->REVIEW->TEST->SHIP in a loop till closure

-

- > **New to fleet-sprint?** Read the

- > [fleet-sprint Getting Started Guide](https://apra-labs.github.io/apra-fleet/fleet-sprint-getting-started.html)

- > ([Markdown](docs/fleet-sprint-getting-started.md) if you're reading this on

- > GitHub -- [PDF](docs/fleet-sprint-getting-started.pdf)) -- a plain-English walkthrough

- > of what it does, what you need to prepare (beads backlog, `deploy.md`,

- > test playbooks, member registration), how to launch and monitor a sprint,

- > and what's automated versus what's still your call.

-

- **Running fleet-sprint after npm install:** the `apra-fleet workflow

- fleet-sprint ...` command above is the same one command for everyone --

- whether you installed via `npm install -g @apralabs/apra-fleet`, the

- standalone binary, or a git-clone dev checkout. There is no separate

- `fleet-sprint` command to install or remember. See

- [the full flag reference](packages/apra-fleet-se/fleet-sprint/docs/README.md)

- for every option.

-

- ## How it works

-

- ### Layered Architecture

-

- <picture>

- <source media="(prefers-color-scheme: dark)" srcset="assets/marketing/fleet-stack-dark.svg">

- <img src="assets/marketing/fleet-stack-light.svg" alt="apra-fleet layered architecture stack: dependencies from OS primitives up to autonomous engineering orchestrators" width="100%">

- </picture>

-

- **Component Docs:** [fleet-sprint](packages/apra-fleet-se/fleet-sprint/docs/README.md) | [auto-sprint.js](packages/apra-fleet-se/apra-pm/docs/sprint-workflow.md) | [apra-fleet-client](packages/apra-fleet-client/docs/overview.md) | [apra-pm](packages/apra-fleet-se/apra-pm/README.md) | [apra-fleet-mcp](docs/mcp-tools.md) | [Agent Roles](packages/apra-fleet-se/docs/role-contracts.md)

-

- ### Fleet Dispatch Topology

-

- ```mermaid

- flowchart LR

- CP["Control Plane<br/>(Server, Engine, Supervisor)"] -->|Dispatch & Sync| M1["MacBook<br/>(Claude)"] & M2["Linux GPU<br/>(vLLM)"] & M3["Cloud VM<br/>(AGY)"] & M4["Windows<br/>(OpenCode)"]

- ```

-

- - **Fleet server**: the control plane. Registers members, dispatches commands and prompts, moves files, brokers credentials. Speaks MCP, so any MCP-capable agent can drive a fleet.

- - **Members**: real machines running provider CLIs. Composes provider-native permissions before every dispatch; unattended modes are scoped, never blanket.

- - **Workflow engine**: runs workflow programs with phases, retries, turn budgets, resumable sessions, per-activity persistent state, and a cooperative pause/resume gate any workflow can hook into.

- - **Supervisor**: always-on layer -- launch, pause/resume, & stop sprints over HTTP, member reservation ledger, crash watchdog (including a live "paused" state and base-branch-drift indicator), run history.

-

- ## Knowledge Layer

-

- Every agent session starts by calling `kb_session_prime`. The KB checks which

- files have changed since last read and returns exactly those. Unchanged files

- are served from cached summaries -- no re-read, no wasted tokens.

-

- ```

- Cold session: kb_session_prime returns stale_files=[a.ts, b.ts, c.ts]

- Agent reads all three, calls kb_capture for each.

- Warm session: kb_session_prime returns stale_files=[], session_warm=true

- Agent works from KB summaries. Zero file reads.

- ```

-

- MCP tools that ship with the KB:

-

- | Tool | What it does |

- |------|--------------|

- | `kb_session_prime` | Prime a session: stale files, fresh summaries, GitNexus call list |

- | `kb_capture` | Store a learning, context-cache, runbook, or knowledge entry |

- | `kb_query` | Two-level FTS retrieval (L1: title+summary, L2: full content) |

- | `kb_list` | Audit-list entries by confidence/type/module/symbol (read-only, no use_count bump) |

- | `kb_context` | Batch file freshness check (single git call for N files) |

- | `kb_invalidate` | Mark files stale immediately (also called by the git hook) |

- | `kb_promote` | Advance confidence: UNVERIFIED -> INFERRED -> CONFIRMED |

- | `kb_harvest` | Extract learnings from a session transcript (auto-fires after execute_prompt) |

- | `kb_export` | Write live CONFIRMED entries to `.fleet/kb-canonical.json` -- the git-shareable team bible |

- | `kb_setup` | Install git hook, write provider config, store remote token encrypted |

-

- Every KB tool call is scoped to the repo it is about -- a fleet server

- handling many members across many repos never lets one repo's learnings land

- in another repo's KB. Scope is normally derived from the caller's repo path;

- tools also accept an explicit `repo_remote_url` so a remote member (whose

- work folder is a path on another host, unreachable from the fleet server's

- filesystem) resolves to the same project KB as a local clone of that repo

- instead of a shared fallback database. The automatic post-prompt harvest and

- the `code_context` KB enrichment path both forward this URL too, and an

- unreachable work-folder path is never silently swapped for the fleet

- server's own working directory -- see

- [Per-repo KB isolation](docs/knowledge-layer.md#per-repo-kb-isolation) for

- the full anchor and cache-keying rules.

-

- The backend is swappable: start with local SQLite, add a central HTTP server for

- a team, or plug in Postgres later -- all via a one-line config change.

-

- See [docs/knowledge-layer.md](docs/knowledge-layer.md) for the full guide.

-

- ## Explore with agents. Operate with programs.

-

- There are two ways to orchestrate agents, and apra-fleet is built on the

- observation that you need both -- at different stages of a workflow's life:

-

- - **Exploration mode.** While a workflow is still being discovered, let an

- LLM orchestrate: flexible, adaptive, and token-hungry -- every step is a

- decision, and every decision costs thinking.

- - **Operation mode.** Once you know what must happen, the control flow

- becomes a deterministic workflow program. Shell, git, and file steps run

- through `execute_command` -- zero tokens. The model is invoked only at

- the corners that genuinely require judgment (`execute_prompt`): review

- this diff, plan this backlog, decide this exception.

-

- <picture>

- <source media="(prefers-color-scheme: dark)" srcset="assets/marketing/cost-collapse-dark.svg">

- <img src="assets/marketing/cost-collapse-light.svg" alt="Cost per apra-fleet e2e run: four real LLM-driven runs ranging $0.46-$3.05, then ~$0.00 / run forever after switching to a deterministic workflow." width="100%">

- </picture>

-

- That is not a projection -- it is this repository's own e2e setup+teardown

- step, before and after we hardened it. Development tokens are not

- operating tokens: pay once to discover the workflow, then run it free.

-

- | | LLM-orchestrated (explore) | Workflow-orchestrated (operate) |

- |---|---|---|

- | Control flow | the model decides each step (tokens) | deterministic program (free) |

- | Shell / git / file steps | narrated through the model | `execute_command`, zero tokens |

- | Where the model runs | everywhere | judgment nodes only (`execute_prompt`) |

- | Cost curve | scales with every step | scales with thinking only |

- | Failure mode | drift and silent retries | typed errors, resumable state |

-

- The collapse is two-dimensional. As a workflow hardens, control flow moves

- from model to program -- and the judgment nodes that remain move from

- frontier models to cheaper ones, because a well-specified task no longer

- needs discovery-grade reasoning. **Develop a workflow with Claude;

- operationalize it on OpenCode against a local or OpenRouter model.** Same

- fleet, same workflow -- swap the members. Tier routing makes it a

- registration change, not a rewrite.

-

- Only a fleet makes that trade possible. Single-provider tools cannot leave

- their vendor; in-process frameworks cannot move orchestration out of the

- token path. Because apra-fleet's unit of execution is the member -- a

- machine plus a provider, swappable at registration -- the same hardened

- workflow runs on frontier models the day you design it and on commodity

- models every day after.

-

- fleet-sprint is this principle, lived: it began as LLM-orchestrated

- exploration; each discovered pattern was hardened into the deterministic

- engine; today the engine drives hour-long autonomous runs in which models

- are consulted only as planner, doer, reviewer, tester, and harvester.

-

- ## Compare to alternatives

-

- | Tool | Overlap | Where apra-fleet differs |

- |------|---------|--------------------------|

- | Single-agent coding assistants | AI writes code | A fleet adds agents that review, test, and deploy each other's work -- across vendors. |

- | CI self-hosted runners | Runs work on other machines | Conversational and stateful, not pipeline-triggered; agents carry context between phases. |

- | SkyPilot / dstack | Multi-machine compute | Coordinates agents and their context, credentials, and permissions -- not just jobs. |

- | Google A2A | Agent-to-agent messaging | An opinionated orchestration and operations layer, not just a transport. |

- | Agent frameworks (LangGraph, CrewAI, ...) | Multi-agent logic | Those compose agents inside one process; apra-fleet operates agents across real machines, providers, and days-long workflows. |

-

- When NOT to use it: a one-off single-file change needs no fleet.

-

- ## Security model, in one paragraph

-

- Secrets are entered out-of-band into a credential store and referenced as

- `{{secure.NAME}}` -- resolved server-side at execution, never visible to

- any LLM or log. Credentials scope to members, expire on TTL, and can carry

- a network egress policy (allow / deny / confirm). Every member runs with

- composed, provider-native permission files -- allow-listed tools, not

- god-mode. VCS access is provisioned and revocable per member, across GitHub,

- Bitbucket, and Azure DevOps -- host differences (URL shape, PR REST dialect,

- auth pattern, error vocabulary) are hidden behind a per-provider descriptor

- rather than leaking into shared code; see

- `docs/design-azure-devops-vcs-auth.md` for the Azure DevOps provider's

- credential-assembly and PAT-lifetime details. Permission

- composition verifies its own delivery: a grant is read back off the target

- member and structurally compared against what was intended before it is

- reported as applied, so a failed or partial write is surfaced as an

- explicit failure rather than a false success.

-

- ## Email Configuration

-

- The fleet `send_email` tool sends email via **SendGrid** or **SMTP**. Secrets

- are stored in the fleet credential store. Non-secret config (provider, host,

- port, from address) is passed by the workflow in each call.

-

- ### Storing secrets (one-time setup)

-

- Store email secrets via the CLI:

-

- ```bash

- # SendGrid API key

- apra-fleet secret --set sendgrid_api_key --persist

-

- # SMTP password

- apra-fleet secret --set smtp_password --persist

- ```

-

- Or via the MCP tool (the path an LLM agent uses):

-

- ```json

- { "name": "sendgrid_api_key", "prompt": "Enter your SendGrid API key", "persist": true }

- ```

-

- Secrets are encrypted in the fleet credential store. They never appear in

- workflow code, config files, or environment variables.

-

- ### Sending email from a workflow

-

- The workflow passes non-secret config inline and calls `send_email`. Load

- your config however you prefer (JSON file, hardcoded, etc.):

-

- ```javascript

- import { parseToolJson } from '@apralabs/apra-fleet-client';

- import { connectFleet } from '@apralabs/apra-fleet-client/server-resolution';

-

- const { fleetApi } = await connectFleet({ env: process.env });

-

- // fleetApi wrappers return the raw MCP tool result ({ content: [...] });

- // parseToolJson extracts the JSON payload.

- const result = parseToolJson(await fleetApi.sendEmail({

- provider: 'smtp',

- host: 'smtp.example.com',

- port: 587,

- user: 'notifications@example.com',

- from: 'noreply@example.com',

- to: 'team@example.com',

- subject: 'Sprint Report',

- body: 'All tasks completed.'

- }));

- console.log(`Sent: ${result.messageId}`);

- ```

-

- The SMTP password resolves from the credential store automatically. It never

- appears in the workflow. See `examples/workflows/email-notify/` for a

- complete runnable example and `docs/email-workflow-guide.md` for the full

- walkthrough.

-

- ### send_email Tool Reference

-

- | Parameter | Type | Required | Description |

- |---|---|---|---|

- | `provider` | `"sendgrid"` or `"smtp"` | no (default: `"sendgrid"`) | Email provider |

- | `from` | string | yes | Sender email address |

- | `host` | string | SMTP only | SMTP server hostname |

- | `port` | number | no (default: 587, or 465 when `secure` is true) | SMTP server port |

- | `user` | string | SMTP only | SMTP username |

- | `secure` | boolean | no (default: false) | Implicit TLS (port 465). When false, STARTTLS is required. |

- | `to` | string or string[] | yes | Recipient email address(es) |

- | `subject` | string | yes | Email subject line |

- | `body` | string | yes | Plain-text email body |

- | `html` | string | no | HTML email body |

- | `cc` | string[] | no | CC recipient addresses |

- | `bcc` | string[] | no | BCC recipient addresses |

- | `attachments` | attachment[] | no | File attachments (base64-encoded) |

-

- Each attachment: `filename` (string), `content` (string, base64), `contentType` (string, optional).

-

- Secrets are resolved from the credential store by name:

- - **SendGrid:** `sendgrid_api_key`

- - **SMTP:** `smtp_password`

-

- Returns: `{ ok: true, messageId }` on success, `{ ok: false, error }` on failure.

-

- ## The packages

-

- | Package | What it is |

- |---|---|

- | `apra-fleet` | The fleet platform: server, CLI, member management, credentials, workflows runtime |

- | `packages/apra-fleet-se` | The software-engineering vertical: fleet-sprint engine, agent contracts, integration suites |

- | `packages/apra-fleet-workflow` | Workflow authoring runtime: state, viewer, checkpointing |

- | `packages/fleet-api-contract` | Typed API contract shared by server and clients |

-

- ## Status and roadmap

-

- apra-fleet is under active development -- by its own fleet. Current focus:

- hardening autonomous sprint execution (the toughest workflow we know of),

- supervisor-orchestrated multi-sprint operation, and the workflow SDK for

- third-party verticals.

-

- ## Documentation

-

- | Topic | Link |

- |-------|------|

- | **fleet-sprint Getting Started Guide (start here, plain English)** | [Website](https://apra-labs.github.io/apra-fleet/fleet-sprint-getting-started.html) - [Markdown](docs/fleet-sprint-getting-started.md) - [PDF](docs/fleet-sprint-getting-started.pdf) |

- | Codebase wiki (architecture, internals, AI Q&A) | [DeepWiki](https://deepwiki.com/Apra-Labs/apra-fleet) |

- | Install, uninstall, the `--llm` flag | [docs/install.md](docs/install.md) |

- | Choosing a provider (roles, gotchas, mixing providers, OpenCode/local models) | [docs/provider-guide.md](docs/provider-guide.md) |

- | Transport, service mode, and supported interfaces | [docs/transport-and-service-mode.md](docs/transport-and-service-mode.md) |

- | Cost model (tiering, shell-over-prompts, measured token spend) | [docs/cost-model.md](docs/cost-model.md) |

- | The PM skill (doer-reviewer sprints, `/pm` commands) | [docs/pm-skill-overview.md](docs/pm-skill-overview.md) |

- | FAQ | [docs/FAQ.md](docs/FAQ.md) |

- | Troubleshooting | [docs/troubleshooting.md](docs/troubleshooting.md) |

- | Keeping Fleet updated (`apra-fleet update`) | [docs/features/update.md](docs/features/update.md) |

- | Live member activity (`apra-fleet watch`, `logging.previewChars`) | [docs/features/watch.md](docs/features/watch.md) |

- | Secure credentials and passwords | [docs/features/oob-auth.md](docs/features/oob-auth.md) |

- | Member category and tags | [docs/features/member-tags.md](docs/features/member-tags.md) |

- | Enabling SSH on a remote machine (if it does not have it yet) | [docs/ssh-setup.md](docs/ssh-setup.md) |

- | Git authentication | [docs/design-git-auth.md](docs/design-git-auth.md) |

- | Cloud compute | [docs/cloud-compute.md](docs/cloud-compute.md) |

- | Architecture | [docs/architecture.md](docs/architecture.md) |

- | Windows shell selection (probe order, gitbash/pwsh7/powershell5, shell vs os) | [docs/windows-shell-selection.md](docs/windows-shell-selection.md) |

- | Cross-shell command construction for member-bound commands | [docs/cross-shell-command-construction.md](docs/cross-shell-command-construction.md) |

- | Knowledge Layer (setup, usage, provider swap) | [docs/knowledge-layer.md](docs/knowledge-layer.md) |

- | Code intelligence provider abstraction | [docs/code-intelligence-providers.md](docs/code-intelligence-providers.md) |

- | Hub-spoke cloud migration plan (historical; see tier-3 ownership ADR) | [docs/hub-spoke-master-plan.md](docs/hub-spoke-master-plan.md) |

- | Tier-3 ownership decision (fleet-dashboard vs `src/hub-service/`) | [docs/adr-tier3-ownership.md](docs/adr-tier3-ownership.md) |

- | Shared hub/dashboard API contract package | [packages/fleet-api-contract/README.md](packages/fleet-api-contract/README.md) |

- | Workflow engine internals (`agent()`/`parallel()`/`pipeline()`, journal, budget, pause/resume) | [packages/apra-fleet-workflow/docs/apra-fleet-workflow-architecture.md](packages/apra-fleet-workflow/docs/apra-fleet-workflow-architecture.md) |

- | Cooperative workflow pause/resume (engine, viewer, supervisor, fleet-sprint) | [docs/features/workflow-pause-resume.md](docs/features/workflow-pause-resume.md) |

- | Supervisor dashboard live-refresh (`/state` + `/events` SSE, tab-activation refresh, in-memory scope expansion) | [docs/features/supervisor-dashboard-live-refresh.md](docs/features/supervisor-dashboard-live-refresh.md) |

- | Writing and running workflow scripts | [packages/apra-fleet-workflow/docs/workflow-guide.md](packages/apra-fleet-workflow/docs/workflow-guide.md) |

- | Authoring a SEA-embedded `apra-fleet workflow` (manifest, entry contract, launcher env vars) | [docs/authoring-workflows.md](docs/authoring-workflows.md) |

- | Workflow launcher fleet-server resolution order (HTTP singleton vs. stdio) | [docs/adr-workflow-server-resolution.md](docs/adr-workflow-server-resolution.md) |

- | Running fleet-sprint (full flag reference; identical for npm-install, standalone binary, and git-clone dev checkout) | [packages/apra-fleet-se/fleet-sprint/docs/README.md](packages/apra-fleet-se/fleet-sprint/docs/README.md) |

- | Auto-sprint overview (autonomous plan-develop-review-publish loop) | [packages/apra-fleet-se/docs/overview.md](packages/apra-fleet-se/docs/overview.md) |

- | Auto-sprint CLI reference | [packages/apra-fleet-se/docs/cli-reference.md](packages/apra-fleet-se/docs/cli-reference.md) |

- | Auto-sprint internals (cycle loop, stall detection, budget, topology) | [packages/apra-fleet-se/docs/architecture.md](packages/apra-fleet-se/docs/architecture.md) |

- | Auto-sprint agent role contracts | [packages/apra-fleet-se/docs/role-contracts.md](packages/apra-fleet-se/docs/role-contracts.md) |

- | fleet-supervisor skill (start/stop/restart/auto-start-on-boot, sprint launch via HTTP API) | [packages/apra-fleet-se/fleet-sprint/skills/fleet-supervisor/SKILL.md](packages/apra-fleet-se/fleet-sprint/skills/fleet-supervisor/SKILL.md) |

- | fleet-supervisor packaging self-containment audit | [packages/apra-fleet-se/docs/supervisor-selfcontainment-audit.md](packages/apra-fleet-se/docs/supervisor-selfcontainment-audit.md) |

- | MCP client SDK overview (transports, `ApraFleet` API) | [packages/apra-fleet-client/docs/overview.md](packages/apra-fleet-client/docs/overview.md) |

- | MCP client SDK API reference | [packages/apra-fleet-client/docs/api-reference.md](packages/apra-fleet-client/docs/api-reference.md) |

- | MCP client SDK getting started | [packages/apra-fleet-client/docs/getting-started.md](packages/apra-fleet-client/docs/getting-started.md) |

-

- ## Community

-

- - Questions and ideas: [GitHub Discussions](https://github.com/Apra-Labs/apra-fleet/discussions)

- - Releases: [GitHub Releases](https://github.com/Apra-Labs/apra-fleet/releases)

- - Issues: [GitHub Issues](https://github.com/Apra-Labs/apra-fleet/issues)

- - What is planned next: [ROADMAP.md](ROADMAP.md)

-

- If Apra Fleet helped you ship faster with better quality, please

- [star the repo](https://github.com/Apra-Labs/apra-fleet) -- it helps others

- find it.

-

- ## Development

-

- Build from source (also the path for Intel Macs):

-

- ```bash

- git clone https://github.com/Apra-Labs/apra-fleet && cd apra-fleet

- npm install && npm run build && npm test

- ```

-

- See [CONTRIBUTING.md](CONTRIBUTING.md) to contribute.

-

- ## License

-

- Apache 2.0 -- see [LICENSE](LICENSE).

-

- ---

-

- <div align="center">

-

- **Stop babysitting agents. Start operating fleets.**

-

- [Quick Start](#quick-start-5-minutes) - [GitHub Issues](https://github.com/Apra-Labs/apra-fleet/issues) - [Apra Labs](https://apralabs.com)

-

- </div>

- </doc>

-

- <doc title="Vocabulary" desc="Shared terminology -- member, task, skill, PM, fleet, doer/reviewer pattern.">

- <!-- llm-context: Defines the terminology used throughout apra-fleet -- member, fleet, PM, doer, reviewer, provider, session, etc. Consult this first when you encounter an unfamiliar fleet-specific term to avoid misinterpreting user requests. -->

- <!-- keywords: vocabulary, terminology, member, fleet, PM, doer, reviewer, provider, session, agent, orchestrator -->

- <!-- see-also: architecture.md (how these concepts relate), ../README.md (practical usage) -->

-

- # Fleet Vocabulary

-

- ## The Problem

-

- "Agent" is overloaded:

- 1. A fleet member (registered machine/folder that does work)

- 2. A background Claude process the PM spawns to coordinate

-

- "The agent is running" -- which one? This causes real confusion in logs, conversation, and status updates.

-

- ## Approach: Names Over Nouns

-

- Most of the time, use the **specific name** and drop the category word entirely:

-

- - "Sent to dev2" -- not "sent to member dev2"

- - "review1 passed PR #13" -- not "reviewer member passed"

- - "dev1 is on main" -- not "the dev1 member is on main"

-

- Names are unambiguous. Category nouns are noise when the name is present.

-

- ## When You Need the Category

-

- For generic references ("list all ___", "register a new ___"), use:

-

- | Term | Meaning |

- |------|---------|

- | **member** (or **worker**) | A registered fleet member. The thing that does the work. |

- | **subagent** | A background Claude process spawned by the PM. Ephemeral. |

- | **session** | A conversation thread on a member. Context persists within it. |

- | **fleet** | The collection of all registered members. |

- | **PM** | The Project Manager -- the master Claude instance that orchestrates everything. |

- | **provider** (or **LLM backend**) | The LLM CLI a member uses: `claude`, `codex`, or `copilot`. Each member has exactly one provider, set at registration and changeable via `update_member`. |

-

- ## Rules

-

- 1. **Prefer the name**: "dev2 rebased" not "the dev2 member rebased"

- 2. **"Subagent" is always "subagent"** -- never just "agent" when referring to a background Claude process

- 3. **"Agent" is banned in PM conversation** -- too ambiguous. Use the name or "member/worker" for fleet members, "subagent" for Claude processes.

- 4. **API keeps `agent_id`** -- backwards compat in code. User-facing language evolves separately.

- 5. **Provider is a property of a member**, not a conversation topic. Say "dev2 uses Codex" not "dev2 is a Codex agent". The member identity (the name) is what matters; the provider is just how it executes prompts.

-

- ## Example Fleet

-

- ```

- PM (orchestrator)

- |

- +-- dev1 (apra-focus, local, claude/standard)

- +-- dev2 (apra-focus-dev2, remote, codex/standard)

- +-- review1 (apra-focus-review, remote, claude/premium)

- +-- review2 (apra-focus-review2, remote, copilot)

- ```

-

- PM spawns **subagents** to interact with **members**. A subagent is ephemeral (dies after task). A member is persistent (registered, has sessions, has state).

-

- Members with different **providers** are interchangeable from the PM's perspective -- same tools, same dispatch pattern, different CLI underneath.

- </doc>

-

- <doc title="Architecture" desc="How the fleet hub, MCP server, and members interact at a system level.">

- <!-- llm-context: This document explains the internal architecture of apra-fleet -- the MCP server, member registry, SSH transport, session management, and how tools are dispatched. Read this when a user asks how fleet works under the hood, or when debugging connectivity or session issues. -->

- <!-- keywords: MCP server, member registry, SSH, transport, session, tool dispatch, child process, local member, remote member, architecture -->

- <!-- see-also: ../README.md (getting started), mcp-tools.md (tool details), vocabulary.md (terminology) -->

-

- # Architecture

-

- ## Why This Exists

-

- AI coding agents are powerful on a single machine. But real work spans many machines -- a dev server, a staging box, a GPU trainer, a production host. Today, if you want Claude Code working across all of them, you SSH in manually, run prompts one at a time, and copy files by hand. There's no single pane of glass.

-

- Apra Fleet gives one Claude instance the ability to orchestrate many. Register machines, push files, run prompts, monitor health -- all through natural language from your terminal. One master, many members.

-

- ## Conceptual Model

-

- The system has three layers of abstraction:

-

- **Fleet** -> **Members** -> **Sessions**

-

- A *fleet* is the collection of all registered machines. A *member* is one machine with a working directory -- the unit you talk to. A *session* is a conversation thread on a member -- Claude remembers context across prompts within a session, and you can reset it to start fresh.

-

- Members come in two flavors:

- - **Remote members** communicate over SSH. They can be any machine you can reach -- Linux VMs, macOS servers, Windows boxes.

- - **Local members** run on the same machine as the master, in a different folder. No SSH needed. Useful for isolating work into separate project directories without spinning up another machine.

-

- This distinction is hidden behind a **Strategy pattern**: every tool interacts with members through a uniform interface. The strategy implementation (remote via SSH, or local via child process) is selected at runtime based on member type. Tools never know or care which kind of member they're talking to.

-

- ## How It Fits Together

-

- ```

- +----------------------------------------------------+

- | Master Machine |

- | |

- | Claude Code CLI <--stdio--> Apra Fleet Server |

- | | |

- | +----------+----------+ |

- | | Member Strategy | |

- | | (uniform interface)| |

- | +--+-----------+-----+ |

- | | | |

- | Remote Strategy Local Strategy |

- | (ssh2 + sftp) (child_process + fs) |

- | | | |

- | SSH | local exec |

- +----------------------------------------------------+

- | |

- +------------+ +--> /other/project/

- v (same machine)

- +--------------+

- | Remote Member |

- | (any OS, |

- | any provider)|

- +--------------+

- ```

-

- The MCP server speaks **stdio** -- the standard transport for Claude Code MCP servers. Claude sends JSON-RPC tool calls, the server executes them, returns results. No HTTP, no ports to open.

-

- ## Layers

-

- The codebase follows a strict layering:

-

- ```

- index.ts <- MCP server entry point, tool registration

- tools/* <- one file per tool, each self-contained

- services/* <- core capabilities (strategy, registry, SSH, file transfer)

- providers/* <- LLM provider adapters (Claude, Antigravity, Codex, Copilot)

- os/* <- OS-specific command builders (Linux, macOS, Windows)

- utils/* <- stateless helpers (crypto, shell escaping)

- types.ts <- shared data structures

- ```

-

- Each layer only depends on the layers below it. Tools never import other tools. Services don't know about the MCP protocol.

-

- ## HTTP Transport & Interactive Sessions

-

- Alongside the stdio MCP transport, the server exposes a `StreamableHTTPServerTransport` on `POST/GET/DELETE /mcp` (`src/services/http-transport.ts`), bound to `127.0.0.1` only. This is what lets a member's own `apra-fleet` MCP server connect back interactively -- distinct from the subprocess/SSH-driven `execute_prompt` path, which stays subprocess-only. See `docs/hub-spoke-wire-protocol.md` for the full wire-level design.

-

- Each `/mcp` connection carries one of two identities:

-

- - **JWT-authenticated** -- a `Bearer` token (`src/services/jwt.ts`, HS256, signed with `~/.apra-fleet/fleet.key`) verified via the pluggable `TokenIssuer` (`src/services/token-issuer.ts`). The token's `workspace_id` claim is the hard security boundary (never `project_id`, which is an optional non-security label) -- Phase 1 uses a local dev-mode issuer (one machine == one implicit workspace); a hub-era issuer swaps in behind the same interface with no token-shape change.

- - **Unauthenticated URL-param fallback** -- a `?member=<id>` query param, trusted only because the server binds to loopback. Legacy friendly-name params are resolved to the member's UUID via the agent registry.

-

- A connected member is tracked in the in-memory `sessionRegistry` (`src/services/session-registry.ts`), keyed on the composite `(workspace_id, member_id)` -- every lookup is workspace-scoped, so a member connected under a different workspace is indistinguishable from "not connected" (existence is never leaked across the boundary). `send_message` (`src/tools/send-message.ts`) uses this registry to push a `notifications/claude/channel` MCP notification to a connected member's live session, flipping its status to `busy`.

-

- That flip is closed by `report_status` (`src/tools/report-status.ts`): a connected member's OWN session calls it (`online` or `idle`) to report it's done responding. There is no `member_id` parameter -- identity comes entirely from the live MCP session the call arrives on, resolved via `sessionRegistry.findBySessionId(extra.sessionId)` (the SDK populates `sessionId` on every tool call's `extra`). This is the tier-2-local status state machine `docs/hub-spoke-wire-protocol.md` section 4 reserves the `presence.member_status` envelope name for, once a hub relays it upward.

-

- Fleet events (`credential:stored`, `task:completed`, `member:status-changed`, `stall:detected`) broadcast only to sessions in the same workspace as the local orchestrator -- never across a workspace wall.

-

- `execute_prompt` (`src/tools/execute-prompt.ts`) itself is dual-path: for a member with NO live interactive session, it behaves exactly as before (subprocess/SSH, unchanged). For a member that IS interactively connected, it routes through the same channel instead of spawning anything -- `send_message` pushes the prompt and the caller awaits the member's `respond_to_message({reply_to, content})` call, correlated purely in-memory by `src/services/pending-responses.ts` (a `msgid` -> pending-promise map, timeout-bound by the same `timeout_s` the subprocess path uses). Mode selection is decided tier-2-locally against this machine's own `sessionRegistry` -- never from caller-side or (future) hub-side state -- so it is unaffected by whether `execute_prompt` is invoked directly or eventually relayed through a hub. This interactive mode is gated to Claude members only: `docs/interactive-injection-provider-survey.md` confirms it is POC-proven on Claude alone (the other five providers are confirmed unsupported or unconfirmed) -- a non-Claude member with a live session (e.g. from `registerMcpEndpoint`, which gives several providers basic MCP tool access) still falls through to the subprocess path.

-

- ## Provider Abstraction

-

- Fleet supports five LLM providers -- Claude Code, Google Antigravity CLI (agy), OpenAI Codex CLI, GitHub Copilot CLI, and OpenCode -- plus a sixth null option, `'none'`, for a plain command executor with no LLM at all (`src/providers/none.ts`). Members can mix providers within a single fleet.

-

- ### How It Works

-

- Each member has an optional `llmProvider` field (`'claude' | 'agy' | 'codex' | 'copilot' | 'opencode' | 'none'`). When absent, it defaults to `'claude'` for backwards compatibility. Every tool that interacts with the member's LLM CLI resolves the provider via `getProvider(agent.llmProvider)` and delegates CLI-specific concerns to the `ProviderAdapter` interface.

-

- A `'none'` member supports `execute_command` (already fully provider-agnostic, no changes needed) but never `execute_prompt` in either mode -- rejected immediately with a clear error rather than reaching `NoneProvider`'s methods, most of which throw by design (there is no CLI, no prompt, no model to build a command from). `register_member` skips CLI/auth verification entirely for these members, and status/detail views show `compute only` in place of a token count.

-

- ```

- +----------+ getProvider() +-----------------+

- | Tool | --------------------> | ProviderAdapter |

- | (generic)| | (per-provider) |

- +----------+ +--------+---------+

- | supplies:

- cliCommand()

- buildPromptCommand()

- parseResponse()

- classifyError()

- authEnvVar

- processName

- ...

- ```

-

- The `OsCommands` layer sits below this: it handles OS-specific shell wrapping (PATH prepend, PowerShell syntax, base64 decode) and delegates CLI-specific parts (binary name, flags, JSON format) to the provider.

-

- ### Provider Files

-

- ```

- src/providers/

- provider.ts - ProviderAdapter interface + shared types

- claude.ts - ClaudeProvider

- agy.ts - AgyProvider

- codex.ts - CodexProvider (NDJSON parser)

- copilot.ts - CopilotProvider

- opencode.ts - OpenCodeProvider (NDJSON parser, local/self-hosted models)

- index.ts - getProvider() singleton factory

- ```

-

- ### Mix-and-Match Fleet

-

- A fleet can have members on different providers simultaneously. The PM dispatches work to members by name -- it doesn't need to know which LLM backend each member uses. The fleet server resolves the correct CLI commands per member at runtime.

-

- ```

- PM (orchestrator, Claude)

- |

- +-- dev1 (claude, remote)

- +-- dev2 (agy, remote)

- +-- dev3 (codex, local)

- +-- dev4 (opencode, local) <- self-hosted model via Ollama

- +-- review (copilot, remote)

- ```

-

- All five members use the same `execute_prompt` tool call. The tool builds provider-correct CLI commands for each.

-

- ### Key Differences Across Providers

-

- - **`max_turns`** - Claude-only. Ignored for Antigravity, Codex, and Copilot.

- - **OAuth credential copy** - Claude-only. Non-Claude providers require an API key (`provision_llm_auth` with `api_key`).

- - **JSON output format** - Codex emits NDJSON (one event per line). All others emit a single JSON object. Handled transparently by `provider.parseResponse()`.

- - **Session resume** - Claude and Antigravity support resuming specific session IDs. Codex and Copilot resume the most recent local session. OpenCode supports session resume via `--session <id>` or `--continue`.

- - **OpenCode** - uses any OpenAI-compatible endpoint (Ollama, vLLM). The user provisions the endpoint; Fleet installs the CLI and agents. Model tiers are set per member at registration via `model_tiers` (since models vary by deployment). Agent files are transformed from Claude format to OpenCode format at install time (tools allowlist -> permission map).

- - **Antigravity (agy) response capture** - `AgyProvider.parseResponse()` extracts both the reply text and, when the CLI's transcript exposes one, a `conversation_id` that is surfaced as the response's `sessionId` -- mirroring Claude's session-id capture instead of returning an empty session id for a provider that does expose one.

-

- ### Terminal-Signal and Dead-Session Detection Invariants

-

- These invariants apply across all providers that can terminate a dispatch without a clean `execute_prompt` return, and are enforced independently of each other as defense-in-depth -- a failure in one layer must not silently convert into a wasted multi-thousand-second wait for the hard dispatch ceiling:

-

- - **Max-turns detection is channel-agnostic.** A CLI can signal "hit the turn limit" through more than one channel in the same transcript stream -- a `type:result` event's `terminal_reason` field, that same event's `subtype` (`error_max_turns`), a distinct standalone `max_turns_reached` transcript event (which can arrive before, or instead of, any result event, e.g. when a hard-timeout kill truncates the stream first), or a plain-text fallback when no result event terminates the stream at all. `src/providers/claude.ts` normalizes every one of these channels to the same `terminalReason: 'max_turns'` on the parsed response, and `src/providers/provider.ts` exposes a single shared classifier (checking the normalized `terminalReason` OR the raw `error_max_turns` subtype) that every call site uses -- there is exactly one place that decides "was this a max-turns termination," never a per-call-site heuristic. This does not add any new MCP reason-enum value; `max_turns_exhausted` already existed and is now reliably reached from every transcript shape that means it.

- - **Stall detection is a superset backstop, not a replacement.** The transcript-mtime cross-check described in `docs/stall-detector-resilience.md` (section 7) exists specifically so that a terminal-signal-detection gap (like the max-turns case above, or any future one) cannot regress into the old failure mode of a dead session sitting unkilled until the hard ceiling -- a frozen transcript is killed within the configured inactivity window regardless of whether content parsing understood why it went quiet.

- - **A busy-lock rejection is verified against process liveness before being honored.** `execute_prompt`'s in-flight lock (`inFlightAgents`, `src/tools/execute-prompt.ts`) can outlive the process it was guarding (a child reaped without its exit handler firing, or an interactive session's underlying CLI process dying after registration). Before returning a `busy` rejection, the dispatch path probes the locked session's pid for liveness -- locally via a direct signal-0 check, remotely via a fresh independent liveness round trip (never the possibly-wedged channel that produced the stale lock), and via the session registry's last-known pid for interactive sessions -- and self-heals (releases the lock, warns, and proceeds with the new dispatch) only on a **definitive** dead-pid reading. Ambiguity (no pid captured at all, e.g. a dispatch that hasn't reached its pid-capture step yet) is always treated as still-busy, never as staleness evidence, so a self-heal attempt can never race a dispatch that is still starting up. This keeps `fleet_status`'s independently-computed busy/idle view and the dispatch gate's busy/idle view from ever disagreeing for longer than one dead-lock detection cycle.

- - **A confirmed stall cancels the in-flight dispatch, not just the remote process.** Killing the remote pid a stalled dispatch was running is necessary but not sufficient -- the MCP `tools/call` that dispatched it can still be sitting on a pending `execCommand()` promise with no way to reject it, so the client falls back to waiting out its own hard deadline even though the server-side work died minutes earlier. The stall path now threads an `AbortController` through `onStall` that is wired into the same strategy-level `execCommand()` abort signal remote strategies already accept, so a confirmed stall settles the pending call immediately with a typed `stalled` dispatch error instead of silently degrading into a multi-thousand-second client-side timeout wait.

- - **The client's dispatch deadline must cover every server-side retry the request can trigger, not just the first attempt.** When the server's own inactivity-timeout handling retries a dispatch once with a fresh session (same `timeout_s`/`max_total_s`), the total server-side worst case is a multiple of the value the client's deadline was derived from. A client deadline sized for only the first attempt can fire before the server's own retry-and-report-cleanly path gets a chance to run, which surfaces a raw transport timeout to the caller instead of a clean typed error. The fix is a single shared retry-budget helper that both sides size their deadlines from, so "the client waits at least as long as the server's own worst-case retry sequence" is a property of one function, not something every dispatch call site has to remember to compute correctly.

-

- See `docs/provider-matrix.md` for the full comparison table.

-

- ### Multi-member topology (fleet-sprint)

-

- The `apra-fleet-se` fleet-sprint runner (`packages/apra-fleet-se/fleet-sprint/runner.js`) dispatches roles across configured members: the orchestrator issues every `bd` command and the git push/PR, doers round-robin across the doer pool, and the reviewer runs from the reviewer pool.

-

- Two topology modes are supported, selected explicitly when a sprint starts:

-

- - **`legacy` mode** -- no cross-member sync layer. Every `bd` command the orchestrator issues runs against the orchestrator member's beads DB, while each doer's own `bd close` runs against **its** member's DB; the sprint git branch is only meaningful if all members operate on the same working state. This coheres in exactly two topologies: **single-member** (one member does everything) or a **verified shared-workspace fleet** (every configured member resolves to the same checkout/beads DB). Independent per-member checkouts are **not supported** in this mode -- non-orchestrator members would silently work against a beads DB and git branch the orchestrator never sees.

- - **`synced` mode** -- an orchestrator-bracketed git+Dolt sync layer wraps every dispatch that reads or writes git-tracked or beads state, reconciling each member's state explicitly instead of assuming it is already shared. This is what makes genuinely independent per-member checkouts safe. The sync layer includes a scripted-first escalation ladder for both git and Dolt conflicts (mechanical detection and resolution first, an agent dispatched with an explicit conflict runbook only as a documented last resort), plus supervisor-owned global coordination (a Dolt push mutex and a child-id allocator) so concurrent cross-member Dolt writes never silently corrupt each other's beads clone.

-

- Guards enforcing whichever mode is selected:

-

- 1. **Branch-ensure everywhere** (both modes): before the first doer round, the runner git-ensures the sprint branch (`fetch` + `checkout -B`) on **every** member in the union of the orchestrator/doer/reviewer pools -- not just the orchestrator -- and non-destructively re-checks-out the branch on each member at the start of later cycles (it never resets to base once work is committed).

- 2. **Topology precondition**: `bin/cli.mjs` calls `checkMemberTopology()` before starting a multi-member sprint. In `legacy` mode it compares an identity signal (`git rev-parse HEAD`) across the configured members and **refuses to start with a clear error** if they disagree (or if a member's signal can't be obtained); in `synced` mode HEADs are allowed to differ, but every member must share the same git remote origin and pass a live beads-Dolt-pull probe. Single-member sprints trivially pass either check.

-

- See `packages/apra-fleet-se/docs/architecture.md` for the full internals of both modes, including the escalation ladders and the always-on supervisor service that launches and tracks sprints.

-

- ## Workflow Subsystem (SEA-embedded workflow runner)

-

- `apra-fleet workflow <name>` runs a self-contained script (an ESM entry point

- under `workflows/<name>/`) against a live fleet connection, from inside the

- single-executable-application (SEA) binary -- no separate Node install and no

- unpacking to a temp directory required. `fleet-sprint` is itself shipped as one

- of these built-in workflows rather than as a bespoke subcommand, so the

- launcher, the packaging, and the docs only need to solve this problem once.

-

- **Two always-separate processes.** The workflow launcher (`apra-fleet

- workflow`, `fleet-sprint`) and the `apra-fleet` MCP server are never merged

- into one process. This is a hard boundary, not an optimization detail --

- see `docs/adr-workflow-server-resolution.md` for the ADR and its explicit

- scope guard against reopening it.

-

- **Server connection resolution is a single shared helper**, not duplicated

- per consumer. `@apralabs/apra-fleet-client/server-resolution`

- (`packages/apra-fleet-client/src/client/server-resolution.mjs`) implements

- one resolution order used identically by `src/cli/workflow.ts` and

- `packages/apra-fleet-se/bin/cli.mjs`:

-

- 1. `APRA_FLEET_TRANSPORT` forced override (`http` fails loud with no

- singleton rather than silently falling back to a private server; `stdio`

- or a set `APRA_FLEET_SERVER_CMD`/`_BIN` goes straight to self-spawn).

- 2. Probe for a healthy HTTP singleton (`checkRunningInstance()` -- the same

- pid + `/health` check the installed service already uses for

- startup-dedup) and attach with zero spawned processes. This is the

- default, steady-state path.

- 3. Fall back to stdio self-spawn (the four-tier command resolution

- `resolveFleetServerCommand()` already had) only when no healthy

- singleton is found.

-

- The rationale for one shared helper over two copies: a launcher that merely

- mirrored the old stdio-only `resolveFleetServerCommand()` would always

- self-spawn a private server even when a healthy HTTP singleton already

- existed, doubling running servers and splitting state. Duplicating the

- resolution order in two languages (TS launcher, MJS fleet-sprint) was

- rejected because it guarantees drift between the two copies over time; see

- the ADR for the full tradeoff writeup.

-

- **Workflow entry contract:** a workflow is a directory under `workflows/`

- containing a `workflow.json` (name, entry, description) and an entry file

- that is either self-executing ESM or exports `main(args)` / `run(args)` /

- a default export. The launcher sets two env vars for every workflow run --

- `APRA_FLEET_SERVER_BIN` and `APRA_FLEET_SE_SCHEMAS_DIR` -- and never

- clobbers a value the caller already set. See `docs/authoring-workflows.md`

- for the full contract, including the raw-node escape hatch and how a

- workflow should import the shared engine/client packages.

-

- **Entry-path escape prevention is security-critical, not incidental:** the

- launcher resolves a workflow's `entry` against its own directory and

- rejects any resolution that escapes it (checked via `path.relative` plus

- `..`/absolute-path detection) before ever executing the file. A workflow

- manifest is not a trusted-by-construction input.

-

- **Packaging:** the workflow runtime, agent schemas, and built-in workflows

- (including fleet-sprint) are embedded as SEA assets by

- `scripts/gen-sea-config.mjs`, proven viable by an earlier spike that

- dynamic `import()` of on-disk ESM works from inside a SEA main script on

- all three target OSes.

-

- **Install/uninstall/update lifecycle (Phases 1-3, closed this sprint,

- `apra-fleet-7pm`):** `install.ts`'s workflow-install step and its extraction

- logic (extract-to-temp-then-rename, with Windows EBUSY retry/backoff) were

- factored out into `src/cli/workflow-assets.ts` so more than one caller can

- share the exact same code path instead of re-implementing it:

-

- - `apra-fleet install` (fresh install) calls `extractWorkflowSubsystemAssets()`

- to lay down `~/.apra-fleet/{node_modules,schemas,workflows/<builtin>}`.

- - `src/cli/workflow.ts`'s launcher self-heals: if a `workflow <name>`

- invocation finds the on-disk payload missing or incomplete

- (`hasWorkflowSubsystemAssets()` is false), it re-extracts from the same

- embedded SEA assets before running, rather than failing with an opaque

- module-not-found error.

- - `apra-fleet uninstall --skill workflows` removes the shared runtime/schema

- dirs plus only the built-in workflow subdirectories recorded in

- `workflows/.installed.json`'s `builtin` array (falling back to the static

- `BUILTIN_WORKFLOW_NAMES` list if that manifest is missing, so a

- partially-installed tree still cleans up). User-authored workflow

- directories are left in place; `workflows/` itself is only removed if

- nothing user-authored remains, and the command prints which user

- workflows it kept.

- - `apra-fleet update` reads back the previously-persisted `--workflows` mode

- and threads it into the re-invoked install, so an update refreshes

- built-in workflow assets to the new version while preserving any

- user-authored workflows already on disk.

-

- **CI coverage (Phase 4, closed this sprint):** `build:binary` smoke tests

- exercise the packaged SEA binary's `workflow` subcommand and the

- fleet-sprint-as-built-in-workflow path end-to-end (not just the source

- `.ts`/`.mjs` files), and a regression-guard test suite

- (`tests/regression-command-surface.test.ts`) pins the full existing CLI

- command surface (`install --help`, `uninstall --dry-run`, `--version`,

- stdio handshake) against golden fixtures so future workflow-subsystem work

- can't silently change unrelated command output.

-

- The workflow subsystem is now installable, self-healing, uninstallable, and

- update-safe end-to-end; remaining gaps for this feature are tracked as

- individual open issues under `apra-fleet-7pm`, not as a phase-level

- placeholder.

-

- **Cooperative pause/resume** is a generic engine primitive available to any

- workflow run (`requestPause()`/`requestResume()`/`setPauseGuard()`), separate

- from cancellation (`requestStop()`): a pause is deferred until in-flight work

- drains to zero and an optional caller-supplied guard confirms the run is at a

- state boundary it considers clean, rather than landing wherever the request

- happened to arrive. The per-run viewer and the multi-sprint supervisor

- dashboard both surface it, and fleet-sprint is the first workflow to attach

- real domain behavior to it (releasing/re-acquiring member reservations and

- resyncing git/beads state across a pause). See

- [docs/features/workflow-pause-resume.md](features/workflow-pause-resume.md)

- for the full design and

- [packages/apra-fleet-workflow/docs/apra-fleet-workflow-architecture.md](../packages/apra-fleet-workflow/docs/apra-fleet-workflow-architecture.md)

- section 4.7 for the engine-level contract.

-

- **The multi-sprint supervisor dashboard's live-refresh** deliberately reuses

- the per-run viewer's own incremental-refresh architecture (a lean JSON state

- endpoint plus a change-signal stream driving a debounced client poll) instead

- of a second, divergent mechanism -- see

- [docs/features/supervisor-dashboard-live-refresh.md](features/supervisor-dashboard-live-refresh.md)

- for the full design, including the in-memory scope-expansion fix that removed

- a per-graph-node subprocess spawn from every dashboard render.

-

- **Dev-mode manifest bundling is a transitive-dependency, not a top-level-only,

- contract.** The dev-mode install path bundles `@apralabs/apra-fleet-client`'s

- own package tree so the workflow runtime can load it without a real npm

- install. Because that client package can itself gain a runtime dependency

- (e.g. for a new transport), the manifest builder must walk and bundle the

- client's own `dependencies`, not just the client package's own files -- a

- manifest that copies only the client's top-level directory silently ships a

- client that throws a module-not-found error the moment it needs a dependency

- it never had bundled alongside it. Any future dependency added to

- `apra-fleet-client` must be re-verified against this bundling path, not just

- against the production/packaged-binary install path, since the two paths

- build the manifest differently.

-

- ## PM Skill

-

- The PM skill and its role agent definitions (planner, plan-reviewer, doer, reviewer, deployer, integ-test-runner, ci-watcher, harvester), plus their shared `agents/schemas/` and `agents/_shared/` assets, live in this monorepo at `packages/apra-fleet-se/apra-pm/`. At build time, `scripts/dist-pm.mjs` copies the skill and agent files into `dist/` (and `scripts/gen-sea-config.mjs` embeds them as SEA assets for the standalone binary), so all three install paths -- dev-mode, npm, and SEA binary -- carry the same set. At install time, agents are written to the provider's agents directory (e.g. `~/.claude/agents/`); `uninstall` removes that directory in the same pass. For OpenCode members, agent frontmatter is transformed from Claude format to OpenCode format during installation. Claude installs additionally get the `auto-sprint-args` skill (the args contract for the `/auto-sprint` workflow), written to `~/.claude/skills/auto-sprint-args`.

-

- Remote fleet members do not share the operator's home directory, so `register_member`, `update_member`, and (starting with the first dispatch after an orchestrator upgrade) `execute_prompt` each independently hash-diff the canonical agent set against the remote box and push anything missing or stale. See [docs/mcp-tools.md](mcp-tools.md) for per-tool behavior details.

-

- ## Key Design Decisions

-

- ### Strategy Pattern for Member Types

-

- Rather than scattering `if (agent.agentType === 'local')` checks across every tool, the local/remote distinction lives in a single place: the strategy factory. Tools call `getStrategy(agent).execCommand(...)` and get back the same result shape regardless of how it was executed. Adding a third member type (e.g., Docker containers, cloud VMs with API-based access) means writing one new strategy class -- no tool changes.

-

- ### Passwords Encrypted at Rest

-

- SSH passwords are encrypted with AES-256-GCM before being written to the registry file. The encryption key is derived from the machine's identity (hostname + OS username), so the registry file is meaningless if copied to another machine. This isn't meant to stop a determined attacker with root access -- it prevents accidental plaintext exposure in backups, screenshots, or config file shares.

-

- ### Connection Pooling with Idle Timeout

-

- SSH connections are expensive to establish (TCP + key exchange + auth). The server pools them in memory and reuses connections across tool calls, with a 5-minute idle timeout that auto-closes unused connections. Timers are `unref()`'d so they don't prevent Node from exiting.

-

- The idle timer is reset by normal pool activity (`execCommand`/`getConnection`), but a stall-detector entry that is still `provisional` (created before it has a log file to poll) does not generate that activity itself -- it only benefits from the stall poller's own incidental probes. The pool's cleanup path therefore never tears down an entry purely on the idle timer firing: it re-checks for an active channel immediately before ending the connection, and only actually closes it when genuinely idle. This closes off a class of latent bug where a long-running dispatch with a provisional-shaped stall entry could otherwise have a live channel reaped out from under it. Separately, pooled SSH connections have keepalive configured (interval + a bounded probe count) so a silently dropped network connection is detected and torn down rather than sitting as a phantom pool entry indefinitely.

-

- ### Base64 Prompt Encoding

-

- Prompts sent to remote members are base64-encoded before being passed through SSH. This sidesteps the shell escaping nightmare of nested quoting across SSH -> bash -> claude CLI, across different operating systems. The remote member decodes before passing to Claude.

-

- ### Session Persistence

-

- Each member stores an optional `sessionId` -- a Claude conversation thread ID. When `resume=true` (the default), subsequent prompts continue the same conversation, so the remote Claude has full context of prior exchanges. Resetting a session is an explicit action, not an accident.

-

- ### File-Based Registry

-

- All fleet state lives in `~/.apra-fleet/data/registry.json` -- a single JSON file in the user's home directory. It's deliberately not in the project directory (won't be git-committed accidentally) and not in a database (no server to run, no migrations). For a fleet of dozens of members, JSON is more than sufficient.

-

- ### Duplicate Folder Prevention

-

- Two members cannot share the same working directory on the same device. For remote members, "same device" means same SSH host. For local members, "same device" is always the master machine. This is enforced during registration and updates. It prevents two members from stomping on each other's files.

-

- ## Tools

-

- The tools break into natural groups in **[mcp-tools.md](mcp-tools.md)**:

-

- **[Lifecycle](mcp-tools.md#1-lifecycle-tools)** -- `register_member`, `list_members`, `update_member`, `remove_member`, `shutdown_server`

- Manage the fleet roster and server lifecycle. Registration validates connectivity, detects the OS, and checks that Claude CLI is available. Removal includes best-effort cleanup of auth credentials on the member.

-

- **[Work](mcp-tools.md#2-work-tools)** -- `send_files`, `execute_prompt`, `execute_command`, `reset_session`

- The core workflow. Push files to a member, run prompts against it, run shell commands directly, manage conversation sessions.

-

- **[Infrastructure](mcp-tools.md#3-infrastructure-tools)** -- `provision_llm_auth`, `setup_ssh_key`, `update_llm_cli`

- One-time setup and maintenance. Provision auth (copy OAuth credentials or deploy API key for any provider), migrate from password to key auth, update the LLM CLI on members.

-

- **[Observability](mcp-tools.md#4-observability-tools)** -- `fleet_status`, `member_detail`

- Two-layer monitoring. `fleet_status` gives a quick summary table across all members with fleet-aware busy detection (distinguishes between Claude processes serving this member vs unrelated Claude activity). `member_detail` drills into one member with connectivity, CLI version, session state, and system resource metrics.

-

- ## Cross-Platform Support

-

- Members can run Windows, macOS, or Linux. The `platform.ts` utility generates the right shell commands for each OS -- different commands for checking processes, reading memory, setting environment variables. The OS is auto-detected during registration (`uname -s` on Unix, `cmd /c ver` on Windows) and stored in the member record so subsequent tool calls don't need to re-detect.

- </doc>

-

- <doc title="Install" desc="Installation, uninstallation, and the --llm/--skill flags.">

- # Install, uninstall, and update

-

- This page covers installing Apra Fleet, what the installer writes, controlling

- which skills are installed, uninstalling, and self-updating.

-

- ## Requirements

-

- - An AI coding agent CLI on the machine where you run Fleet - Claude Code,

- Antigravity (agy), Codex, or Copilot.

- - SSH access to any remote machines you want to register as members. The local

- machine needs nothing extra; remote members need only an SSH server.

-

- ## Quick install

-

- Installation is the default action -- just run the binary with no arguments (or double-click it

- on Windows).

-

- **macOS (Apple Silicon)**

- ```bash

- curl -fsSL https://github.com/Apra-Labs/apra-fleet/releases/latest/download/apra-fleet-installer-darwin-arm64 -o apra-fleet-installer && chmod +x apra-fleet-installer && ./apra-fleet-installer

- ```

-

- **Linux (x64)**

- ```bash

- curl -fsSL https://github.com/Apra-Labs/apra-fleet/releases/latest/download/apra-fleet-installer-linux-x64 -o apra-fleet-installer && chmod +x apra-fleet-installer && ./apra-fleet-installer

- ```

-

- **Windows (x64)** -- download `apra-fleet-installer-win-x64.exe` and double-click it, or run in PowerShell:

- ```powershell

- Invoke-WebRequest -Uri https://github.com/Apra-Labs/apra-fleet/releases/latest/download/apra-fleet-installer-win-x64.exe -OutFile apra-fleet-installer.exe; .\apra-fleet-installer.exe

- ```

-

- Intel Macs: there is no prebuilt `darwin-x64` binary -- build from source (see

- the Development section of the [README](../README.md)).

-

- ## Manual install

-

- Download the installer for your platform from

- [GitHub Releases](https://github.com/Apra-Labs/apra-fleet/releases):

-

- - `apra-fleet-installer-linux-x64` -- Linux (x86_64)

- - `apra-fleet-installer-darwin-arm64` -- macOS (Apple Silicon)

- - `apra-fleet-installer-win-x64.exe` -- Windows

-

- Double-click the downloaded file, or run it from the terminal. Installation is the default action:

-

- ```bash

- # macOS (Apple Silicon) -- no subcommand needed; installation is the default

- chmod +x apra-fleet-installer-darwin-arm64 && ./apra-fleet-installer-darwin-arm64

-

- # Linux (x64)

- chmod +x apra-fleet-installer-linux-x64 && ./apra-fleet-installer-linux-x64

- ```

-

- ```powershell

- # Windows

- .\apra-fleet-installer-win-x64.exe

- ```

-

- > The `install` subcommand is still accepted and does the same thing:

- > `./apra-fleet-installer install` works exactly as before.

-

- ## What `install` writes

-

- | Path | What it is |

- |------|-----------|

- | `~/.apra-fleet/bin/apra-fleet[.exe]` | The fleet binary |

- | `~/.apra-fleet/hooks/` | Shell hooks (statusline, etc.) |

- | `~/.apra-fleet/scripts/` | Helper scripts |

- | `~/.apra-fleet/node_modules/` | Shared on-disk workflow runtime (`@apralabs/apra-fleet-workflow`, `@apralabs/apra-fleet-client`, vendored `ajv` + deps) that `apra-fleet workflow <name>` and any user-authored workflow resolve bare specifiers against -- see `docs/authoring-workflows.md` |

- | `~/.apra-fleet/schemas/` | Installed agent role verdict/input JSON schemas (17 files); the `APRA_FLEET_SE_SCHEMAS_DIR` default the workflow launcher sets |

- | `~/.apra-fleet/workflows/` | Installed workflows (`.installed.json` + one directory per workflow, built-in or user-authored); run with `apra-fleet workflow <name> [args...]` -- see `docs/authoring-workflows.md` |

- | `~/.claude/skills/fleet/` | Fleet skill (MCP tool docs for Claude) |

- | `~/.claude/skills/pm/` | PM orchestration skill |

- | `~/.claude/skills/pm/cost.js` | Auto-generated CJS module with sprint cost functions (all providers with PM) |

- | `~/.claude/workflows/auto-sprint.js` | Full auto-sprint workflow (Claude only) |

- | `~/.claude/skills/auto-sprint-args/` | Args contract for the `/auto-sprint` workflow (Claude only) |

- | `~/.claude/skills/fleet-sprint-cli/` | How to launch `apra-fleet workflow fleet-sprint` -- flag contract, preconditions, detached launch (all providers) |

- | `~/.claude/agents/` | PM role-agent files (planner, doer, reviewer, etc.), plus `schemas/` and `_shared/` -- written whenever PM is installed and the provider has an agents directory (not codex/copilot) |

-

- For other providers, these are written to that provider's skill/config directories. For example, for Antigravity (`agy`), settings are written to `~/.gemini/antigravity-cli/settings.json`, and hooks / MCP configs are merged into `~/.gemini/config/hooks.json` and `~/.gemini/config/mcp_config.json`.

-

- This local install only covers the machine you run it on. Remote fleet members get their own copy of the PM agent files independently -- `register_member` and `update_member` push them on first contact, and `execute_prompt` re-checks and re-provisions any missing or stale files on first dispatch to that member each server run (so an existing member picks up new agent files after you upgrade Fleet, without needing to be re-registered). Local members are unaffected -- they share the operator's home directory above.

-

- The install also registers the MCP server (`claude mcp add apra-fleet`) and

- configures a status bar icon showing fleet member activity.

-

- ### Two sprint entry points -- do not confuse them

-

- `apra-fleet` ships **two separate, independently maintained** sprint

- implementations:

-

- | | `auto-sprint` (Claude Code workflow) | `fleet-sprint` (apra-fleet CLI workflow) |

- |---|---|---|

- | Written by | `install` (table above) | The root `@apralabs/apra-fleet` npm package's `bin.fleet-sprint`, resolving to `dist/fleet-sprint.mjs` |

- | Providers | Claude Code only | Any provider a fleet member is registered with (Claude, Codex, Copilot, Antigravity/agy) |

- | Source package | `packages/apra-fleet-se/apra-pm/.claude/workflows/auto-sprint.js` | `packages/apra-fleet-se` (esbuild-bundled, apra-fleet-3ns.2) |

- | Model selection | Literal Claude model names | Fleet's `cheap`/`standard`/`premium` tier keywords, per-member |

- | How you run it | `/auto-sprint <bead-ids>` inside a Claude Code session (the Workflow tool) | `apra-fleet workflow fleet-sprint --issue ... --members ... --branch ... --base ...` -- the normal form. The same bundle is also exposed directly as a `fleet-sprint` bin (`npx fleet-sprint ...`, or bare if `@apralabs/apra-fleet` is installed with `-g`). See `packages/apra-fleet-se/docs/cli-reference.md` |

-

- If you installed `apra-fleet` via `npm install -g @apralabs/apra-fleet`

- (or `npx @apralabs/apra-fleet`), the `fleet-sprint` bin is available

- immediately alongside `apra-fleet` -- no separate install step. It requires

- the `apra-fleet` MCP server to be reachable (it spawns/connects to it over

- stdio the same way `apra-fleet` itself does); see

- `packages/apra-fleet-se/docs/cli-reference.md` for its server- and

- schema-resolution order across dev/bundled/standalone layouts.

-

- ### The `apra-fleet workflow <name>` subcommand

-

- `install` also populates `~/.apra-fleet/node_modules/`, `~/.apra-fleet/schemas/`,

- and `~/.apra-fleet/workflows/` (see the directory table above) so that

- `apra-fleet workflow <name> [args...]` -- the SEA-binary workflow runner --

- can run built-in workflows (`fleet-sprint`, `hello-world`) or any

- user-authored workflow with zero system Node required. See

- `docs/authoring-workflows.md` for the full authoring contract.

-

- The workflow launcher and the `apra-fleet` MCP server it talks to are

- always separate processes. Set `APRA_FLEET_TRANSPORT=http` (the default) or

- `APRA_FLEET_TRANSPORT=stdio` to control how the launcher reaches that

- server: `http` (default) attaches to the already-running installed-service

- singleton at `http://localhost:${APRA_FLEET_PORT:-7523}/mcp` and spawns

- nothing; `stdio` self-spawns a private server the same way the `fleet-sprint`

- bin does. See `docs/adr-workflow-server-resolution.md` for the full

- resolution order (this same order also governs where role schemas resolve

- from in the installed-binary case: `APRA_FLEET_SE_SCHEMAS_DIR`, set by the

- launcher to `~/.apra-fleet/schemas`, is now tier 1 of the schema resolution

- described in `packages/apra-fleet-se/docs/cli-reference.md`).

-

- **What `install` does NOT do:**

-

- - No system-level changes -- no `/usr/local`, no PATH modification, no

- admin/sudo required.

- - No network calls beyond `claude mcp add` -- the binary stays local.

- - No background services or daemons -- the fleet server starts on demand when

- your AI coding agent connects.

-

- ## The `--skill` flag

-

- By default, `install` writes both the fleet and PM skills. Use `--skill` to

- control exactly which skills are installed:

-

- | Flag | Skills installed |

- |------|------------------|

- | `install` (no flag) | fleet + pm (default) |

- | `install --skill all` | fleet + pm |

- | `install --skill fleet` | fleet only |

- | `install --skill pm` | fleet + pm (pm depends on fleet) |

- | `install --skill none` | neither |

- | `install --no-skill` | neither (same as `--skill none`) |

-

- ## Install for other providers (Antigravity, Codex, Copilot)

-

- By default, `install` configures Apra Fleet for **Claude Code**. Use the `--llm`

- flag to install for a different provider instead:

-

- ```bash

- apra-fleet --llm agy # Google Antigravity CLI

- apra-fleet --llm codex # OpenAI Codex CLI

- apra-fleet --llm copilot # GitHub Copilot CLI

- apra-fleet --llm claude # Claude Code (the default)

- ```

-

- The `install` subcommand is also accepted and does the same thing:

- `apra-fleet install --llm agy` works exactly as before.

-

- `--llm` decides which provider's configuration the installer writes to. The MCP

- server registration, hooks, statusline, permissions, and skills all go into that

- provider's config directory -- for example `~/.gemini/antigravity-cli/` for

- Antigravity -- instead of `~/.claude/`. To support more than one provider on the

- same machine, run `install` once per provider.

-

- `--llm` combines with `--skill`, e.g. `apra-fleet install --llm agy --skill

- pm`. Supported values: `claude` (default), `agy`, `codex`, `copilot`.

-

- After a non-Claude install, load the server by restarting that provider's CLI --

- only Claude Code uses `/mcp`.

-

- ### Agy note

-

- `apra-fleet install --llm agy` configures Fleet for the Google Antigravity CLI.

- Agy uses Google OAuth by default -- a browser-based login flow is required per

- machine, so `provision_llm_auth` does **not** work for remote agy members today.

-

- For headless or remote members, set `ANTIGRAVITY_API_KEY` (obtain from

- [Google AI Studio](https://aistudio.google.com)) in the environment before

- invoking fleet commands. The agy CLI checks env vars before falling back to

- OAuth.

-

- ## Uninstall

-

- The built-in uninstall command surgically removes MCP registration,

- permissions, hooks, status line, skill directories, and PM agent files

- (`~/.claude/agents/`, or the equivalent provider directory) without touching

- your other settings:

-

- ```bash

- apra-fleet uninstall

- ```

-

- | Flag | Effect |

- |------|--------|

- | `--dry-run` | Preview what would be removed, without modifying anything |

- | `--force` | Automatically stop the running fleet server before uninstalling |

- | `--yes` | Skip the confirmation prompt |

- | `--llm <provider>` | Remove only a specific provider (`claude`, `agy`, `codex`, `copilot`) |

- | `--skill fleet\|pm\|workflows\|all` | Remove only the specified skill directories (default: `all`) |

-

- `--skill workflows` removes the shared workflow runtime and schemas

- (`~/.apra-fleet/node_modules/`, `~/.apra-fleet/schemas/`) plus only the

- built-in workflow subdirectories under `~/.apra-fleet/workflows/` (read from

- `workflows/.installed.json`'s `builtin` list, falling back to the static

- built-in name list if that manifest is missing). Any user-authored

- `workflows/<name>/` directories are left in place, and the command reports

- which ones it kept; the `workflows/` root itself is only removed if nothing

- user-authored remains in it.

-

- Examples:

-

- ```bash

- # Preview the full uninstall

- apra-fleet uninstall --dry-run

-

- # Full uninstall, stop server automatically

- apra-fleet uninstall --force --yes

-

- # Remove only PM skills across all providers

- apra-fleet uninstall --skill pm

-

- # Remove only Claude's fleet skills

- apra-fleet uninstall --llm claude --skill fleet

-

- # Remove only the workflow runtime + built-in workflows, keep user-authored ones

- apra-fleet uninstall --skill workflows

- ```

-

- If the fleet server is running, uninstall aborts and tells you to re-run with

- `--force`. Full detail: [docs/features/uninstall.md](features/uninstall.md).

-

- ## Customizing model tier mapping

-

- By default, each provider maps the three tiers (`cheap`, `standard`, `premium`)

- to hardcoded model names. You can override any of these per-provider by creating

- a `config.json` file in the Fleet data directory:

-

- ```

- ~/.apra-fleet/data/config.json

- ```

-

- If you set `APRA_FLEET_DATA_DIR`, the file lives at

- `$APRA_FLEET_DATA_DIR/config.json` instead.

-

- **Schema example:**

-

- ```json

- {

- "providers": {

- "agy": {

- "modelMapping": {

- "cheap": "GPT-OSS 120B (Medium)",

- "standard": "Gemini 3.1 Pro (High)",

- "premium": "Claude Opus 4.6 (Thinking)"

- }

- },

- "claude": {

- "modelMapping": {

- "cheap": "claude-haiku-4-5",

- "premium": "claude-opus-4-7"

- }

- }

- }

- }

- ```

-

- Provider keys: `claude`, `codex`, `copilot`, `agy`. Tier keys:

- `cheap`, `standard`, `premium`. All fields are optional -- omitted tiers fall

- back to the provider's built-in default.

-

- **Precedence:** per-member override (`update_member --model-cheap/standard/premium`)

- > user config > hardcoded provider default.

-

- If the file is missing, Fleet proceeds with built-in defaults. If the JSON is

- malformed, Fleet logs a warning to stderr and ignores the file.

-

- ## Self-update

-

- Update the fleet binary to the latest release:

-

- ```bash

- apra-fleet update

- ```

-

- This checks the latest GitHub release, downloads the installer for your

- platform, and re-runs it automatically. The server restarts with the new

- binary. If you are already on the latest version it reports so and exits. Full

- detail: [docs/features/update.md](features/update.md).

-

- ### Stopping a running server before an overwrite install

-

- `install --force` (and `update`, which drives the same path) must stop the

- currently-running server before copying the new binary over the installed

- path, since the OS refuses to overwrite a binary that is still mapped into a

- running process. A single termination signal followed by a fixed delay is not

- reliable: a singleton that is mid-request can take longer to exit than an

- arbitrary fixed sleep, and a copy attempted before it actually exits fails

- outright and leaves the old server running.

-

- The install path instead polls process liveness over a bounded grace window

- after the initial termination signal, and escalates to a harder kill signal

- if the process is still alive once that window elapses, polling again over a

- second (shorter) window before giving up. The binary copy is only attempted

- once the old process is confirmed gone. Symmetrically, the "stopped running

- server" success message is gated on that same confirmation rather than

- printed unconditionally -- if the process is still detected running after

- both the initial signal and the escalation, install reports a clear error

- (with the manual kill command for the platform) and exits non-zero instead of

- proceeding into a copy that would fail anyway or claiming success it can't

- back up.

-

- ### Replaying the npm-publish smoke step locally with an unrelated server running

-

- CI's "Pack + install into a clean temp prefix (fleet-sprint smoke test)" step

- packs the CLI and installs it into an isolated, throwaway prefix. That step

- (and any local replay of it) is safe to run even while an unrelated

- apra-fleet server is already up on the same machine, because the

- running-process guard is scoped to the install being performed rather than

- to any apra-fleet process anywhere on the OS: it only fires when the running

- server's data dir matches the data dir this install targets, or when the

- running executable it detects lives under the install prefix being written

- (the ETXTBSY case). A server running against a different data dir and a

- different install prefix does not trip it.

-

- To replay the step locally, isolate the install the same way CI does by

- pointing these env vars at throwaway locations before running

- `apra-fleet install`:

-

- - `HOME` (or `USERPROFILE` on Windows) -- so the default data dir and install

- prefix resolve under a temp directory instead of your real home.

- - `APRA_FLEET_DATA_DIR` -- overrides the data dir directly if you want it

- separate from `HOME`.

- - The install prefix (where the packed CLI is installed) -- point it at a

- clean temp directory distinct from any prefix an existing server was

- installed into.

-

- `install --force` is not needed for this replay, and must not be used just

- to kill an unrelated apra-fleet server -- `--force` exists to stop the

- server that owns the install being overwritten, not to clear the machine of

- unrelated servers so a differently-scoped install can proceed. As long as

- the data dir and install prefix are isolated from any running server, a

- plain `apra-fleet install` (no `--force`) completes without the guard

- firing.

- </doc>

-

- <doc title="Update" desc="Keeping Fleet and its members updated.">

- # Self-Update Command

-

- ## What it does

-

- `apra-fleet update` checks GitHub for the latest stable release and, if a newer

- version exists, downloads its installer and runs it -- no manual download step

- needed.

-

- ## Flags

-

- | Command | Effect |

- |---------|--------|

- | `apra-fleet update` | Check for and install the latest stable release. |

- | `apra-fleet update --check` | Report whether a newer release exists, without installing. |

- | `apra-fleet update --help` | Show usage. |

-

- ## Behaviour

-

- `apra-fleet update` with no flags:

-

- 1. Prints `Checking for updates...`.

- 2. Fetches release metadata from

- `https://api.github.com/repos/Apra-Labs/apra-fleet/releases/latest` with a

- 5-second timeout. On a non-OK response it prints

- `Error: Could not check for updates (Status: <code>)` and stops.

- 3. Compares the release tag against the installed version (the `_<hash>` build

- suffix is stripped first). If the tag is a pre-release (`-alpha`, `-beta`,

- `-rc`) or is not newer than the installed version, it prints an

- "up to date" message and stops.

- 4. Selects the installer asset for the current platform:

-

- | Platform | Asset |

- |----------|-------|

- | Windows x64 | `apra-fleet-installer-win-x64.exe` |

- | macOS ARM | `apra-fleet-installer-darwin-arm64` |

- | Linux x64 | `apra-fleet-installer-linux-x64` |

-

- If no matching asset is found it prints

- `Error: Could not find installer for platform <platform>` and stops.

- 5. Prints `Updating to <tag> -- restarting...`, downloads the installer into the

- system temp directory, and (on macOS/Linux) marks it executable.

- 6. Spawns the installer detached, then exits with status 0:

-

- ```

- <installer> install --force --llm <provider> --skill <skill> --workflows <mode>

- ```

-

- `--force` makes the installer stop the running apra-fleet server before

- replacing the binary. The `--llm`, `--skill`, and `--workflows` values come

- from `install-config.json` (see below).

-

- Any unexpected error is caught and printed as `Error: Update failed -- <message>`.

-

- ## install-config.json

-

- `update` reads `~/.apra-fleet/data/install-config.json` and uses the first

- provider entry to recover the `--llm` provider, its `--skill` set, and its

- persisted `workflowsMode` (`all` or `none`), so the update preserves the

- original install configuration -- including refreshing built-in workflow

- assets to the new version while leaving any user-authored workflows on disk

- untouched. `workflowsMode` is optional for backward compatibility: an

- `install-config.json` written before the workflow subsystem existed won't

- have the field, so `update` defaults it to `all`, matching a fresh install.

- If the file is missing or cannot be parsed, `update` prints a warning and

- falls back to `--llm claude --skill all --workflows all`.

-

- ## Stopping and restarting the server

-

- The installer is run with `--force`, so it stops the running apra-fleet server

- (and its workers) before overwriting the binary. The server is not a daemon --

- it starts again on demand the next time an LLM CLI connects to it (run `/mcp` in

- Claude Code, or restart the CLI).

-

- `update` spawns the installer detached and exits 0 immediately, so it prints

- `Updating to <tag> -- restarting...` before the installer has actually finished.

- To confirm the new version is in place, check afterwards:

-

- ```

- apra-fleet --version

- ```

-

- ## Notes

-

- - `apra-fleet update` always installs when a newer stable release exists. Use

- `apra-fleet update --check` first if you only want to see whether one is

- available.

- - The 5-second timeout covers the release-metadata check only; the installer

- download itself is not time-bounded.

- - The installer overwrites the binary in place with no `.bak`. To roll back,

- download the previous release's installer and run it with `--force`.

- </doc>

-

- <doc title="SSH Setup" desc="Enabling SSH access on a remote machine.">

- <!-- llm-context: Step-by-step guide for enabling SSH on remote machines so they can be registered as fleet members. Covers Linux (OpenSSH), macOS, and Windows (OpenSSH Server). Consult when a user can't connect to a remote member or needs to set up SSH for the first time. -->

- <!-- keywords: SSH, setup, OpenSSH, Linux, macOS, Windows, sshd, firewall, key auth, remote member, connectivity -->

- <!-- see-also: ../README.md (member registration after SSH is ready), adr-oob-password.md (password handling) -->

-

- # SSH Server Setup for Fleet Members

-

- Enable SSH on remote machines so they can be registered with `register_member`.

-

- ---

-

- ## Windows

-

- Windows is the most involved. Run all commands in **PowerShell as Administrator**.

-

- ### 1. Install OpenSSH Server

-

- ```powershell

- # Check if already installed

- Get-WindowsCapability -Online | Where-Object Name -like 'OpenSSH.Server*'

-

- # Install it

- Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0

- ```

-

- Or: **Settings > Apps > Optional Features > Add a feature > OpenSSH Server**.

-

- ### 2. Start sshd and enable on boot

-

- ```powershell

- Start-Service sshd

- Set-Service -Name sshd -StartupType Automatic

- ```

-

- ### 3. Firewall rule

-

- The installer usually creates this, but verify:

-

- ```powershell

- # Check

- Get-NetFirewallRule -Name *ssh*

-

- # Create if missing

- New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH Server (sshd)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22

- ```

-

- ### 4. Admin user gotcha

-

- If your user is in the **Administrators** group, SSH ignores `~/.ssh/authorized_keys`. Keys must go in:

-

- ```

- C:\ProgramData\ssh\administrators_authorized_keys

- ```

-

- And the file needs restricted permissions:

-

- ```powershell

- # Add your public key

- Add-Content C:\ProgramData\ssh\administrators_authorized_keys "ssh-ed25519 AAAA... you@host"

-

- # Fix permissions (must be owned by SYSTEM/Administrators only)

- icacls C:\ProgramData\ssh\administrators_authorized_keys /inheritance:r /grant "SYSTEM:(F)" /grant "Administrators:(F)"

- ```

-

- ### 5. Verify

-

- From another machine:

-

- ```bash

- ssh user@windows-host "echo ok"

- ```

-

- ---

-

- ## macOS

-

- ### 1. Enable Remote Login

-

- **System Settings > General > Sharing > Remote Login** (toggle on).

-

- Or via terminal:

-

- ```bash

- sudo systemsetup -setremotelogin on

- ```

-

- ### 2. Verify

-

- ```bash

- ssh user@mac-host "echo ok"

- ```

-

- ---

-

- ## Linux (Ubuntu/Debian)

-

- ### 1. Install and start

-

- ```bash

- sudo apt install openssh-server

- sudo systemctl enable --now ssh

- ```

-

- ### 2. Verify

-

- ```bash

- ssh user@linux-host "echo ok"

- ```

-

- ---

-

- ## Jetson / Embedded Linux

-

- Usually pre-installed. Just confirm it's running:

-

- ```bash

- sudo systemctl status sshd

- ```

-

- If not running:

-

- ```bash

- sudo systemctl enable --now sshd

- ```

-

- ---

-

- ## Troubleshooting

-

- | Symptom | Cause | Fix |

- |---|---|---|

- | **Connection refused** | sshd not running | Start the service (see above) |

- | **Permission denied** | Wrong password or key not deployed | Check `~/.ssh/authorized_keys` on target; on Windows admin users, check `administrators_authorized_keys` (see above) |

- | **Connection timed out** | Firewall blocking port 22 | Add inbound rule for TCP/22 |

- | **Windows: key auth not working for admin user** | Keys in `~/.ssh/authorized_keys` are ignored for admin accounts | Move keys to `C:\ProgramData\ssh\administrators_authorized_keys` and fix permissions |

- | **Host key verification failed** | Host key changed (reinstall, new machine) | `ssh-keygen -R <host>` to clear old key |

- </doc>

-

- <doc title="Out-of-Band Auth" desc="Securely managing credentials and passwords for fleet members.">

- # OOB Auth -- Terminal Mechanism and SSH/Headless Fallback

-

- Covers the design of the out-of-band credential collection system and the SSH/headless fix shipped in Sprint 1 (#106).

-

- ---

-

- ## Background: Why OOB Auth Exists

-

- Credentials (passwords, API keys, confirmation prompts) must not pass through the LLM -- the model must never see plaintext secrets. The OOB (out-of-band) mechanism collects credentials in a separate UI context, passes them over a local socket, and delivers them to the fleet server without them appearing in the prompt stream.

-

- ---

-

- ## Unix Domain Socket (UDS) Architecture

-

- The fleet server creates a socket at `~/.apra-fleet/data/auth.sock` (Linux/macOS) or a Windows named pipe equivalent. This is a filesystem object -- any process running as the same user on the same machine can reach it.

-

- **Flow:**

-

- 1. A tool requiring a credential calls `collectOobInput()` (in `src/services/auth-socket.ts`).

- 2. `collectOobInput` registers a pending auth request with a 10-minute TTL via `createPendingAuth()`.

- 3. It calls `launchAuthTerminal()` to open a terminal window running `apra-fleet secret --set <memberName>`.

- 4. The launched process prompts the user, reads input with masked display (LLM cannot see it), and sends the value over the UDS as a JSON message.

- 5. `collectOobInput` **blocks** -- `waitForPassword()` awaits a Promise that resolves only when the credential arrives over the socket (or a cancellation/timeout fires). The call does not return early with a "Waiting..." status.

- 6. On receipt, the credential is consumed from the pending store and returned to the caller. The tool call then completes with a success message of the form `[OK] NAME stored [session/persistent]. Use {{secure.NAME}} in commands.`

-

- **Key property:** The UDS socket is a filesystem object -- no GUI or display server is required to write to it. Any process on the machine, including one launched in a second SSH terminal, can deliver credentials.

-

- ---

-

- ## Display Detection -- The #106 Fix

-

- ### Problem

-

- `launchAuthTerminal` attempted GUI terminal emulators in order (`gnome-terminal -> xterm -> x-terminal-emulator`) on Linux. On SSH sessions:

-

- - `which gnome-terminal` succeeds even when `$DISPLAY` is unset (the binary exists but can't connect).

- - Spawn succeeds -> process exits immediately -> **"[FAIL] Password entry cancelled"** error fires.

- - The error implies the user cancelled, not that the environment is headless.

-

- The same issue on Windows: `start /wait cmd.exe` opens a window on the physical console, invisible to the SSH user.

-

- ### Solution

-

- Two environment-variable checks added to `auth-socket.ts`:

-

- ```typescript

- // Returns true when X11 or Wayland display is available

- export function hasGraphicalDisplay(): boolean {

- return Boolean(process.env.DISPLAY || process.env.WAYLAND_DISPLAY);

- }

-

- // Returns true when running on an interactive Windows desktop session

- // SSH sessions and service contexts have SESSIONNAME !== 'Console'

- export function hasInteractiveDesktop(): boolean {

- return process.env.SESSIONNAME === 'Console';

- }

- ```

-

- `launchAuthTerminal` checks these **before** attempting any terminal emulator:

-

- - **Linux, `$DISPLAY` and `$WAYLAND_DISPLAY` both unset:** Skip all GUI terminal emulators. Return actionable fallback message.

- - **Windows, `SESSIONNAME !== 'Console'`:** Skip `cmd.exe start /wait`. Return actionable fallback message.

- - **GUI desktop (display available):** Unchanged -- auto-launches terminal as before.

-

- ### Why check env vars rather than probing the socket

-

- Probing (attempting a spawn and checking exit code) is what caused the misleading error in the first place. Env var checks are fast, zero-side-effect, and accurate for the cases that matter: X11/Wayland forwarding sets `$DISPLAY`, and Windows service contexts have a distinct `SESSIONNAME`.

-

- Edge case accepted: X11 forwarding where `$DISPLAY` is set but the forwarded display is unreachable. This is acceptable -- if the terminal fails to launch, existing fallback logic catches it, and a user with X11 forwarding active almost certainly has a working display.

-

- **Windows OOB window close:** Closing the OOB terminal window on Windows now returns immediately with a cancellation message. Previously, closing the window caused the tool call to hang for up to 5 minutes waiting for a password that would never arrive. The fix attaches a close-signal handler so that window dismissal resolves the pending Promise with a cancellation error immediately.

-

- ---

-

- ## The `! apra-fleet secret --set <name>` Pattern

-

- On headless environments, the fallback message instructs the user to run:

-

- ```

- ! apra-fleet secret --set <actual-member-name>

- ```

-

- The `!` prefix is the Claude Code "run in shell" operator -- it executes the command in the user's terminal without passing it to the LLM. This is the **single-terminal approach**: the user does not need to open a second window; they run the secret collection command inline in the same Claude Code session.

-

- The message includes the **actual member name** (not a placeholder). The member name is available at the point `launchAuthTerminal` is called -- it is passed as the `memberName` parameter.

-

- **Full fallback message text (Linux headless, credential-collection mode):**

- ```

- fallback:No graphical display detected (SSH or headless session).

-

- Run this in a separate terminal to provide the credential:

- ! apra-fleet secret --set <memberName>

-

- Alternatively, pre-store the value with credential_store_set and reference it as {{secure.NAME}} in the credential field.

- ```

-

- **Full fallback message text (Linux headless, egress-confirm mode):**

- ```

- fallback:No graphical display detected (SSH or headless session).

-

- Run this in a separate terminal to confirm:

- ! apra-fleet secret --confirm <memberName>

-

- Alternatively, pre-store the value with credential_store_set and reference it as {{secure.NAME}} in the credential field.

- ```

-

- The `fallback:` prefix is a protocol marker consumed by `collectOobInput` to distinguish the fallback path from a successful terminal launch. It is stripped before the message reaches the user.

-

- ---

-

- ## Fallback: Second Terminal

-

- When the `!` operator isn't available or the user is in a non-Claude Code context, the fallback instruction is to open a second terminal and run `apra-fleet secret --set <memberName>` there. Because the UDS socket is a filesystem object, the second terminal's `apra-fleet secret --set` process connects to the same socket and delivers the credential to the waiting fleet server -- no GUI required.

-

- ---

-

- ## Re-entrancy and Stale State

-

- If a terminal launch fails (fallback path), `collectOobInput` cleans up the pending auth state:

- - Clears the `passwordWaiters` entry for the member

- - Without this, `hasPendingAuth()` returns `true` on the next call, the re-entrant path skips `launchAuthTerminal`, and the call hangs waiting for a credential that will never arrive.

-

- This cleanup ensures retries always start fresh.

- </doc>

-

- <doc title="Git Authentication" desc="How git authentication works across fleet members.">

- <!-- llm-context: Design doc for fleet's git authentication system — scoped token provisioning via GitHub Apps, PATs, Bitbucket, and Azure DevOps. Read when a user asks how to give members git access, how tokens are scoped, or how credentials are managed. -->

- <!-- keywords: git auth, GitHub App, PAT, Bitbucket, Azure DevOps, token, scope, provision, revoke, credential, push, pull, clone -->

- <!-- see-also: ../README.md (step-by-step git auth setup), design-vcs-auth-onboarding.md (onboarding flow) -->

-

- # Design: Git Authentication for Fleet Members

-

- ## Problem

-

- Fleet members need git access (clone, push, force-push, issue management) across multiple git hosts (GitHub, Azure DevOps, Bitbucket, GitLab). Today there's no standardized way to provision git credentials to members, and no way to scope permissions per member role.

-

- Key requirements:

- - **Multi-host**: Same abstraction across GitHub, Azure DevOps, Bitbucket, GitLab, self-hosted

- - **Scoped permissions**: Read-only members shouldn't be able to push; dev members shouldn't force-push to main

- - **Short-lived tokens**: Compromised member = limited blast radius

- - **Zero user plumbing**: Users declare intent ("this member needs read access"), fleet handles the rest

- - **Audit trail**: Every token mint logged with member name, scope, timestamp

-

- ## Design

-

- ### User-Facing Config

-

- Members declare git access in their registration or member config:

-

- ```yaml

- members:

- code-analyst:

- host: 192.168.1.13

- work_folder: /Users/akhil/git/ApraPipes

- git_access: read

- git_repos: [Apra-Labs/ApraPipes]

-

- feature-dev:

- host: 192.168.1.13

- work_folder: /Users/akhil/git/ApraPipes

- git_access: push

- git_repos: [Apra-Labs/ApraPipes]

-

- release-bot:

- host: 192.168.1.14

- work_folder: /home/deploy/releases

- git_access: admin

- git_repos: ["*"]

-

- project-mgr:

- host: local

- work_folder: C:\akhil\project-tracking

- git_access: issues

- git_repos: [Apra-Labs/ApraPipes, Apra-Labs/apra-lic-mgr]

- ```

-

- ### Access Levels

-

- | Level | Git operations | Non-git |

- |---|---|---|

- | `read` | clone, pull, fetch, blame, log | - |

- | `push` | read + push to branches (branch protection blocks main/force-push) | - |

- | `admin` | read + push + force-push + tags + releases | CI/CD triggers |

- | `issues` | - (no code access) | issues, PRs, projects, comments |

- | `full` | admin + issues | Everything |

-

- ### Backend: GitHub App Token Minting

-

- For GitHub-hosted repos, use a **GitHub App** installed on the org.

-

- ```

- ┌─────────────────────────────────────────────┐

- │ apra-fleet-app (GitHub App) │

- │ Installed on: Apra-Labs org │

- │ App private key stored on PM/master │

- │ │

- │ Max permissions (app-level): │

- │ - contents: write │

- │ - issues: write │

- │ - pull_requests: write │

- │ - actions: write │

- │ - administration: write │

- └──────────────┬──────────────────────────────┘

- │

- PM mints scoped tokens per member at runtime:

- │

- ├──→ code-analyst: { contents: read, repos: [ApraPipes] }

- ├──→ feature-dev: { contents: write, repos: [ApraPipes] }

- ├──→ release-bot: { contents: write, admin: write, repos: [*] }

- └──→ project-mgr: { issues: write, pull_requests: write }

- ```

-

- **Token minting flow:**

-

- ```typescript

- // Using @octokit/app

- import { App } from "@octokit/app";

-

- const app = new App({

- appId: FLEET_GITHUB_APP_ID,

- privateKey: FLEET_GITHUB_APP_KEY,

- });

-

- async function mintGitToken(agent: Agent): Promise<string> {

- const octokit = await app.getInstallationOctokit(installationId);

-

- const { token } = await octokit.request(

- "POST /app/installations/{installation_id}/access_tokens",

- {

- installation_id: installationId,

- repositories: agent.git_repos, // scoped to specific repos

- permissions: mapAccessLevel(agent.git_access), // scoped permissions

- }

- );

-

- return token; // valid for 1 hour

- }

-

- function mapAccessLevel(level: string): Record<string, string> {

- switch (level) {

- case "read": return { contents: "read" };

- case "push": return { contents: "write" };

- case "admin": return { contents: "write", administration: "write", actions: "write" };

- case "issues": return { issues: "write", pull_requests: "write" };

- case "full": return { contents: "write", administration: "write", issues: "write", pull_requests: "write", actions: "write" };

- }

- }

- ```

-

- **Credential deployment to member:**

-

- ```typescript

- async function provisionGitAuth(agent: Agent): Promise<void> {

- const token = await mintGitToken(agent);

-

- // Configure git credential helper on the member

- await agent.executeCommand(

- `git config --global credential.helper '!f() { echo "password=${token}"; }; f'`

- );

-

- // Or more robustly, write a credential helper script

- await agent.executeCommand(`cat > ~/.fleet-git-credential << 'EOF'

- #!/bin/sh

- echo "protocol=https"

- echo "host=github.com"

- echo "username=x-access-token"

- echo "password=${token}"

- EOF

- chmod +x ~/.fleet-git-credential

- git config --global credential.helper ~/.fleet-git-credential`);

- }

- ```

-

- ### Backend: Azure DevOps

-

- Use an **Azure AD App Registration** (Service Principal):

-

- ```typescript

- // Using @azure/identity + azure-devops-node-api

- const credential = new ClientSecretCredential(tenantId, clientId, clientSecret);

- const token = await credential.getToken("499b84ac-1321-427f-aa17-267ca6975798/.default");

-

- // Deploy to member as PAT-style credential

- await agent.executeCommand(

- `git config --global credential.helper '!f() { echo "password=${token.token}"; }; f'`

- );

- ```

-

- ### Backend: Bitbucket

-

- Use a **Bitbucket OAuth Consumer** or **Repository Access Token**:

- - OAuth Consumer: org-level, token minting via client_credentials grant

- - Repository Access Token: per-repo, created via Bitbucket API, scoped permissions

-

- ### Backend: Self-hosted / GitLab

-

- - GitLab: **Project Access Tokens** or **Group Access Tokens** via API

- - Self-hosted: SSH keys (fallback — no token API available)

-

- ### Token Lifecycle

-

- ```

- Member startup / first git operation

- │

- ▼

- PM mints scoped token (1hr TTL)

- │

- ▼

- Deploy credential to member via execute_command

- │

- ▼

- Member uses git normally (clone/push/etc)

- │

- ▼

- Token nearing expiry? Auto-refresh before next git operation

- │

- ▼

- Member deregistered? Token expires naturally (1hr max)

- ```

-

- ### MCP Tool Interface

-

- New tool: `provision_git_auth`

-

- ```typescript

- // Input

- {

- agent_name: "feature-dev",

- // Optional overrides (defaults come from member config):

- git_host?: "github" | "azure" | "bitbucket" | "gitlab",

- access_level?: "read" | "push" | "admin" | "issues" | "full",

- repos?: string[],

- }

-

- // Output

- {

- status: "ok",

- host: "github.com",

- access_level: "push",

- repos: ["Apra-Labs/ApraPipes"],

- expires_at: "2026-03-03T16:00:00Z",

- token_prefix: "ghs_****" // masked for audit

- }

- ```

-

- ### Security Properties

-

- | Property | How it's achieved |

- |---|---|

- | **Least privilege** | Token scoped to declared repos + access level |

- | **Short-lived** | 1hr tokens, auto-refreshed |

- | **Auditable** | Token minting logged with member, scope, timestamp |

- | **Revocable** | Remove member = token expires naturally; revoke app installation for emergency |

- | **No secrets on members** | Members never see the app private key, only short-lived tokens |

- | **Compromised member** | Max 1hr window, scoped to declared repos only |

-

- ### Comparison with Alternatives

-

- | | SSH Keys | PATs | GitHub App (this design) |

- |---|---|---|---|

- | Per-member scoping | No | Manual | Automatic |

- | Token lifetime | Forever | Days-years | 1 hour |

- | Multi-host | Same key everywhere | Different per host | Abstracted |

- | User effort | Generate + register keys | Generate + distribute tokens | Declare `git_access: push` |

- | Revocation | Manual key removal | Manual token revocation | Auto-expires |

- | Audit | SSH logs | None built-in | Full mint log |

-

- ## Implementation Plan

-

- 1. **GitHub App setup** — create app, install on org, store private key in fleet config

- 2. **`provision_git_auth` tool** — mints scoped token, deploys credential to member

- 3. **Auto-provisioning** — mint token on member startup or first git operation

- 4. **Auto-refresh** — check token expiry before git operations, refresh if needed

- 5. **Multi-host backends** — Azure DevOps, Bitbucket, GitLab adapters (same `provision_git_auth` interface)

- 6. **Member config** — add `git_access` and `git_repos` fields to member registration

- </doc>

-

- <doc title="Provider Guide" desc="Choosing an LLM provider -- strengths, role recommendations, and gotchas.">

- <!-- llm-context: User-facing guide for choosing an LLM provider in apra-fleet. Consult when a user asks which provider to use for a role (PM, doer, reviewer), what each provider is good at, or what limitations to expect. For CLI flags, credential paths, and integration internals, see provider-matrix.md. -->

- <!-- keywords: provider, Claude, Antigravity, Codex, Copilot, choose, role, PM, doer, reviewer, gotchas, limitations, context window, max_turns, OAuth -->

- <!-- see-also: ../README.md (provider setup instructions), provider-matrix.md (full CLI and integration reference) -->

-

- # Choosing an LLM Provider

-

- Fleet supports Claude, Antigravity (agy), Codex, and Copilot. Members can run different providers and mix them freely within a single fleet.

-

- ## Provider strengths

-

- - **Claude** - Balanced coding and reasoning; fine-grained per-tool permissions via `settings.local.json`.

- - **Antigravity** - High-performance Gemini-based agentic CLI; supports large context windows, background tasks, and native beads task tracking.

- - **Codex** - Structured-output enforcement via `--output-schema`; native subagent parallelism for concurrent subtasks with less orchestration overhead.

- - **Copilot** - Multi-model marketplace (Claude + GPT families in one CLI); auto-compaction keeps sessions running indefinitely.

-

- ## Recommended provider by role

-

- | Role | Recommended | Why |

- |------|-------------|-----|

- | PM (orchestrator) | Claude Code or Antigravity (agy) | Both plan and orchestrate well - both support planning, background tasks, and premium models (e.g., Opus / premium-tier). |

- | Doer | Any provider | Sonnet, Antigravity, Codex, Copilot - mix freely. |

- | Reviewer | Premium-tier models | Catches subtle issues smaller models miss. |

-

- ## Gotchas worth knowing

-

- - **`max_turns` is Claude-only.** On Codex, Copilot, and Antigravity, use `timeout_s` instead to bound execution time.

- - **Copilot needs a paid GitHub Copilot subscription** (Pro, Business, or Enterprise) and has the smallest context window (64K). It is best suited for smaller, focused tasks.

-

- ## Mixing providers in one fleet

-

- Every member runs its own LLM backend, and they collaborate across vendors. Put

- a Claude doer with an Antigravity reviewer, or the reverse -- the reviewer's model

- disagrees with the doer's by construction, so it catches issues a same-model

- review would wave through.

-

- A fleet that has run in production:

-

- ```

- pm-1 Opus 4.7 orchestrator

- doer-1 Sonnet 4.6 feature work

- doer-2 Antigravity large-context tasks

- reviewer Opus 4.7 final review

- ```

-

- **OpenCode and local models.** OpenCode works with any OpenAI-compatible

- endpoint (Ollama, vLLM, etc.), so it is the provider for self-hosted models.

- The model endpoint is the user's responsibility -- Fleet installs the CLI and

- agents but does not provision or manage the inference server. Configure the

- provider and base URL in `opencode.json`; see

- [opencode-exploration.md](opencode-exploration.md) for details.

-

- Because OpenCode members can run any model, model tiers (cheap / standard /

- premium) are set per member at registration via `model_tiers` in

- `register_member`. A single-model entry fills all three tiers.

-

- **Registering a member from a shell.** `apra-fleet register-member --name

- <name> --path <folder> [options]` is a shell-drivable equivalent of the

- `register_member` MCP tool, for contexts that can run shell commands but

- cannot make MCP tool calls (scripted setup, CI, an agent role without MCP

- tool access). It shares the exact same validation and registration logic as

- the MCP tool -- both converge on one underlying registration function, so

- the two entry points can never drift apart. Run `apra-fleet register-member

- --help` for the full flag reference.

-

- ---

-

- To override which model each tier resolves to on a per-provider basis, see

- [Customizing model tier mapping](install.md#customizing-model-tier-mapping).

-

- ---

-

- Extending Fleet's provider support, or need the full CLI / integration detail? See [docs/provider-matrix.md](provider-matrix.md).

- </doc>

-

- <doc title="Cloud Compute" desc="AWS and cloud compute integration for fleet members.">

- <!-- llm-context: This guide covers apra-fleet's AWS EC2 integration — auto start/stop, GPU-aware idle detection, long-running tasks, cost tracking, and custom workload detection. Consult when a user asks about cloud instances, GPU workloads, cost management, or task monitoring. -->

- <!-- keywords: AWS, EC2, cloud, GPU, nvidia-smi, idle detection, auto stop, cost tracking, long-running task, monitor_task, cloud_control -->

- <!-- see-also: ../README.md (general setup), architecture.md (how fleet manages members) -->

-

- # Cloud Compute Guide

-

- ## 1. Overview

-

- Cloud compute extends apra-fleet with full EC2 lifecycle management:

-

- - **Auto-start**: stopped EC2 instances start automatically when a tool is called

- - **Idle auto-stop**: instances stop themselves after a configurable period of inactivity

- - **Long-running tasks**: background task wrapper survives SSH disconnects, auto-retries on crash, keeps idle manager from stopping the instance while work is running

- - **GPU monitoring**: `fleet_status`, `member_detail`, and `monitor_task` report live GPU utilization via `nvidia-smi`

- - **Cost visibility**: uptime and estimated on-demand cost shown in status output

-

- ---

-

- ## 2. Architecture

-

- ```

- PM (Claude)

- |

- | MCP calls

- v

- apra-fleet MCP server (this machine)

- | |

- | SSH (execCommand) | AWS CLI

- v v

- EC2 instance AWS EC2 API

- (cloud member) (start/stop/describe)

- ```

-

- The MCP server runs on your local machine. It talks to EC2 via the AWS CLI and to the instance via SSH. The idle manager runs inside the server process, polling every 60 seconds.

-

- ---

-

- ## 3. AWS Setup

-

- ### IAM permissions required

-

- The AWS identity (user or role) running the server needs:

-

- ```json

- {

- "Effect": "Allow",

- "Action": [

- "ec2:StartInstances",

- "ec2:StopInstances",

- "ec2:DescribeInstances"

- ],

- "Resource": "*"

- }

- ```

-

- ### AWS CLI configuration

-

- Install the AWS CLI and configure credentials:

-

- ```bash

- aws configure # default profile

- aws configure --profile apra # named profile (use cloud_profile param)

- ```

-

- Verify access:

-

- ```bash

- aws ec2 describe-instances --instance-ids i-0abc123def456789a --region us-east-1

- ```

-

- ---

-

- ## 4. Registering a Cloud Member

-

- ```

- register_member(

- friendly_name = "gpu-trainer",

- work_folder = "/home/ubuntu/training",

- member_type = "remote",

- username = "ubuntu",

- cloud_provider = "aws",

- cloud_instance_id = "i-0abc123def456789a",

- cloud_region = "us-east-1",

- cloud_profile = "apra", # optional AWS CLI profile

- key_path = "/home/you/.ssh/gpu-trainer.pem",

- cloud_idle_timeout_min = 30, # auto-stop after 30min idle

- )

- ```

-

- **Parameter notes:**

-

- | Parameter | Required | Notes |

- |---|---|---|

- | `cloud_provider` | yes | Only `"aws"` supported |

- | `cloud_instance_id` | yes | EC2 instance ID, e.g. `i-0abc…` |

- | `cloud_region` | no | Default: `us-east-1` |

- | `cloud_profile` | no | AWS CLI named profile |

- | `key_path` | yes | Path to SSH private key on this machine; also sets the SSH `key_path` for the member |

- | `cloud_idle_timeout_min` | no | Default: 30. Per-agent idle timeout in minutes. |

-

- The instance does **not** need to be running at registration time. The server will start it on first use.

-

- ---

-

- ## 5. Auto-Start

-

- When `execute_command`, `execute_prompt`, or `send_files` is called on a cloud member, `ensureCloudReady()` runs first:

-

- 1. Calls `aws ec2 describe-instances` to get current state

- 2. **stopped** → calls `aws ec2 start-instances`, waits for `running` state

- 3. **stopping** → waits for `stopped`, then starts

- 4. **pending** → waits for `running`

- 5. **running** → verifies public IP is current, updates registry if changed

- 6. **terminated / shutting-down** → throws error (cannot be used)

-

- After the instance is running:

- - Polls SSH port (TCP connect) every 2 seconds, up to 60 seconds

- - Re-provisions Claude OAuth credentials (`provision_llm_auth`) — F5

- - Re-mints GitHub App tokens if the member has git repos configured — F5

-

- The returned agent object has the fresh public IP. All subsequent SSH calls use it.

-

- ---

-

- ## 6. Idle Auto-Stop

-

- The idle manager runs in the background and checks all cloud members every 60 seconds.

-

- **Stop conditions (all must be true):**

- 1. Member has been idle longer than `cloud_idle_timeout_min` (per-agent) or the global fallback

- 2. Instance is currently `running`

- 3. No GPU compute processes detected (`nvidia-smi` shows no active jobs)

- 4. No fleet or other Claude processes running in the work folder

-

- **Timer reset:** Every successful tool call (`execute_command`, etc.) calls `touchAgent()`, which resets the idle timer for that member.

-

- **Server restart persistence (R-9):** On startup, the idle manager pre-loads `lastActivity` from each member's `lastUsed` timestamp in the registry. A member that was active 5 minutes before a server restart will not be stopped until the full timeout expires from that last-used time.

-

- **Safe default:** If activity cannot be determined (SSH unreachable, nvidia-smi error), the stop is deferred. Unknown = don't stop.

-

- ---

-

- ## 7. Long-Running Tasks

-

- For GPU training jobs or other tasks that outlast an SSH session, use `long_running=true`:

-

- ```

- execute_command(

- member_id = "<gpu-trainer-id>",

- command = "python train.py --epochs 100 --data /data/train",

- long_running = true,

- max_retries = 3,

- restart_command = "python train.py --resume checkpoint.pt", # F1

- )

- ```

-

- **What happens:**

-

- 1. A bash wrapper script is generated and base64-encoded

- 2. The wrapper is decoded and written to `~/.fleet-tasks/<task_id>/run.sh` on the member

- 3. Launched with `nohup bash run.sh &` — survives SSH disconnect

- 4. Returns immediately: `Task launched: task_id=task-<id>`

-

- **Wrapper behavior:**

-

- - Writes PID to `task.pid`, JSON status to `status.json`

- - Background loop touches `~/.fleet-tasks/<task_id>/activity` every 5 minutes while running — this prevents the idle manager from stopping the instance during active work (F3)

- - On non-zero exit: retries up to `max_retries` times using `restart_command` (F1)

- - `restart_command` is designed for checkpoint resume (different flags on retry)

- - Falls back to `command` if `restart_command` not provided

- - On completion or max retries: updates `status.json`, removes `task.pid`

-

- **Checking progress:**

-

- ```

- monitor_task(

- member_id = "<gpu-trainer-id>",

- task_id = "task-lx4k2z",

- auto_stop = true, # stop instance automatically when task completes

- )

- ```

-

- Returns JSON:

-

- ```json

- {

- "taskId": "task-lx4k2z",

- "status": "running", // running | completed | failed | retrying | unknown

- "exitCode": null,

- "retries": 0,

- "started": "2026-03-18T10:00:00Z",

- "updated": "2026-03-18T10:45:00Z",

- "pidAlive": true,

- "gpuUtilization": 87,

- "logTail": "Epoch 45/100, loss=0.234..."

- }

- ```

-

- ---

-

- ## 8. cloud_control Reference

-

- Manual control over a cloud member's instance:

-

- ```

- cloud_control(member_id="<id>", action="start") # start + wait for SSH ready

- cloud_control(member_id="<id>", action="stop") # stop immediately (bypasses idle timer)

- cloud_control(member_id="<id>", action="status") # show current state + cost

- ```

-

- **Actions:**

-

- | Action | Behaviour |

- |---|---|

- | `start` | Calls `ensureCloudReady` — starts the instance, waits for SSH, re-provisions auth |

- | `stop` | Calls `aws ec2 stop-instances` directly — immediate, no idle check |

- | `status` | Calls `getInstanceDetails` — returns state, IP, instance type, uptime, estimated cost |

-

- **`stop` vs idle auto-stop:** `cloud_control stop` bypasses all activity checks. Use it to forcefully stop an instance regardless of what's running. The idle manager's stop goes through GPU + process checks first.

-

- ---

-

- ## 9. Cost Estimation

-

- `fleet_status` and `member_detail` show an estimated running cost based on:

-

- - Instance type (from `aws ec2 describe-instances`)

- - Uptime = `now - LaunchTime` (from the same API call)

- - Hourly rate from a built-in lookup table (`src/services/cloud/cost.ts`)

-

- **Supported families:** g4dn, g5, p3, p4d, t3, m5, c5 (us-east-1 on-demand pricing)

-

- **Limitations:**

- - Rates are hard-coded approximations. Actual AWS charges may differ due to spot pricing, savings plans, data transfer, EBS, etc.

- - Instance types not in the table show `?` for cost.

- - The lookup table is in `src/services/cloud/cost.ts` — edit `HOURLY_RATES` to add custom types or update prices.

-

- ---

-

- ## 10. Supported Platforms

-

- Cloud compute features are designed and tested for **Linux** EC2 instances (Ubuntu, Amazon Linux 2, Debian).

-

- | Feature | Linux | macOS | Windows |

- |---|---|---|---|

- | GPU detection (`nvidia-smi`) | ✅ Full | ❌ Not supported | ❌ Not supported |

- | Long-running task wrapper | ✅ Full | ⚠ Untested | ❌ Not supported |

- | Idle activity monitoring | ✅ Full | ⚠ Partial | ❌ Not supported |

- | Auto-start / auto-stop | ✅ Full | ✅ Full | ✅ Full |

- | SSH connectivity | ✅ Full | ✅ Full | ⚠ Requires OpenSSH |

-

- **Notes:**

- - Registering a cloud member with a non-Linux OS will succeed but show a warning about unsupported features.

- - The task wrapper script (`long_running=true`) uses `bash`, `nohup`, and POSIX shell utilities. It is not compatible with Windows Command Prompt or PowerShell.

- - macOS GPU detection is not supported (`nvidia-smi` is not available on Apple Silicon or macOS in general).

- - Only `aws` is supported as a `cloud_provider`. GCP and Azure support is planned.

- </doc>

-

- <doc title="Writing Skills" desc="How to write your own skill for use in the fleet.">

- # Writing a skill

-

- Fleet ships two skills -- `fleet` (the MCP tool reference) and `pm` (the Project

- Manager workflow). A skill is how you package a *workflow* on top of Fleet so it

- can be invoked by name and reused. This page explains what a skill is and how to

- build your own.

-

- ## What a skill is

-

- A skill is a directory of Markdown that your AI coding agent loads as

- instructions. It is not compiled code and there is no plugin API to implement --

- a skill *describes* a workflow, and the agent carries it out using Fleet's MCP

- tools.

-

- The PM skill, for example, is Markdown that tells the agent how to plan a

- sprint, dispatch a doer, run a reviewer, and raise a PR. Everything it does, it

- does by calling Fleet tools like `register_member` and `execute_prompt`.

-

- ## Where skills live

-

- Skills are installed into your provider's skills directory:

-

- | Provider | Directory |

- |----------|-----------|

- | Claude | `~/.claude/skills/<name>/` |

- | Antigravity (agy) | `~/.gemini/antigravity-cli/skills/<name>/` |

-

- Fleet's installer writes `fleet/` and `pm/` there. Your own skill is just

- another directory alongside them.

-

- ## Anatomy

-

- A skill directory contains one required file and any number of supporting ones:

-

- ```

- my-skill/

- SKILL.md <- required: the entry point

- helper-notes.md <- optional: sub-documents the agent reads on demand

- templates/ <- optional: files the skill sends to members

- scripts/ <- optional: helper scripts

- ```

-

- ### SKILL.md

-

- `SKILL.md` opens with YAML frontmatter, then the workflow body:

-

- ```markdown

- ---

- name: my-skill

- description: One sentence on what this skill does and when to use it.

- note: This skill requires the 'fleet' skill to function.

- ---

-

- # My Skill

-

- You are a ... that ...

-

- ## Step 1

- ...

- ```

-

- - `name` -- the skill's identifier; matches the directory name.

- - `description` -- used to decide when the skill is relevant. Be specific.

- - `note` -- optional; declare a dependency on the `fleet` skill if you call

- Fleet MCP tools.

-

- Keep `SKILL.md` focused. Push detail into sub-documents and reference them by

- filename so the agent loads them only when needed -- this is how the `pm` skill

- keeps `SKILL.md` short while `single-pair-sprint.md`, `doer-reviewer.md`, and

- the `tpl-*.md` templates carry the depth.

-

- ## The tools a skill can use

-

- A skill coordinates agents through Fleet's MCP tools. The most common:

-

- | Tool | Use |

- |------|-----|

- | `register_member` | Add a machine or local workspace as a member. |

- | `execute_prompt` | Run an AI prompt on a member. |

- | `execute_command` | Run a shell command on a member (no tokens). |

- | `send_files` / `receive_files` | Move files to and from a member. |

- | `compose_permissions` | Generate provider-native permission config. |

- | `fleet_status` / `member_detail` | Inspect member state. |

-

- The `fleet` skill documents the full tool set. Your skill should activate the

- `fleet` skill (via the `note` field) rather than re-documenting tools.

-

- ## Build your own

-

- 1. Create `~/.claude/skills/my-skill/SKILL.md` with the frontmatter above.

- 2. Write the workflow as numbered steps, in plain imperative prose. Reference

- Fleet tools by name where the agent should call them.

- 3. Move long reference material into sibling `.md` files; link them by filename.

- 4. Test by invoking the skill in your AI coding agent and watching it run.

-

- ## Worked examples

-

- The two skills in this repository are the best reference:

-

- - [`skills/fleet/SKILL.md`](../skills/fleet/SKILL.md) -- the MCP tool reference

- and member-management mechanics.

- - [`skills/pm/SKILL.md`](../skills/pm/SKILL.md) -- a full multi-step workflow:

- sprint variants, doer-reviewer pairing, templates, and lifecycle commands.

-

- Read those alongside this page when designing your own.

- </doc>

-

- <doc title="Beads" desc="How Fleet uses Beads -- the bundled open-source issue tracker and the PM skill's persistent task database.">

- <!-- llm-context: Reference for how Apra Fleet uses Beads, the bundled open-source issue tracker. Consult when a user asks about bd commands, task tracking, sprint epics, backlog management, or the PM skill's persistent task state. -->

- <!-- keywords: Beads, bd, task, epic, sprint, backlog, pm, lifecycle, dependency, issue tracker, bd ready, bd create, bd close -->

- <!-- see-also: ../README.md (PM skill overview), ../skills/pm/SKILL.md (PM skill reference), ../skills/pm/beads.md (internal PM Beads rules) -->

-

- # Beads -- Fleet's Persistent Task Tracker

-

- Beads is a bundled open-source local issue tracker installed alongside Fleet by

- `apra-fleet install`. It provides the `bd` CLI and serves as the PM skill's

- persistent task database across all sprints and sessions.

-

- ## What Beads Does

-

- - **One central DB** -- the PM agent runs `bd init` once in its own working

- directory. A single Beads database tracks all projects, all members, and all

- sprints. Run `bd list --all --pretty` for a global view without reading files.

- - **Epics and tasks** -- each sprint gets an epic; each PLAN.md task gets a child

- task with priority, assignee, and dependency links (`bd dep add`).

- - **Cross-session persistence** -- task state survives PM restarts. On session

- start, run `bd ready` for an instant cross-sprint view of what is in flight.

- - **Lifecycle hooks** -- the PM skill calls `bd` at every phase boundary: init,

- plan approval, task dispatch, verify checkpoint, reviewer findings, and cleanup.

-

- ## Common Commands

-

- | Command | What it does |

- |---------|-------------|

- | `bd ready` | Show all unblocked, open tasks across all sprints |

- | `bd list --all --pretty` | Full tree: all projects, members, tasks, status |

- | `bd create "<title>" -p <pri> --parent <epic-id> --assignee <member>` | Create a task |

- | `bd create "<title>" --id <explicit-id>` then `bd update <explicit-id> --parent <epic-id>` | Create a task with a pre-decided id under a parent -- `bd create` rejects a call that combines `--id` and `--parent` in the same invocation, so an explicit id and a parent link are always two separate calls, never one |

- | `bd show <id> --json` | Full detail for one task |

- | `bd update <id> --status in_progress --assignee <member>` | Mark a task in progress |

- | `bd close <id>` | Mark a task complete (idempotent) |

- | `bd reopen <id>` | Reopen a prematurely closed task |

- | `bd dep add <task-id> <blocks-id>` | Declare a dependency |

- | `bd search "<term>" --status all --json` | Search tasks; check before creating to avoid duplicates |

- | `bd note <id> "<text>"` | Attach a note (e.g. PR URL at sprint close) |

-

- Priority values: `0` = highest, `3` = lowest (backlog).

-

- ## PM Lifecycle Hooks

-

- The PM skill integrates Beads at these points in every sprint:

-

- | PM command | Beads action |

- |-----------|-------------|

- | `/pm init` | `bd init` (idempotent); create sprint epic; record epic ID in status.md |

- | `/pm plan` (after approval) | `bd create` one task per PLAN.md item; `bd dep add` for dependencies |

- | `/pm start` / task dispatch | `bd update <id> --status in_progress --assignee <member>` |

- | VERIFY checkpoint done | `bd close <id>` |

- | Reviewer CHANGES NEEDED | `bd create` a task per HIGH finding |

- | `/pm cleanup` | `bd close <epic-id>`; `bd note <epic-id> "PR: <url>"` |

-

- ## Backlog and Deferred Items

-

- ```bash

- # Defer an item at low priority

- bd create "<description>" -p 3 --parent <epic-id>

-

- # Show full tree including deferred items

- bd list --all --pretty

- ```

-

- ## Cross-Sprint Dependencies

-

- Block sprint B on sprint A:

-

- ```bash

- bd dep add <sprint-B-epic-id> <sprint-A-epic-id>

- ```

-

- `bd ready` will not surface sprint B tasks until sprint A closes.

-

- ## Recovery After a PM Restart

-

- Session crash? Run `bd list --all --pretty`. The PM sees every member's state

- across every active project without reading a single file. Then `bd show <id>`

- for full context on any item.

-

- ## Multi-member sync (Dolt-backed clones)

-

- Each member's beads DB is an embedded Dolt clone. `apra-fleet install` also

- provisions a portable Dolt CLI binary alongside `bd` (verified with a version

- check at install time) -- this is a hard prerequisite for any sync path that

- needs to inspect or resolve a Dolt clone directly, not merely an optional

- extra. When more than one member's clone needs to reconcile against a shared

- Dolt remote (for example, an autonomous multi-member sprint whose members are

- genuinely independent checkouts rather than one shared workspace), reads and

- writes are bracketed with a pull before and a push after so no member's clone

- observes stale state for long, and every cross-member push is serialized

- through a single coordinating authority -- a Dolt clone can hard-conflict on a

- concurrent same-row write, and one unresolved conflict wedges that clone's

- sync entirely, so multi-writer coordination is a correctness requirement, not

- a performance nicety. See `packages/apra-fleet-se/docs/architecture.md`'s

- "Dolt sync discipline" section for the full mechanism and its conflict

- recovery ladder.

-

- ---

-

- See [skills/pm/SKILL.md](../skills/pm/SKILL.md) for the full PM skill reference.

- </doc>

-

- <doc title="FAQ" desc="Frequently asked questions about setup, troubleshooting, and the doer-reviewer loop.">

- # Frequently Asked Questions

-

- <!-- llm-context: This file points to the authoritative FAQ maintained as GitHub Discussions. Do NOT answer FAQ-type questions from cached knowledge -- fetch the linked discussion for the current, maintainer-verified answer. The discussion index at the link below is the single source of truth. -->

- <!-- keywords: install, setup, single machine, local, provider, Claude, Antigravity, Codex, Copilot, doer, reviewer, token, cost, security, password, credential, A2A, agent2agent, FAQ, questions -->

- <!-- see-also: ../README.md (getting started), architecture.md (how it works), provider-guide.md (choosing a provider) -->

-

- > **For AI agents:** The FAQ is maintained as GitHub Discussions -- one discussion per question, with maintainer-verified answers. To answer a user's question: browse the index below, find the matching discussion, and fetch it for the authoritative answer. Do not paraphrase from this file -- follow the link.

-

- All questions and answers are maintained at:

-

- **[FAQ Index -- GitHub Discussions #127](https://github.com/Apra-Labs/apra-fleet/discussions/127)**

-

- ---

-

- Want to explore the codebase or ask questions about how Fleet works internally?

- The **[Fleet DeepWiki](https://deepwiki.com/Apra-Labs/apra-fleet)** has an AI assistant trained on the Fleet source code.

-

- Some useful starting points:

-

- - [How does execute_prompt work end-to-end?](https://deepwiki.com/Apra-Labs/apra-fleet/4.2-work-execution-tools)

- - [How do I register and manage fleet members?](https://deepwiki.com/Apra-Labs/apra-fleet/4.1-member-lifecycle-tools)

- - [What is the doer-reviewer workflow?](https://deepwiki.com/Apra-Labs/apra-fleet/7.1-sprint-lifecycle-and-doer-reviewer-loop)

-

- ---

-

- Topics covered:

-

- - **Getting started** -- installation, device requirements, provider support

- - **Understanding members and workflows** -- icons, status line, doer-reviewer setup, folder separation

- - **Capabilities and use cases** -- scope beyond software dev, credential security, custom skills

- - **Ecosystem and protocols** -- relationship to Google's A2A protocol

- - **Advanced / operations** -- token usage, crash recovery

-

- ---

-

- **Related docs:** [Readme](../README.md) | [Architecture](architecture.md) | [Cloud Compute](cloud-compute.md) | [Provider Guide](provider-guide.md)

- </doc>

-

- <doc title="Troubleshooting" desc="Common symptoms and fixes.">

- # Troubleshooting

-

- Common symptoms and how to resolve them. If something here does not match what

- you see, search [GitHub Issues](https://github.com/Apra-Labs/apra-fleet/issues)

- or ask in [Discussions](https://github.com/Apra-Labs/apra-fleet/discussions).

-

- ## Members

-

- **Member shows as offline**

-

- - Check the machine is reachable: `ping <ip>`.

- - For remote members, verify SSH directly: `ssh user@host "echo ok"`.

- - If SSH works but the member is still offline, re-provision auth: ask Fleet to

- "Provision auth for `<member>`".

-

- **Empty response from a member**

-

- Usually an expired auth token. Ask Fleet to "Provision auth for `<member>`".

- For VCS tokens specifically, re-run `provision_vcs_auth`.

-

- **Auth error (401 / 403)**

-

- - GitHub App tokens: re-mint with `provision_vcs_auth`.

- - Bitbucket / Azure DevOps: the token likely expired -- get a fresh one, then

- re-provision and retry. See the `auth-*.md` references in the fleet skill.

-

- **Member blew past a checkpoint**

-

- Check what actually happened on the member:

- ask Fleet to run `cat progress.json` on it.

-

- **Long-running background task fails to launch on a Windows member**

-

- `execute_command`'s `long_running` mode on Windows launches the task

- detached via `Invoke-CimMethod Win32_Process.Create` (WMI). If this fails,

- check that the WMI service (`Winmgmt`) is running on the member and that the

- fleet SSH user has permission to create processes via WMI -- `monitor_task`

- will otherwise report the task as immediately failed/missing rather than

- running.

-

- ## Permissions

-

- **Permission denied on a member**

-

- Fleet can configure member permissions. Ask it to, for example, "Grant

- `build-server` permission to run `npm install`". Under the hood this runs

- `compose_permissions`, which writes provider-native config:

-

- | Provider | Config location |

- |----------|-----------------|

- | Claude | `.claude/settings.local.json` |

- | Codex | `.codex/config.toml` (approval mode) |

- | Copilot | `.github/copilot/settings.local.json` |

-

- **Permission granted but still denied on Claude**

-

- Claude Code only honors `.claude/settings.local.json` permissions once the

- project folder is a **trusted workspace**. If the member's work folder has

- never been opened and trusted in Claude Code directly, the permissions Fleet

- writes there are inert. Open the folder in Claude Code once and accept the

- trust prompt, then retry.

-

- **A deploy step is blocked at the permission pre-check even though similar commands are allowed**

-

- Check the allowlist for the *exact* subcommand the deploy step invokes, not

- just the binary name -- an allowlist entry scoped to one subcommand (e.g.

- `Bash(<tool> start)`) does not cover a different subcommand of the same

- binary (e.g. `Bash(<tool> run ...)`) even though both start the same

- long-running process. Read the deploy runbook's own Permissions section for

- the full list of required command prefixes and diff it against

- `.claude/settings.json` / `.claude/settings.local.json` before assuming the

- underlying operation itself is unsafe or needs a workaround -- add the

- missing prefix and re-trigger.

-

- ## Timeouts

-

- A dispatch can end in two distinct ways:

-

- - **Inactivity timeout (`timeout_s`)** -- fires when no stdout/stderr output

- arrives for N seconds (default 300s / 5 min). It is transport-level, so it

- applies to every member and provider. The usual cause is a build or test

- runner that buffers output (`npm test`, `vitest`, `cargo build`) and stays

- silent for long stretches even while working. Fix: raise `timeout_s` to

- 600-1200 for build/test dispatches.

- - **Total timeout (`max_total_s`)** -- fires after N seconds of wall-clock time

- regardless of output. Use it as a hard ceiling on long jobs, alongside

- `timeout_s` when you want both a silence guard and a wall-clock cap.

-

- ## Credentials

-

- **A token or password appeared in command output**

-

- Store the secret with `credential_store_set`, then reference it as

- `{{secure.NAME}}` in `execute_command`. Fleet redacts it to `[REDACTED:NAME]`

- before the LLM ever sees the output. See

- [docs/features/oob-auth.md](features/oob-auth.md).

-

- **Rotate a credential without re-provisioning**

-

- Run `credential_store_delete name=<NAME>` then `credential_store_set

- name=<NAME>`. The new value is picked up immediately on the next

- `execute_command` that references `{{secure.NAME}}`.

-

- ## Git

-

- **Cannot push workflow files or merge PRs from a member**

-

- Minted VCS tokens may lack CI/CD permissions. Run those operations from your

- main AI coding session instead -- it has your full git credentials. See

- [docs/design-git-auth.md](design-git-auth.md).

-

- ## Build & native dependencies

-

- **`npm ci`/`npm install` fails compiling a native module (e.g. `better-sqlite3`) via `node-gyp`**

-

- This is a toolchain/environment incompatibility, not a code defect -- do not

- patch the deploy or build scripts to work around it. On macOS it shows up as

- Xcode Command Line Tools' `libc++` headers rejecting newer C++20 constructs

- (`concept`, `requires`, `output_iterator_tag`/`contiguous_iterator_tag`) that

- the Node header set expects, once the Node major version and the installed

- CLT/node-gyp versions drift out of the combination that was actually tested.

- Resolve by aligning the toolchain, not the source tree: update Xcode Command

- Line Tools (`xcode-select --install` / reinstall), or build against a Node

- version known to match the installed CLT, or update `node-gyp` itself. Check

- the npm debug log path printed in the failure output for the exact compiler

- error before assuming this is the cause.

-

- ## Stuck agents

-

- **A member is stuck after a session reset**

-

- Escalate the model tier (cheap -> standard -> premium) and retry. If it is still

- stuck, the task likely needs a human decision -- inspect `progress.json` and

- intervene directly.

-

- ## Logs

-

- For unexplained behavior -- missing output, silent failure, unexpected results

- -- check the server logs:

-

- ```

- $APRA_FLEET_DATA_DIR/logs/fleet-<pid>.log

- ```

-

- These are JSON lines. Filter by member or by tool with `jq`:

-

- ```bash

- jq 'select(.member_id == "<uuid>")' fleet-<pid>.log

- jq 'select(.tag == "<tool>")' fleet-<pid>.log

- ```

-

- The **Fleet Logs** section of the fleet skill's `SKILL.md` has the full field

- reference and more `jq` examples.

- </doc>

-

- <doc title="Roadmap" desc="What is planned next for Apra Fleet.">

- # Roadmap

-

- This roadmap is grounded in the repo's actual git history and issue tracker

- (beads), not aspiration. "Shipped" means it is on `main`. "In flight" means

- there is a recently-active branch with real commits behind it. The forward

- sections are a projection of where the current trajectory leads; priorities

- shift based on what the fleet itself surfaces while building this codebase.

-

- Have an idea? [Open a feature request](https://github.com/Apra-Labs/apra-fleet/issues/new/choose).

-

- ---

-

- ## What's shipped

-

- ### The fleet-sprint engine and the workflow platform (v0.4.0 line)

-

- - **`fleet-sprint`: an autonomous multi-agent sprint engine** -- plan ->

- develop -> review -> harvest cycles run by planner/doer/reviewer role

- agents against a real git repo, with beads (`bd`) as the task DAG and

- Dolt as the sync backend. Renamed from `auto-sprint` to end the

- confusion with Claude Code's unrelated `/auto-sprint` script; source

- lives in `packages/apra-fleet-se/fleet-sprint/`, and this repository is

- itself built by it (see the dashboard recording in README.md).

- - **`apra-fleet workflow <name>`: a general-purpose workflow runner** --

- the `packages/apra-fleet-workflow` engine gives workflows typed errors,

- budget enforcement, resumable/replayable runs, and cooperative `/stop`

- cancellation. fleet-sprint is the first workflow shipped on it.

- - **`apra-fleet-se`: an always-on multi-sprint supervisor (preview)** --

- runs several sprints concurrently against a shared fleet with a

- member + issue-scope reservation ledger, a PID-liveness watchdog with

- restart re-adoption, orchestrator-bracketed git+Dolt sync with a

- scripted-first conflict ladder, and a sprint-stack dashboard (running

- sprints, history, backlog tree) on one reverse-proxied port. Shipped in

- the big v0.4.0 merge (`732dc38d`) and hardened in #362; still gated

- "preview" because the full end-to-end supervisor smoke cycle has not yet

- passed cleanly.

- - **Sprint reliability hardening (#362 and the mac1 dogfood run)** --

- credential-auth self-heal via `provision_vcs_auth`, orphaned-CLI

- recovery, PID-liveness "lease of life" so a flaky channel no longer

- produces false empty-response failures, a stall detector that watches

- all transcript activity (not just chat) and kills confirmed stalls,

- and dispatch-vs-sync failure separation so a Dolt/git push hiccup no

- longer burns a full LLM re-dispatch (`apra-fleet-6z8.*`).

- - **MCP transport defaults to streamable HTTP** -- one long-lived

- `apra-fleet run` server on `localhost:7523/mcp` shared by every

- provider, replacing per-session stdio subprocesses; `--stdio` remains

- as an alias.

- - **apra-pm "comes home"** -- the `vendor/apra-pm` submodule is gone;

- apra-pm is a package-local dependency at `packages/apra-fleet-se/apra-pm`,

- which removed a whole class of silent submodule drift in CI/e2e/packaging.

- - **Deterministic e2e harness** -- fleet setup/teardown is a script, not

- an LLM improvisation (#353), checkpoint consolidation is deterministic

- (#360), and turn-budget/git-auth/log-collection flakes were fixed (#359).

-

- ### Providers and members (v0.3.x line)

-

- - **OpenCode provider** -- full adapter (NDJSON parseResponse, session

- management, permissions/auth), per-member `model_tiers` with

- dispatch-time resolution and validation against available models,

- GLM-4.5-Air premium default, e2e suites on GitHub-hosted runners, and

- `docs/opencode-getting-started.md`. This is the door to local/self-hosted

- OpenAI-compatible models.

- - **Antigravity (agy) provider maturation** -- `--agent` flag dispatch,

- session-resume fixes, safety rationalization (`docs/agy-safety-rationalization.md`).

- - **Role-agent file installation, including remote members** --

- `apra-fleet install` writes planner/doer/reviewer/plan-reviewer agent

- definitions into each provider's agents directory, and `update_member`

- provisions them to remote members too (#336).

- - **Member categories and tags** -- `category` plus up to 10 `tags` on

- register/update, tag-filtered `list_members`, and tag-driven

- `compose_permissions` profile merging (#314). This shipped what the old

- roadmap called "member groups".

- - **No-LLM members** -- `llm_provider: none` for machines that only run

- commands or host services (GPU nodes, relays).

- - **Live member activity viewer** -- `apra-fleet watch` streams what every

- member is doing (#319).

- - **CLI ergonomics** -- `install` is the default action, `run` starts the

- MCP server (v0.3.3); bare Claude model aliases instead of pinned dated

- IDs (v0.3.4); npm packaging and the SEA binary coexist, including the

- Windows `build:binary` fix.

-

- ### Shipped, then deliberately pulled back

-

- - **Knowledge bank v1 (#296)** -- kb_query/kb_harvest/kb_promote MCP

- tools, KB server, HTTP provider -- landed on `main` and was then

- reverted (#303). The capability is being re-approached through the

- code-intelligence abstraction (see in flight below) with eval evidence

- this time, rather than re-landed wholesale.

-

- ---

-

- ## Currently in flight

-

- These are the recently-active branches with substantive work behind them,

- described honestly.

-

- 1. **Supervisor viewer parity -- `feat/supervisor-viewer-parity`** (active

- today). Bringing the apra-fleet-se supervisor dashboard up to parity

- with the fleet-sprint viewer: server-side beads data for a Backlog

- tree, Sprints/Backlog tabs, server-side filtering, header filters,

- multi-select, workflow-UX language, plus Windows fixes (bd spawn via

- shell, POSIX-only agent-existence check in execute_prompt). Mid-to-late

- stage: the views exist and are being polished, not scaffolded.

-

- 2. **apra-fleet-se productization and packaging -- `kj/fix-apra-fleet-se`

- on top of `feat/sprint-service-1`** (active today). Making the

- supervisor service actually installable and runnable outside the dev

- tree: ESM bundle require-shim for CJS deps, shipping all runner

- siblings, dev-manifest fixes for the retired vendor path, plus

- supervisor test hardening (contention-aware scaled timeouts, live smoke

- retest evidence for the eft.5x-8x bug family, engine round-resume

- semantics). Late stage for packaging; the end-to-end supervisor smoke

- gate is the remaining exit criterion.

-

- 3. **Code intelligence / knowledge layer, round two --

- `feat/code-intelligence-abstraction`, `chore/merge-main-into-code-intel`,

- and the `test/kb-eval-a`/`test/kb-eval-b` A/B eval branches** (active

- this week). A pluggable code-intel provider abstraction

- (codebase-memory-mcp as default), a per-member `codeIntelProvider`

- field with per-member routing in tool dispatch, repo-scoped

- `kb_harvest` auto-harvest wired to execute_prompt completion, and

- paired eval sprints measuring whether the KB actually helps. Mid stage:

- core routing and harvest provenance are implemented and tested; the

- merge-back branch (`chore/merge-main-into-code-intel`) is reconciling

- it with current main.

-

- 4. **Live dogfood hardening channel -- `fleet-sprint/mac1`** (active

- today). Not a feature branch but a real fleet-sprint run on a macOS

- member whose output is a steady stream of engine fixes: non-ASCII

- sanitization in task descriptions, stall-poller timestamp scoping,

- failed-streak exclusion from review dispatch, cost usage field-name

- fix, POSIX-sh pipefail guards. This loop is now the primary way engine

- bugs get found.

-

- 5. **agy dynamic agent transform -- `feat/agy-agent-transform`** (last

- commits ~5 days ago). Transforms fleet agent definitions into

- Antigravity-native rules at dispatch time and fixes session resume

- (FLEET_SESSION_ID parsing, `--continue` fallback). Small and mid stage.

-

- ---

-

- ## Near-term (next few weeks)

-

- The near-term is dominated by finishing what v0.4.0 opened, not by new

- surface area.

-

- - **Pass the supervisor end-to-end smoke gate and drop the "preview"

- label.** The declared acceptance for the sprint-service epic -- a full

- plan-develop-review-harvest cycle through apra-fleet-se against a live

- sandbox -- has failed five attempted cycles; the root causes found so

- far (phantom interactive sessions, dead-session timeouts, duplicate

- doer commits on retry) are fixed but not yet proven together. This is

- the single most important open item in the repo.

- - **Cut the v0.4.0 release.** `docs/release-notes-v0.4.0.md` is drafted

- and `package.json` already says 0.4.0; landing supervisor-viewer-parity

- and the se packaging fixes is what stands between the draft and a tag.

- - **Merge supervisor viewer parity** so the supervisor dashboard is a

- strict superset of the fleet-sprint viewer, and make apra-fleet-se the

- single supported entry point for running sprints (the CLI stays as the

- low-level path).

- - **Land the code-intelligence abstraction with eval evidence.** The

- kb-eval A/B branches exist precisely to avoid a second #303-style

- revert: merge only what the paired sprints show is a measurable win,

- behind the per-member/per-repo provider routing that already has tests.

- - **Windows parity as a standing near-term theme.** The recent bug record

- (POSIX-only agent checks, bd ENOENT spawns, pipefail on non-bash shells,

- the silent Windows SEA build failure) says cross-platform drift is the

- most common regression class; expect continued small fixes plus

- regression guards rather than one big effort.

- - **Finish and merge the agy agent transform** so all providers get role

- agents through one transform pipeline instead of provider-specific hacks.

-

- ## Mid-term (1-3 months)

-

- Extrapolating from what the architecture is clearly reaching toward:

-

- - **Hub-spoke cloud mode becomes usable.** The groundwork is already in

- the tree -- `packages/fleet-api-contract`, `docs/hub-spoke-master-plan.md`,

- the wire-protocol and hub-service-deployment docs, and the identity

- model shipped in v0.4.0 -- but `apra-fleet join` / spoke mode is not

- end-to-end yet. With HTTP transport now the default and the supervisor

- a long-lived service, a hub that remote spokes attach to is the natural

- next step, and it is the prerequisite for any hosted offering.

- - **Dashboard auth and RBAC.** `docs/dashboard-oauth-rbac-design.md`

- exists for a reason: the moment the supervisor dashboard is the front

- door to a shared, always-on service (and especially a hub), it needs

- login and roles. Expect this to ride immediately behind hub-spoke.

- - **A second real workflow on apra-fleet-workflow.** The engine was

- explicitly built for more than fleet-sprint (`docs/authoring-workflows.md`,

- the workflow-core boundary refactoring doc). The credible proof that

- "any workflow" is real is a shipped second workflow -- likely something

- operational (release playbook automation or an e2e/integration runner)

- since those already exist as semi-manual scripts in this repo.

- - **Knowledge layer graduates from experiment to default.** If the

- kb-eval results hold up, the trajectory is: code-intel provider on by

- default for sprint members, repo-scoped harvest on session close, and

- then the central/team KB server re-landed from the reverted #296 work --

- in that order, each behind evidence.

- - **Cost governance surfaces in the dashboard.** The pieces exist

- (`docs/cost-model.md`, `get_member_model_pricing`, per-turn cost

- calculation that just got a field-name fix on mac1); the missing piece

- is per-sprint/per-member cost rollups where operators actually look --

- the supervisor dashboard.

- - **Sprint calibration data starts steering sprints.** The engine already

- writes `sprint calibration` and `sprint-analysis` commits every cycle;

- the obvious next step is feeding that history back into planning

- (cycle-count estimates, model-tier selection, stall-timeout tuning)

- instead of only recording it.

-

- ## Long-term (3-12+ months)

-

- - **Hosted fleet / fleet-as-a-service on the hub-spoke foundation.** Once

- spoke mode and dashboard RBAC exist, a managed hub is mostly an ops

- problem, not an architecture problem. Multi-fleet federation

- (hub-of-hubs) is the step after, and should stay speculative until at

- least two independent hubs exist in practice.

- - **Workflows beyond software engineering.** The README already markets

- "any domain" (retail replenishment, logistics exceptions, intake

- triage); making that true requires the mid-term items first: a proven

- second workflow, workflow authoring docs that outsiders can follow, and

- an extension/distribution story for workflows that are not baked into

- the repo.

- - **Enterprise governance: audit trail and policy.** The security

- substrate is unusually strong already (OOB secrets, per-provider

- permission composition, the permission-block-surfacing convention); the

- missing enterprise piece is an immutable audit log of fleet operations

- and secret usage, which becomes mandatory the moment a hosted hub has

- more than one tenant.

- - **Close the self-development loop fully.** The endgame the dogfooding

- is pointing at: the supervisor runs continuously against this repo,

- fleet-sprint files, fixes, reviews, and ships its own bugs with the

- human as reviewer-of-last-resort, and the calibration/KB layers make

- each sprint measurably cheaper than the last. Everything above --

- supervisor smoke gate, knowledge layer, cost rollups, calibration

- feedback -- is a component of that loop.

-

- ### Deliberately not on the roadmap (revisited from the previous version)

-

- Items the old roadmap carried that the evidence does not currently

- support prioritizing: gbrain integration (superseded by the

- code-intelligence abstraction), Playbooks as a separate feature (largely

- subsumed by the workflow engine), Slack notifications and a Terraform

- provider (no recent activity or demand signal in the tracker). They can

- return if demand shows up.

-

- ---

-

- ## Contributing

-

- Pick an item above, open an issue to discuss your approach, then submit a

- PR. See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

- </doc>

-

- <doc title="Contributing" desc="How to contribute code, docs, and skills.">

- # Contributing to apra-fleet

-

- Thank you for your interest in contributing! This document explains how to get involved.

-

- ## Reporting Bugs

-

- Use the [Bug Report](https://github.com/Apra-Labs/apra-fleet/issues/new/choose) issue template on GitHub. Include as much detail as possible -- reproduction steps, environment info, and error output are especially helpful.

-

- ## Requesting Features

-

- Use the [Feature Request](https://github.com/Apra-Labs/apra-fleet/issues/new/choose) issue template. Describe the problem you're trying to solve, your proposed solution, and any alternatives you've considered.

-

- ## Development Setup

-

- **Prerequisites:** Node.js 20+, npm

-

- ```bash

- git clone https://github.com/Apra-Labs/apra-fleet.git

- cd apra-fleet

- npm install

- npm run build

- ```

-

- `npm install` auto-installs the git pre-commit hook via the `prepare` script. To install manually, run `node scripts/install-hooks.mjs`. The hook lives at `.github/hooks/pre-commit`.

-

- ## Running Tests

-

- ```bash

- npm test

- ```

-

- For watch mode during development:

-

- ```bash

- npm run test:watch

- ```

-

- ## Branch Naming

-

- | Type | Pattern | Example |

- |------|---------|---------|

- | Feature | `feature/<short-description>` | `feature/ec2-support` |

- | Bug fix | `fix/<short-description>` | `fix/ssh-timeout` |

- | Docs | `docs/<short-description>` | `docs/contributing-guide` |

-

- Always branch from `main`.

-

- ## Commit Message Convention

-

- Use the [Conventional Commits](https://www.conventionalcommits.org/) format:

-

- ```

- <type>(<scope>): <short summary>

- ```

-

- Common types: `feat`, `fix`, `docs`, `chore`, `refactor`, `test`

-

- Examples:

- - `feat(members): add EC2 instance support`

- - `fix(ssh): handle connection timeout gracefully`

- - `docs: update contributing guide`

-

- ## Pull Request Process

-

- 1. Fork the repo and create your branch from `main`.

- 2. Make your changes, following the code style notes below.

- 3. Run `npm run build` and `npm test` -- both must pass.

- 4. Open a PR against `main` using the PR template.

- 5. A maintainer will review your PR. Address any feedback.

- 6. Once approved, a maintainer will merge it.

-

- ## Code Style

-

- - **Language:** TypeScript. Match the style of surrounding code.

- - **Formatting:** No enforced formatter currently -- keep indentation and style consistent with existing files.

- - **No unnecessary abstractions:** Prefer simple, direct code over premature generalization.

- - **Error handling:** Only handle errors at real system boundaries (user input, SSH, external APIs). Don't add fallbacks for scenarios that can't happen.

- - **ASCII only:** No non-ASCII characters in committed files. Use `--` for em-dashes, `->` for arrows, `[OK]` for checkmarks.

-

- ## For AI Agents

-

- If you are an AI agent (or a human using an AI agent) contributing to this project, this section covers the patterns and conventions that matter most.

-

- ### Dev-mode install

-

- Build and install from source without touching the packaged binary:

-

- ```bash

- npm run build && node dist/index.js install

- ```

-

- This registers the MCP server from your local `dist/` build. Skill files are read from `skills/` on disk -- no rebuild needed to iterate on them.

-

- ### File map

-

- | Path | What it contains |

- |------|-----------------|

- | `src/` | TypeScript source for the MCP server, CLI commands, and providers |

- | `skills/fleet/` | Fleet skill -- tools for managing members, tasks, and files |

- | `skills/pm/` | PM skill -- orchestration patterns, doer-reviewer loop, deploy flows |

- | `hooks/` | Shell hooks that run on Claude Code events (statusline, pre-push, etc.) |

- | `CLAUDE.md` | Role-specific instructions (not committed -- each agent has its own) |

- | `AGENTS.md` | Shared project context for all agents |

-

- ### Testing skill changes

-

- Skills are Markdown files -- edits take effect immediately without a rebuild. After editing `skills/fleet/` or `skills/pm/`: