run-eval-harbor ยท diff
git:20260916.52756f6 to git:20261010.ce857cb
64 added, 23 removed. Audit A to A.
What this update changes in the checks
No check changed: every fixed check gives the same answer before and after this update.
A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.
---
name: run-eval-harbor
- description: 'Run and diagnose existing Harbor evaluations against local production LobeHub or LobeHub Cloud. Use for eval infrastructure, target preflight, lh CLI injection, Harbor smoke or job runs, resume, and failure triage. Excludes authoring Harbor tasks and product acceptance.'
+ description: 'Run and diagnose existing Harbor or FrontierHarness evaluations against local production LobeHub or LobeHub Cloud, on a local Docker host, SSH worker, or Runta. Use for eval infrastructure, target preflight, lh CLI injection, Harbor smoke or job runs, resume, and failure triage. Excludes authoring Harbor tasks and product acceptance.'
---
# Run Eval Harbor
Run existing Harbor evaluations against either this checkout's isolated local
production harness or a remote LobeHub target. Use `create-task` to author or
grade tasks and `acceptance` for product acceptance.
## Ask First
- Before preparing or running anything, obtain these independent choices:
+ Before preparing or running anything, obtain these independent choices. Do not
+ use `runta` as a LobeHub server target or `checkout` as an execution environment:
- 1. Target: `local` production server from this checkout, or `cloud`/remote.
- 2. CLI: `checkout` build from `apps/cli`, or published `npm` release.
- 3. Exact `LH_AGENT_ID`; the selected agent already owns its model.
- 4. Eval repository path and whether the user wants a new job or a resume.
- 5. Credentials: for cloud, require its CLI API key in the eval repository's
- ignored `.env`. For local, ask whether the selected agent's provider
- credential is already stored in LobeHub; if not, ask for the provider's real
- environment variable name and secret before starting the server.
- 6. For local, obtain explicit confirmation that port `3210` and every configured
- eval infrastructure port are unreachable from the public internet and other
- untrusted networks. Do not bootstrap the local stack without confirmation.
+ 1. LobeHub server: `local` production server from this checkout, or
+ `cloud`/remote.
+ 2. Execution environment: this local host, an SSH Docker worker, or Runta runtimes. The maintained
+ Runta workflow currently targets a cloud/remote LobeHub server.
+ 3. CLI source: `checkout` build from `apps/cli`, or published `npm` release.
+ 4. Agent selection: cloud requires an exact `LH_AGENT_ID`; local uses the seeded
+ user's builtin `inbox` agent unless the user explicitly selects a different
+ agent.
+ 5. Eval repository path and whether the user wants a new job or a resume.
+ 6. Credentials: for cloud, require its CLI API key in the eval repository's
+ ignored `.env`. Local bootstrap seeds its CLI key. Reuse a provider
+ credential already configured in LobeHub or the ignored local env; if the
+ smoke reports `InvalidProviderAPIKey`, report the resolved agent's
+ provider/model and ask only for that provider's real credential.
+ 7. For a local LobeHub server, obtain explicit confirmation that port `3210` and
+ every configured eval infrastructure port are unreachable from the public
+ internet and other untrusted networks. Do not bootstrap the local stack
+ without confirmation.
+ For the Runta execution environment, also collect the harness repository and
+ pinned commit, provider/key choice required by FrontierHarness, and the
+ Terminal-Bench versus DeepSWE task subset. The Lh Cloud agent's model remains
+ selected by `LH_AGENT_ID`; do not replace it with FrontierHarness's `--model`
+ unless the run is explicitly non-comparable.
+
Do not infer these choices. DeepSeek is only one provider example, not a
required credential or model.
## Guardrails
- In local mode, run LobeHub on port `3210` with `bun run start`; never target a
dev server or port `3010`. Compose owns infrastructure; LobeHub stays on the
host.
- In cloud mode, never start local infrastructure or rewrite server/gateway
addresses. Official Cloud should use the CLI's default addresses.
- Create `docker-compose/eval/.env` from `.env.example` only when absent; never
overwrite an existing file.
- The local Compose stack publishes host ports and uses fixed development
credentials, including the seeded CLI key and gateway service token. Never
run it on a host where those ports are reachable by an untrusted network.
- - Never infer `inbox`, choose a separate model, or override the agent with
- `DEFAULT_AGENT_CONFIG`. Never invent, print, or commit secrets.
+ - Never infer `inbox` for a cloud server, choose a separate model, or override
+ the agent with `DEFAULT_AGENT_CONFIG`. A local server may resolve the isolated
+ seeded user's builtin `inbox` through the authenticated CLI. Never invent,
+ print, or commit secrets.
- LobeHub uses localhost service URLs. Harbor containers use Docker-reachable
host URLs. Never interchange them.
- Preflight is target-specific and read-only: local checks the local production
stack; cloud checks the remote server/gateways. It does not validate API keys,
agents, provider credentials, or model access.
- Do not run a model-backed Harbor job without an explicit user request.
- Before every requested real job or resume, run the shared model-backed smoke
- for the chosen target and CLI mode. Stop if either preflight or smoke fails.
+ for the chosen server target and CLI source. Stop if either preflight or smoke
+ fails.
## Run
- Read exactly one route after the answers above:
+ Read exactly one reference for the selected LobeHub server:
- - Local target: [references/local.md](references/local.md)
- - Cloud/remote target: [references/cloud.md](references/cloud.md)
+ - Local server: [references/local.md](references/local.md)
+ - Cloud/remote server: [references/cloud.md](references/cloud.md)
- The CLI selection is orthogonal to the target. `checkout` injects the built
- `apps/cli`; `npm` installs the release package. Both routes run their preflight
- and then `scripts/run-smoke.sh <local|cloud> <checkout|npm> ...` before the
- external eval repository's own job command.
+ When the execution environment is Runta, additionally read the
+ [Runta/FrontierHarness playbook](references/runta.md). It overlays the cloud
+ server route; it is not a third server target.
+
+ For FrontierHarness on a local Docker host or SSH worker, use
+ `scripts/frontierharness-local.py start` to launch Harbor/Pier queues and
+ `collect` to prepare results and usage for reporting. See each subcommand's `--help`.
+
+ CLI source is orthogonal to the server target and execution environment:
+ `checkout` injects the built `apps/cli`; `npm` installs the release package. For
+ local execution, run the selected server preflight and then
+ `scripts/run-smoke.sh <local|cloud> <checkout|npm> ...` before the external eval
+ repository's own job command. The maintained Runta workflow uses a checkout CLI
+ artifact, two restored suite runtimes, and a detached start/status/collect
+ workflow; do not substitute a normal Harbor job for it.
+
+ Harbor and Pier default to `LH_RUN_MODE=agent`. Set `LH_RUN_MODE=task` to create
+ one persistent LobeHub Task per trial and run it on the connected eval device;
+ operation polling, output collection, and device cleanup remain the same. Task
+ mode records usage with `lh task usage`; agent mode keeps the existing usage
+ evidence flow scoped to the operation.
+
+ The runner exports persisted topic interactions to `agent/trajectory.json` with
+ `lh topic export` during cleanup, including interrupted runs. This requires a
+ CLI/server with ATIF export support. Export currently supports single-agent
+ topics without threads or branches; failures are logged without changing the
+ trial result. The trajectory includes recorded reasoning, tool calls/results,
+ and message usage; it does not reconstruct runtime-only model context.
## Diagnose
- PostgreSQL, Redis, RustFS, QStash, or Compose state: local eval infrastructure.
- Port `3210`, migrations, API-key auth, or `/api/version`: LobeHub.
- Ports `8787`/`8788`, gateway health, or service tokens: gateway.
- Docker-only connectivity: bridge address, published port, or host firewall.
- CLI upload/install: `LH_CLI_SOURCE` or `apps/cli/dist`.
- Reward/verifier behavior: the Harbor task; use `create-task` before changing it.
## Harbor Reference
For Harbor commands beyond these scripts, consult the official
[Harbor Skills](https://github.com/harbor-framework/skills), especially its
`harbor-cli` skill. This harness pins `harbor==0.23.0`; when guidance differs,
the pinned CLI's `--help` is authoritative.