harness-engineering-playbook · diff
git:20260628.9edbf07 to git:20260628.aa6a326
116 added, 35 removed. Audit A to A.
---
name: harness-engineering-playbook
category: governance
- description: Implement OpenAI Harness Engineering practices in any repository — AGENTS.md, PLANS.md, deterministic smoke/test/lint harness commands, strict architecture boundaries, observability from day 1, and entropy-control audits for reliable autonomous agent runs.
+ description: Implement OpenAI Harness Engineering practices in any repository. Use when setting up or refactoring agent-first workflows, writing or upgrading AGENTS.md and PLANS.md, creating deterministic smoke/test/lint/typecheck harness commands, defining strict architecture boundaries and data-shape contracts, wiring observability from day 1, and adding entropy-control checks plus CI automation for reliable autonomous runs.
---
# Harness Engineering Playbook
- A skills.sh-compatible skill that operationalizes the practices from OpenAI's Harness Engineering guide. Use it to set up or refactor agent-first workflows so that autonomous runs are repeatable, observable, and safe.
+ Use this skill to operationalize the practices from OpenAI's Harness Engineering guide in a repo that agents can run against repeatedly and safely.
- ## Install
+ ## What To Load
- ```bash
- npx skills add broomva/skills --skill harness-engineering-playbook
- ```
+ - Use `references/openai-harness-practices.md` for the full practice-to-artifact mapping.
+ - Use `references/rollout-checklist.md` for phased adoption in active repos.
+ - Use `assets/templates/` when creating or updating harness files.
- ## What It Does
+ ## Inputs
- - Bootstraps harness artifacts: `AGENTS.md`, `PLANS.md`, `docs/ARCHITECTURE.md`, `docs/OBSERVABILITY.md`, `Makefile.harness`, and CI workflows.
- - Wraps deterministic commands behind `make smoke`, `make check`, `make ci` so agents can run them reliably.
- - Enforces strict module boundaries and data-shape contracts.
- - Wires structured observability (correlation IDs, key transitions) from day 1.
- - Adds entropy-control audits and nightly harness checks to prevent docs drift and flaky scripts.
+ - Target repository path.
+ - Existing command surface (`make`, `npm`, `cargo`, `pytest`, etc.).
+ - Existing CI workflows and branch protections.
## Workflow
- 1. **Baseline** the target repo — detect language, toolchain, and existing CI.
- 2. **Bootstrap** harness artifacts from templates (interactive wizard or shell script).
- 3. **Apply** the nine Harness Engineering practices across repo artifacts.
- 4. **Validate** with `audit` — treat any `MISSING` or `FAIL` as blocking.
- 5. **Iterate** after real agent runs — patch gaps and re-audit.
+ 1. Baseline the repo and detect existing workflows.
+ 2. Bootstrap harness artifacts and templates.
+ 3. Apply all nine Harness Engineering practices.
+ 4. Run harness audit checks and repair gaps.
+ 5. Iterate after real agent runs.
- ## Quick Start
+ ## Step 1: Baseline The Repo
- ```bash
- # Interactive wizard (recommended)
- python3 .agents/skills/harness-engineering-playbook/scripts/harness_wizard.py init <repo-path> --profile control
+ - Identify language/toolchain and canonical entrypoints.
+ - Inventory existing checks, scripts, and CI jobs.
+ - Record current pain points for agent runs: setup drift, unclear docs, flaky tests, missing trace IDs, slow loops.
- # Shell fallback
- ./scripts/bootstrap_harness.sh <repo-path>
+ Use a short baseline note inside `PLANS.md` so decisions remain durable.
- # Audit
- python3 .agents/skills/harness-engineering-playbook/scripts/harness_wizard.py audit <repo-path>
+ ## Step 2: Bootstrap Harness Artifacts
+
+ Run:
+
+ ```bash
+ ./scripts/bootstrap_harness.sh <repo-path>
```
- ## Profiles
+ This script installs safe defaults from `assets/templates/`:
- | Profile | Scope |
- |------------|----------------------------------------------------|
- | `baseline` | Core harness artifacts only |
- | `control` | Baseline + control-system primitives |
- | `full` | Control + entropy controls, nightly audit, CI |
+ - `AGENTS.md`
+ - `PLANS.md`
+ - `docs/ARCHITECTURE.md`
+ - `docs/OBSERVABILITY.md`
+ - `Makefile.harness` (+ `-include Makefile.harness` in `Makefile`)
+ - `scripts/harness/{smoke,test,lint,typecheck}.sh`
+ - `.github/workflows/harness.yml`
- ## Source
+ By default, existing files are not overwritten. Pass `--force` to replace template-managed files.
- OpenAI Harness Engineering guide: <https://openai.com/index/harness-engineering/>
+ ## Step 3: Apply The Nine Practices
- ## License
+ Implement each practice directly in repo artifacts.
- MIT
+ ### 1. Make Easy To Do Hard Thing
+
+ - Ensure hard, high-value tasks are one command away (`make smoke`, `make check`, `make ci`).
+ - Keep setup and cleanup scripted.
+ - Make smoke checks cheap enough for frequent use.
+
+ ### 2. Communicate Actionable Constraints With Compact Docs
+
+ - Keep `AGENTS.md` short, concrete, and command-first.
+ - Document non-obvious constraints and guardrails.
+ - Keep docs close to code and update with behavior changes.
+
+ ### 3. Structure Codebase With Strict Boundaries And Flow
+
+ - Define module boundaries in `docs/ARCHITECTURE.md`.
+ - Parse and validate data at boundaries; use typed contracts for internal flow.
+ - Prefer one abstraction per module and one clear ownership path.
+
+ ### 4. Build Observability In From Day 1
+
+ - Emit structured logs/events with correlation IDs.
+ - Capture key transitions in long-running workflows.
+ - Define minimum observable fields in `docs/OBSERVABILITY.md`.
+
+ ### 5. Optimize For Agent Flow, Not Human Flow
+
+ - Treat context as a first-class system dependency.
+ - Use `PLANS.md` for multi-step/multi-hour tasks.
+ - Front-load durable context (scope, constraints, checkpoints) so restarts stay cheap.
+
+ ### 6. Bring Your Own Harness
+
+ - Standardize repo-local wrappers (`Makefile.harness`, `scripts/harness/`).
+ - Wrap local infra actions in deterministic scripts.
+ - Make agent behavior reproducible across machines and runs.
+
+ ### 7. Prototype In Natural Language First
+
+ - Draft logic and tests in prose before coding.
+ - Review edge cases in prose and lock acceptance criteria.
+ - Translate approved prose into code and tests.
+
+ ### 8. Invest In Static Analysis And Linting
+
+ - Pin formatter/linter/typechecker versions where practical.
+ - Enforce checks in both local workflow and CI.
+ - Run static checks before long tests to shorten failure loops.
+
+ ### 9. Manage Entropy
+
+ - Add periodic audits for docs drift, flaky checks, and dead scripts.
+ - Keep templates synchronized with real workflows.
+ - Remove stale abstractions quickly to keep agent context clean.
+
+ For a detailed artifact matrix, load `references/openai-harness-practices.md`.
+
+ ## Step 4: Validate
+
+ Run:
+
+ ```bash
+ ./scripts/audit_harness.sh <repo-path>
+ ```
+
+ Treat any `MISSING` or `FAIL` result as blocking before calling harness setup complete.
+
+ ## Step 5: Iterate On Real Runs
+
+ - Observe one full agent run from clean checkout to merged change.
+ - Patch harness gaps immediately.
+ - Re-run audit.
+ - Keep `AGENTS.md`, `PLANS.md`, and architecture docs aligned with current behavior.
+
+ ## Adaptation Rules
+
+ - Preserve existing project conventions and replace templates incrementally.
+ - Do not overwrite user-authored files without explicit approval.
+ - Keep command names stable; change internals behind wrappers.
+ - Favor deterministic, scriptable workflows over ad-hoc interactive steps.