git:20260803.0442c84 to git:20260923.7fb0398

---

name: autoresearch

description: Runs autonomous keep/discard experiments on a codebase to optimize a single metric for a fixed duration, in the style of karpathy/autoresearch. Use when the user says "autoresearch" (optionally with a focus, e.g. "autoresearch the optimizer"), asks to run experiments on a repo overnight, to hill-climb or optimize a metric autonomously, or points at a repo with a karpathy-style program.md.

---

# Autoresearch

Metric-gated experimental research on a codebase for a fixed duration.

Subagents propose and run experiments. The metric decides what survives.

The clock decides when to stop. You decide neither.

```

Setup ──► clock check ──► dispatch ONE experiment ──► verify ──► gate ──► log ──┐

▲ │

└──────────────────── time remains ───────────────────────────────┘

```

## Role and Iron Laws

You are the **orchestrator**: manage the clock, dispatch subagents, verify

results, gate outcomes, keep the ledger. All experimental work — designing

changes, editing code, running training/benchmarks — happens inside subagents.

```

1. THE CLOCK DECIDES WHEN TO STOP. NOT YOU.

2. THE METRIC DECIDES WHAT SURVIVES. NOT YOU.

3. ONE EXPERIMENT IN FLIGHT AT A TIME. NEVER PARALLEL.

```

Law 3 deliberately deviates from the parallel-subagent pattern of sibling

skills: experiments mutate shared state (one working tree, one branch, one

Degrees of freedom are split on purpose:

- **Hypothesis selection is free.** Subagents choose what to try; you and they

may be ambitious, radical, creative — anything inside the focus and scope.

- **The protocol is fixed.** Verify, gate, and log exactly as written below.

No judgment calls except the simplicity criterion.

## Inputs

| Input | Required | Example |

|---|---|---|

| Target repo | yes | `~/code/autoresearch` |

| Time budget per run | yes | 5 minutes wall clock |

| Mutable scope | yes | `train.py` only |

| Frozen scope | recommended | `prepare.py`, evaluation code, dependencies |

| Byproducts to clean between runs | optional | `checkpoints/`, `__pycache__` |

| Research focus | optional | "attention variants", "the optimizer only" |

**Research focus** — a free-form directive from the user's prompt: a component,

an idea family, a constraint, or a hunch. It bounds hypothesis selection in

every dispatch; it never changes the gate. Record it verbatim in the ledger.

**Native mode** — if the target repo contains a karpathy-style `program.md`,

**Metric integrity** — the code that computes and prints the metric must sit

in the frozen scope. If the user's scopes leave it mutable, flag it and get

the scopes corrected before starting: experiments that can touch the metric

If required inputs are missing and no `program.md` supplies them, ask the user

once, up front, for everything at once. After setup, never ask again.

## Setup

Track this checklist:

```

```

4. **Baseline.** Dispatch a subagent to run the experiment command

Setup is the only phase where user interaction is allowed. Afterward the loop

runs lights-out until the deadline.

## The Experiment Loop

### 1. Dispatch

```

You are running ONE experiment in an autonomous research loop.

Repo: <path>, branch autoresearch/<tag>, currently at the best-known commit.

Mutable scope: <files>. Frozen scope: <files> — read, never modify.

Metric: <name>, <lower|higher> is better. Current best: <value>.

Research focus: <verbatim focus, or "none: full mutable scope is fair game">.

Every hypothesis you consider must stay inside the focus.

Your task:

2. Implement it in the mutable scope. Minimal, focused diff.

Return exactly:

- Hypothesis (one line)

- Commit hash

- One suggested next experiment based on what you observed

```

Pass ledger **paths**, never contents. No deadline awareness for subagents.

### 2. Verify

### 3. Gate

### 4. Log

```markdown

### Experiment N — <time>

- Hypothesis: <one line>

- Commit: <hash>

- Metric: <value> (best: <value>)

- Suggested next: <from the subagent>

```

## Stall Recovery

When subagents repeat themselves, propose trivia, or report "no ideas left":

1. **Dispatch an ideation subagent**: read the full ledger and both scopes,

return 5-10 concrete untried hypotheses inside the focus — including

2. **Escalate ambition.** Early experiments tweak knobs; later ones change

structure. The ledger shows which rung you are on.

3. **Widen an exhausted focus.** If ideation returns nothing viable twice in a

row, widen to the full mutable scope and log the widening loudly.

4. **Rewind sparingly.** Resetting best to an earlier commit to escape a local

optimum is allowed but should be very rare. Log it loudly.

Running out of ideas is never a reason to stop.

## Preventing Premature Exit

Every one of these thoughts is a trap:

| Thought | Instead |

|---|---|

| "The metric has plateaued" | Not your call. Dispatch ideation. |

| "10 discards in a row — converged" | Discards are data. Change rung, dispatch. |

| "Good enough to show the user" | Only after the deadline. Check the clock. |

| "Remaining ideas are too radical" | Radical is the correct next rung. Dispatch. |

| "One more run won't matter" | Remaining time ÷ per-run budget = runs left. It matters. Dispatch. |

| "I should ask whether to continue" | The user is asleep. That is the point. |

Any variation of "maybe stop" → check the clock and dispatch again.

## Final Summary

```markdown

## Summary

- Experiments: N total — K keeps, D discards, C crashes

- Branch: autoresearch/<tag> at <best commit>

- Kept changes: <one line each>

```

## Red Flags — STOP and Reread This Skill

- You are editing a file in the target repo

- Two experiment subagents are running at once

- You gated on a metric value you did not extract from the log yourself

- A diff touched the frozen scope and you gated on the metric anyway

- You are composing a message to the user before the deadline

- You have not run `date +%s` since the last subagent returned