git:20260828.711be31 to git:20260829.0613d3f

---

name: nobrainer-autoimprove

description: "Use when the owner says nb-autoimprove, deep-autoresearch, code-autoresearch, nobrainer-capture-lesson, or nobrainer-continuous-improvement; asks to improve a skill or prompt; or wants a measurable bounded baseline-variant-evaluation loop instead of one subjective rewrite."

---

# NoBrainer Autoimprove

Improve one artifact only when a frozen evaluation proves the change is better.

This skill is explicitly inspired by Andrej Karpathy's

[autoresearch](https://github.com/karpathy/autoresearch) experiment loop and

adapts its measure-change-keep-or-revert idea to skills, prompts, instructions,

checklists and other reviewable artifacts. It is an independent adaptation, not

an official Karpathy project.

## When not to loop

Fix an obvious typo, dead link or deterministic defect directly. Use the

project's normal test-driven workflow for code with a strong existing harness.

Do not start when success cannot be measured, the target cannot be isolated, or

the proposed mutation touches credentials, production, safety controls,

external publishing or irreversible data without an owner-approved experiment.

## From feedback to durable improvement

Treat an owner correction as evidence, not automatic permission to rewrite the

system. Classify it first:

- a task-local clarification changes only the current task;

- an explicit durable preference, decision or verified fact may be captured

through `nobrainer-wiki` mode `ADD` only when its scope, confidentiality and

configured `LEARNING_WRITE_POLICY` permit it;

- a repeatable behavior gap becomes a regression scenario for this skill;

- a deterministic defect is fixed directly and covered by a test.

When `AGENT_ERROR_CORRECTED` is raised by `nobrainer-ultra`, preserve the exact

failing scenario and error fingerprint before editing. Correct the active task

first. Then classify the smallest reusable prevention candidate and honor the

project's configured policy before any durable learning write:

- `AUTO_SCOPED`: promote at most one minimal, sourced rule to its one canonical

governed project-local store;

- `ASK`: prepare one exact single-store diff and request the persistence

decision without applying it;

- `OFF`: keep the candidate task-local and do not create a durable diff or

modify `AGENTS.md`, `tasks/lessons.md` or a wiki.

Use this experiment when behavior needs generalization beyond the deterministic

active-task correction. The experiment's own target and write scope still need

the normal authorization; learning policy does not grant an unrelated edit.

For a behavior change, preserve the failing example, define the expected

response, and run the bounded experiment below. One anecdote does not justify a

broad inferred user trait or a global instruction rewrite. Record what was

learned, its source, scope and rollback so personalization remains visible and

correctable.

## Experiment contract

Freeze before changing the target:

```text

TARGET: <one artifact and allowed write scope>

OBJECTIVE: <one observable improvement>

BASELINE_VERSION: <commit/hash/copy>

DEVELOPMENT_EVAL: <representative cases used to iterate>

HOLDOUT_EVAL: <unseen cases used only for promotion>

HARD_GATES: <format, safety, forbidden values, links, deterministic tests>

SCORE: <criteria, weights, aggregation, direction>

BUDGET: <rounds, variants, time/tokens/cost>

PLATEAU: <minimum meaningful delta and dry rounds>

ROLLBACK: <restore procedure and readback>

OWNER_GATES: <actions not authorized by the loop>

```

Preserve the original target and run in an isolated copy, branch or worktree.

One coordinator owns the champion and log; generators must not race on the same

file.

For production code, freeze the harness and metric before the first candidate.

Prefer one behavioral change per round, preserve a control/baseline path, and

record the exact commit, test data and environment so a gain can be reproduced.

The loop may optimize code only inside an approved branch and write scope; it

does not deploy, tune on production feedback silently or weaken protected tests.

## Build a trustworthy eval

Use deterministic checks wherever possible. For subjective quality define a

concrete rubric whose criteria map to the target workflow. For a skill this

usually includes trigger precision, actionability, correctness, failure/stop

coverage, compatibility and concision.

Freeze two scenario sets:

- development cases exercise common, edge, adversarial and non-trigger inputs;

- holdout cases detect overfitting and are not shown to generators or used to

select an iteration.

Score the unchanged baseline on development only. Evaluate the sealed baseline

and selected champion against holdout cases together, exactly once, during

`FINAL_HOLDOUT`. Do not reveal holdout cases, criterion results or failure clues

to generators before that check. If the champion fails, create a new holdout,

re-baseline and start a new bounded experiment instead of tuning against the

failed holdout. If a rubric, weights or cases change, invalidate prior scores

and re-baseline. Never tune against one anecdote and call the artifact generally

improved.

For LLM judgment use independent judges, anonymized/shuffled candidates,

per-criterion reasons, median aggregation and pairwise tie-breaks. Judges do not

generate candidates and do not know which candidate is incumbent. Record model

or capability substitutions because judge drift affects comparability.

## Bounded loop

```text

FRAME -> BASELINE -> GENERATE -> HARD_GATE -> DEV_SCORE -> SELECT -> GRAFT

-> REPEAT_OR_STOP -> FINAL_HOLDOUT -> PROMOTE_OR_REVERT

```

### Generate

Produce three to six complete candidates with different hypotheses, not

paraphrases. Useful strategies include restructure, compress, expand missing

failure coverage, sharpen routing, add a concrete example, remove duplication,

or invert the flow around stop conditions.

### Hard gate and score

Run format, security, public-clean, deterministic tests and repository-specific

validators before spending judge budget. A hard-gate failure is disqualified,

not averaged away.

Score surviving candidates on the unchanged development eval. Select the best

median score only when it exceeds the current champion beyond expected judge

noise. On a near-tie prefer the simpler, shorter, lower-risk candidate.

### Graft

Inspect criterion vectors. A losing candidate may contain the strongest trigger

or rollback section. Build one challenger that grafts only those distinct

strengths onto the champion, then rerun all gates and scoring. Keep it only when

it wins.

### Holdout and promotion

After development iteration has stopped, run the final champion and original

baseline once on the untouched frozen holdout. Do not use that result to tune

another candidate in the same experiment.

Promotion requires:

- all hard gates pass;

- meaningful improvement on the primary score;

- no unacceptable regression in a protected criterion or scenario;

- target-workflow review when quality is subjective;

- scoped diff, retained baseline and tested rollback.

A development gain that disappears on holdout is overfitting. Revert to the

previous champion and record the negative result. Any further tuning requires a

new holdout, a new baseline and a new experiment record.

Prefer the smallest promoted change. Continuous improvement beats delayed

perfection only when each increment passes its current acceptance gates; the

principle never converts known failure or missing proof into success.

## Stop conditions

Stop on the first of:

- minimum meaningful improvement is not reached for the configured dry rounds;

- score reaches a practical ceiling without protected regressions;

- experiment budget is exhausted;

- candidates converge on the same shape;

- hard gate regresses during development;

- final holdout regresses, which ends promotion rather than starting a new

iteration;

- target, constraints, frozen eval or owner authorization changes.

Do not continue indefinitely. A null result is valid: retain the baseline and

report that no candidate proved better.

## Round log

```text

ROUND: <integer>

CHAMPION_IN: <version and development score>

CANDIDATES: <strategy, hard-gate result, criterion vector, score>

WINNER: <candidate or NONE>

GRAFT: <source ideas and result or NONE>

CHAMPION_OUT: <version and delta>

REGRESSIONS: <protected criteria/cases or NONE>

JUDGE_AGREEMENT: <dispersion and tie-break>

STOP_CHECK: <continue or exact condition>

```

## Final report

Return the winning artifact, baseline-to-final development and holdout scores,

hard-gate/test output, full compact round log, protected regressions, budget