experiment-planner ยท diff
git:20260810.0d9c5ea to git:20260905.1cfe2a2
126 added, 33 removed. Audit A to A.
---
name: experiment-planner
- description: Use when exploring a deep-learning or computer-science research idea before implementation or paper writing. Converts claims into pilot-first experiment matrices covering ablations, diagnostics, robustness, failure analysis, resource coordination, and paper-story viability.
+ description: Use when exploring, continuing, or accepting a deep-learning or computer-science research idea. Converts claims into pilot-first experiment matrices covering diagnostics, failure analysis, authorized execution, evaluation, and paper-story viability.
license: MIT
---
# Experiment Planner
## Overview
- Use this skill before paper writing when the user needs to turn a research idea
- into a testable story and experiment plan. It is an adapter over existing
+ Use this skill when the user needs to explore a research idea, continue an
+ authorized experiment workflow, or accept and interpret results. It turns
+ claims into a testable story and experiment plan. It is an adapter over existing
research-agent ideas, not a replacement for the user's writing, review,
rebuttal, figure, evidence, or GitHub release skills.
## Core Boundaries
- Default domain: general deep learning and computer science research. Adapt to
collaborative perception, 3D perception, or autonomous driving only when the
task context calls for it.
- For discussion-only planning, keep output in chat unless the user asks for a
saved artifact. When the user asks to implement or run experiments in a
repository, persist the pre-run result contract: update the paper's final
LaTeX tables when a manuscript is in scope; otherwise update the existing
experiment-planning document or create `experiment-plan.md` at the repository
root.
- Do not launch long experiments, deploy GPU jobs, modify code, or retry failed
- runs unless the user explicitly asks for execution.
+ runs unless the user explicitly asks for execution or a still-valid approval
+ covers the ordinary continuation. A bounded same-config recovery stays
+ within that approval; do not silently expand its scope.
- Do not replace `paper-section-playbook`, `paper-refinement-skills`,
`paper-review-panel`, `rebuttal-response-skills`, `paper-visual-craft`, or
`github-project-release`; hand off to them only after the research plan or
results are ready.
- Treat external projects as references, not installed dependencies. Read
`references/source-map.md` before discussing provenance or upgrading this
skill from upstream sources.
+ - During approved execution, a still-valid earlier user authorization covers
+ ordinary implementation, local bug fixes, queueing in-scope runs, and
+ evaluation. It does not authorize a new experiment family, dataset, seed
+ policy, or paper-only TODO; those remain explicit opt-ins.
+ ## Entry Modes
+
+ Choose the mode that matches the current request:
+
+ - **Exploration**: turn a new research direction into a falsifiable claim. Use
+ the complete matrix for a substantial new study; a small pilot can use the
+ minimal question/comparison/protocol-plus-budget/endpoint/decision contract.
+ - **Approved execution**: continue within the frozen contract and the scope of
+ a still-valid user authorization. Do not silently expand the experiment
+ family, dataset, seed policy, or resource budget.
+ - **Acceptance**: compare observed artifacts with the frozen contract and record
+ whether the relevant gate passes, fails, or is inconclusive.
+
+ For a continuation, load the latest plan, results, and pass receipt when
+ available.
+ Reuse still-valid decisions about the claim, baseline, metrics, budget, seed,
+ and protocol. Do not restart the initial grill, literature pass, or full matrix
+ unless a material change or unresolved blocker makes it necessary; ask only
+ questions that affect the next decision.
+
## Default Workflow
- 1. **Grill consensus**: use `$grill-me` style interaction to clarify problem,
- motivation, proposed claim, baseline/control, compute budget, success
- criteria, and unacceptable shortcuts. Ask one high-impact question at a time
- when the answer changes the experiment plan.
- 2. **Literature inspiration**: after a preliminary consensus, use
- `$research-evidence` for related papers, novelty risk, prior experiment
- patterns, and unsupported claims. Use `$search-first` when the task may need
- existing code, datasets, tools, or implementations.
+ 1. **Grill consensus**: for a substantial new exploration, use `$grill-me`
+ style interaction to clarify problem, motivation, proposed claim,
+ baseline/control, compute budget, success criteria, and unacceptable
+ shortcuts. For a small pilot, record only the minimal contract. For a
+ continuation, reuse the settled contract and ask only changed or blocking
+ questions.
+ 2. **Literature inspiration**: for a substantial new exploration or a changed
+ novelty claim, use `$research-evidence` for related papers, novelty risk,
+ prior experiment patterns, and unsupported claims. Use `$search-first` when
+ the task may need existing code, datasets, tools, or implementations. A
+ small pilot may defer this pass when it does not change the decision. Reuse
+ prior literature decisions when the claim and scope are unchanged.
3. **Story viability check**: decide whether the idea can support a clean paper
story: important problem, credible gap, specific method difference, feasible
validation, and claims that will not outrun the evidence.
4. **Claim freeze**: freeze the smallest verifiable claim before planning runs.
Avoid changing the story repeatedly while experiments are running.
- 5. **Paper/table contract freeze**: before scheduling runs, define the final
- main-result, ablation, and necessary diagnostic tables. For every metric,
- record its plain-language definition, unit, direction, aggregation, and any
- delta reference. Use `--` for unavailable values and do not write claims
+ 5. **Contract freeze**: before scheduling a substantial study or full runs,
+ define the final main-result, ablation, and necessary diagnostic tables. For
+ every metric, record its plain-language definition, unit, direction,
+ aggregation, and any delta reference. For a small pilot, freeze only the
+ endpoint, comparison, protocol-plus-budget, and decision gate. Use the
+ in-scope project's explicit missing-value schema and do not write claims
from placeholder cells.
6. **Idea validation first**: design the smallest pilot/smoke/sanity experiment
that can falsify or support the core hypothesis. If multiple GPUs are idle,
parallelize only independent exploration runs with clear ownership.
7. **Minimum sufficient matrix**: only after the pilot passes, add the main
result and claim-critical ablations. Add robustness, diagnostics, efficiency,
qualitative results, or failure analysis only when they support a paper
claim or answer a credible reviewer question; do not add them for symmetry.
8. **Subagent coordination**: keep the main session responsible for planning,
task decomposition, and final result acceptance. Use `explorer` for read-only
repo/config/protocol investigation. Use `worker` for implementation with
explicit file or module ownership. Do not manually override subagent model or
reasoning settings unless the user explicitly requests it.
- 9. **Run discipline**: test that the command starts and produces plausible small
- outputs; remove test data after smoke checks; launch the full run only after
- sanity passes; inspect the first few samples/logs/artifacts; stop continuous
- monitoring once the run is confirmed healthy unless the user asks otherwise.
- 10. **Explicit-only expansion policy**: record one fixed seed and keep compared
+ 9. **Impact-based validation**: prioritize checks by their ability to block the
+ next decision or invalidate the claim. Reuse a passing check or receipt when
+ code, data, configuration, protocol, relevant environment, and dependencies
+ are unchanged. Run the smallest useful CPU/configuration smoke check, then
+ proceed to a feasible authorized GPU pilot; do not add endless CPU gates
+ before that pilot. Do not use hashes or hash parity as routine acceptance
+ checks.
+ 10. **Run discipline**: test that the command starts and produces plausible
+ small outputs; remove test data after smoke checks; launch the full run only
+ after the relevant sanity gate passes; inspect early samples, logs, and
+ artifacts; stop continuous monitoring once the run is confirmed healthy
+ unless the user asks otherwise. Record a compact reusable pass receipt with
+ the scope, checks/evidence, decision, and next action; add a command/run
+ reference or exception only when needed for interpretation or recovery.
+ 11. **Acceptance discipline**: freeze the acceptance rule, metric, comparison,
+ and claim boundary before inspecting the outcome. Never weaken a gate,
+ switch the primary metric, or redefine the baseline after seeing a result.
+ A diagnostic gate tests a bug, confound, or mechanism; it is not by itself
+ the whole scientific verdict.
+ 12. **Explicit-only expansion policy**: record one fixed seed and keep compared
runs under the same evaluation and checkpoint-selection policy. Use one
- training run by default. Do not add CL experiments, multi-seed or
- repeated-seed runs, or another auxiliary experiment family unless the user
- explicitly requests that exact experiment. Variance concerns, a small
- margin, inexpensive runs, idle GPUs, reviewer expectations, or venue norms
- do not count as authorization. Mention such experiments only as unrun
- options or limitations when useful.
- 11. **Default decisions**: ask only questions whose answers materially change
+ training run by default. New experiment families, datasets, multi-seed or
+ repeated-seed policies, or other auxiliary runs require explicit opt-in.
+ An earlier authorization covers ordinary implementation, local bug fixes,
+ queueing in-scope runs, and evaluation only. A paper TODO, idle GPU, small
+ margin, low cost, reviewer expectation, or venue norm is not authorization.
+ Keep the resource and time budget honest; mention unrun options only as
+ limitations or explicitly opt-in follow-ups.
+ 13. **Default decisions**: ask only questions whose answers materially change
the plan. If a non-critical choice goes unanswered, use the recommended
default and record it as an assumption.
+ ## Validation and Failure Taxonomy
+
+ Name the relevant gate before running it and do not treat all failures as the
+ same:
+
+ - **Engineering failure**: the command, data path, configuration, checkpoint,
+ or artifact contract is broken. Repair the implementation or protocol before
+ interpreting scientific results.
+ - **Optimization failure**: the intended method does not train, converge, or
+ improve the fixed target under a fair protocol. Report it as a method or
+ tuning failure, not automatically as a failure of the underlying idea.
+ - **Prototype failure**: a sound, deployable pilot misses its predeclared
+ mechanism endpoint, including a negative endpoint despite a plausible
+ mechanism. Report that measured benchmark endpoint, but do not infer
+ impossibility or falsification of the broader mechanism.
+ - **Core-construct failure**: the predeclared construct-level test contradicts
+ the central hypothesis under the frozen protocol. Close or narrow that claim
+ unless new evidence justifies a separately approved design.
+ - **Inconclusive/blocked**: noisy or confounded evidence, missing support, an
+ invalid protocol, or an unresolvable engineering issue prevents a gate from
+ being interpreted. Do not relabel this status as a scientific failure.
+
+ Diagnostic gates can localize a failure or block a particular interpretation,
+ but passing one does not establish the entire scientific story. Keep the gate,
+ evidence, and remaining uncertainty explicit.
+
## Output Contract
Default to a concise in-chat experiment matrix. Before producing a matrix, read
- `references/experiment-matrix.md`.
+ `references/experiment-matrix.md`. The full 19-field matrix is for a substantial
+ new study. A small new pilot needs only the minimal
+ question/comparison/protocol-plus-budget/endpoint/decision contract. For a
+ continuation, return only changed fields, the next decision, or a compact pass
+ receipt, and reference unchanged fields from the existing plan.
- The matrix must include:
+ The substantial-study matrix must include:
- `research question`
- `core hypothesis`
- `paper claim`
- `storyline`
- `literature inspiration`
- `baseline/control`
- `table contract`
- `metric definitions`
- `idea validation experiment`
- `expected signal`
- `failure modes`
- `diagnostic checks`
- `follow-up experiments`
- `subagent/task ownership`
- `compute/resource assumptions`
- `seed policy`
- `success gate`
- `claim gate`
- `next action`
- Use `unknown` or `needs user input` for unresolved fields instead of inventing
- project facts. Keep recommendations executable, but do not perform execution
- inside this skill unless the user asks for implementation or running commands.
+ Use the in-scope project's explicit missing-value schema. If none is defined,
+ use `TBD (pending)`, `N/A (inapplicable)`, or `unknown (needs user input)` as
+ appropriate; do not force `--` or invent project facts. Keep recommendations
+ executable, but do not perform execution inside this skill unless the user asks
+ for implementation or running commands, or a still-valid prior authorization
+ already covers the ordinary continuation.
## Handoff Rules
- Use `$research-evidence` before making novelty, citation, or literature
coverage claims.
- Use `$search-first` before proposing new implementation utilities, pipelines,
tool integrations, or dataset-processing code.
- Use writing skills only after the experiment story is stable enough to draft
a paper section, rebuttal, review, table, or figure.
- For code work, assign `worker` tasks with disjoint write scopes and remind the
worker not to revert others' changes.
- For investigation, assign `explorer` tasks that are specific, read-only, and
non-overlapping with the main session's current work.
## Failure Modes To Catch
- The idea is interesting but not falsifiable with available data or compute.
- The proposed contribution is only a presentation change, not a testable method
or analysis difference.
- The baseline/control is missing, unfair, or weaker than the claim requires.
- The pilot experiment cannot distinguish mechanism from implementation noise.
- The plan jumps to full benchmark runs before smoke and sanity checks pass.
- The story changes after seeing results without recording a clear reason.
+ - A continuation reopens settled decisions or asks for the entire initial
+ matrix without a material change.
+ - Engineering, optimization, prototype, and core-construct failures are
+ conflated.
+ - A diagnostic gate is treated as the complete scientific verdict.
+ - An acceptance threshold is weakened, or the primary metric/baseline is
+ changed, after the outcome is visible.
+ - Validation expands into hash checks or endless low-impact CPU gates instead
+ of reaching a feasible downstream pilot.
+ - A new experiment family, dataset, seed policy, or paper-only TODO is smuggled
+ into an existing authorization or budget.
- Subagents receive vague tasks, overlapping write scopes, or authority to run
long jobs without main-session acceptance.