llm4ad-task-builder git:20260723.07072d9A

Immutable. This exact content never changes and is served at /api/v1/blob/0121bdf4f1c0efd4.

---
name: llm4ad-task-builder
description: "Use when a user wants to build an LLM4AD_Next task package — a runnable directory that lets the LLM4AD platform evolve an algorithm for their problem. Covers what files a task package contains, each file's contract, and how the package is run on the LLM4AD platform."
---

# Building an LLM4AD_Next Task Package

## What LLM4AD_Next is

LLM4AD_Next is an automated algorithm-design platform: it combines an LLM with
evolutionary optimization to discover and improve algorithms automatically. The
user describes a problem (e.g. "evolve a solver for the Traveling Salesman
Problem"), marks the code region to evolve with `# EVOLVE_START` / `# EVOLVE_END`,
and the platform repeatedly asks an LLM to propose better code inside that region,
scores each candidate with a custom evaluator, and keeps the best across
generations. Better algorithms emerge through guided search rather than manual
trial and error.

Your job with this skill: turn a user's problem into a **complete, runnable task
package**, then verify it actually runs before handing it over.

## What a task package is

A task package is a self-contained directory with everything the platform needs to
evolve algorithms for one problem:

| File | Purpose |
|---|---|
| `config.yaml` | Master config for the whole pipeline (evolution params, providers, evaluator, dataset). This is what gets run. |
| `<name>_evaluator.py` | A `BaseEvaluator` subclass that runs a candidate algorithm on test data and returns a score + metrics. Defines what "better" means. |
| `<algo_dir>/<algo>.py` | The algorithm, with the function to evolve wrapped in `# EVOLVE_START` / `# EVOLVE_END`. Reads input as a JSON CLI arg, prints a JSON result. |
| `debug_run.py` | Runs the full pipeline once (`LLM4AD("config.yaml").run()`) — a smoke test. |
| `test_evaluator.py` | Loads the evaluator via the dispatcher to confirm it imports and is wired correctly. |
| `data/sample/*.json` | 2-3 small test instances the evaluator scores algorithms against. |

## The contracts each file must satisfy

**Algorithm file** (`<algo_dir>/<algo>.py`):
- The algorithm directory is the one named in `version_control.local_path`; the
  platform copies it into a git worktree for each candidate and edits only the code
  between the markers.
- The function to evolve is wrapped exactly in `# EVOLVE_START` and `# EVOLVE_END`
  comment markers (the platform only edits code between them).
- It reads its input as a single JSON string from `sys.argv[1]`, and prints a JSON
  result to stdout — so the evaluator can run it as a subprocess.
- The file name must match the name the evaluator looks for (see below). In the TSP
  example both sides use `solve.py`.

**Evaluator** (`<name>_evaluator.py`):
- Subclasses `BaseEvaluator`, decorated `@BaseEvaluator.register("<name>")`, with a
  no-argument `__init__(self)` (the base `__init__` takes no config).
- Implements `metrics` (list of `Metric(name, type=MINIMIZE|MAXIMIZE, weight, description)`),
  `name`, and `async def evaluate(self, cfg: EvalContext) -> EvaluationResult`.
- `cfg: EvalContext` carries `cfg.project_root` (the candidate's worktree dir),
  `cfg.data_path` (the current data file), and `cfg.timeout` (seconds per instance).
- `evaluate` locates the algorithm file **by hardcoded name** under
  `cfg.project_root` (e.g. `Path(cfg.project_root) / "solve.py"`), runs it as a
  subprocess on `cfg.data_path`, parses its JSON output, validates it, and returns
  `EvaluationResult(score, metrics, success, ...)` (negative cost for a minimization
  objective). That hardcoded name **must** equal the algorithm file's actual name —
  this is the most common wiring mistake.

**config.yaml** (10 sections — generated programmatically, not hand-written):
- `providers` use `${LLM_BASE_URL}` / `${LLM_API_KEY}` / `${LLM_MODEL}` env
  placeholders (the platform fills them in).
- The evaluator `module` reference (`<file>.py:<ClassName>`), the `metrics` list,
  the `dataset` path (`data/sample`), and the coder `prompt_template`'s EVOLVE block
  must all be consistent with the other files.

**debug_run.py** / **test_evaluator.py**: standard boilerplate that runs
`config.yaml` end-to-end and loads the evaluator, respectively.

See `reference/config-template.yaml` for the config shape and `reference/example-tsp.md`
for a pointer to a complete, known-good package in this repo to adapt.

## The information to gather before building

A correct package needs these decided (ask the user; let the agent fill sensible
defaults where the user has no preference):

1. **Problem description** — what is being optimized. (Required — ask.)
2. **Function/algorithm to evolve + input/output format** — ask if the user has any
   constraints; if not, the agent designs it.
3. **Evaluation metric(s)** — what to minimize/maximize. (Required — ask clearly.)
4. **Reuse existing code/evaluator?** — user provides existing code, or generate from
   scratch. (Ask.)
5. **Data source** — user provides data, or generate sample instances. (Ask.)
6. **Programming language** — default Python. (Offer as a choice.)
7. **Project name** — the agent proposes one from the description; confirm with user.

## Build workflow

1. Understand the problem and the 7 items above (inspect any user-provided code/data).
2. Write the algorithm file first; verify it runs standalone and prints valid JSON.
3. Write 2-3 sample data files under `data/sample/`.
4. Write the evaluator; keep its metrics consistent with `config.yaml`.
5. Write `config.yaml`, `debug_run.py`, `test_evaluator.py`.
6. **Self-test (required):** run `test_evaluator.py` (evaluator loads — no LLM, no
   network, cheap and always runnable). Then run `debug_run.py`, which executes the
   full pipeline and **does call the LLM**: it needs the `LLM_BASE_URL` /
   `LLM_API_KEY` / `LLM_MODEL` env vars set (the config uses these placeholders) and
   consumes tokens over the network. Read every error, fix the relevant file,
   re-run. Not done until both run cleanly. If no provider credentials are available,
   say so — a missing-credential failure of `debug_run.py` is not a package defect.

## Automated algorithm design methods (reference — only when the user asks)

Always build new task packages with the default **Island GA** method. Do NOT
proactively suggest switching to another method — let the user bring it up.

When the user asks about methods or explicitly requests a switch, help them
using the reference below. After applying a switch, ask whether they want to
continue adjusting parameters or are satisfied.

Each method requires a matching pair of `evolution.type` + `planner.type`.
Switching methods means **replacing the entire `evolution` section` (parameters
differ per method) and updating `planner.type`. A fresh run is required — you
cannot resume a prior run after changing the evolution type.

---

**Island GA** (default — use this for all new builds)

```yaml
evolution:
  type: "island_ga"
  max_generations: 50
  num_islands: 5              # number of parallel populations
  island_population_size: 20  # individuals per island
  mutation_rate: 0.3
  crossover_rate: 0.5
  tournament_size: 3
  early_stop_patience: 10
  migration_interval: 5       # gens between cross-island exchanges
  migration_rate: 0.1         # fraction of individuals migrated
  migration_strategy: "best"  # best | random | elite | worst
  migration_topology: "ring"  # ring | fully_connected | hierarchical | mesh
  parallel_islands: true      # run islands concurrently
planner:
  type: "llm_evolution"
```

**EoH** — Evolution of Heuristics *(ICML 2024)*  
Single-objective, rank-based truncation. Runs E1 (and optionally E2/M1/M2)
operators each generation.

```yaml
evolution:
  type: "eoh"
  max_generations: 50
  population_size: 5          # top-k individuals kept after truncation
  selection_num: 2            # parents selected per E1/E2 crossover
  max_sample_nums: 100        # LLM call budget cap for the run
  num_samplers: 1             # parallel candidates per operator per gen
  use_e2_operator: true       # enable E2 (backbone-motivated crossover)
  use_m1_operator: true       # enable M1 (structural mutation)
  use_m2_operator: true       # enable M2 (parameter mutation)
  seed_path: null             # optional path to a seed heuristic file
planner:
  type: "eoh_evolution"
```

**ReEvo** — Reflective Evolution *(NeurIPS 2024)*  
Adds two reflection signals to genetic search: short-term (comparing worse/better
parent pairs → better crossover) and long-term (accumulated history → better
elite mutation).

```yaml
evolution:
  type: "reevo"
  max_generations: 50
  population_size: 8          # individuals in the population
  mutation_rate: 0.5          # fraction of pop used for elite mutations per gen
  max_sample_nums: 100        # LLM call budget cap for the run
  num_samplers: 1             # parallel crossover candidates per step
  seed_path: null             # optional path to a seed heuristic file
planner:
  type: "reevo_evolution"
```

**MCTS-AHD** — Monte Carlo Tree Search for AHD *(ICML 2025)*  
Tree search over algorithm space: selects nodes via UCT, expands with
e1/e2/m1/m2/s1 operators. Not generational; `max_generations` is interpreted
as MCTS iterations.

```yaml
evolution:
  type: "mcts_ahd"
  max_generations: 1000       # MCTS iterations (not generations)
  init_size: 4                # initial child nodes expanded under root
  population_size: 10         # active algorithm pool size
  selection_num: 2            # parents per e1/e2 operator
  max_sample_nums: 100        # LLM call budget cap for the search
  alpha: 0.5                  # UCT progressive-widening parameter
  lambda_0: 0.1               # UCT exploration constant (decays with budget)
  max_depth: 10               # maximum tree depth
  seed_path: null             # optional path to a seed heuristic file
planner:
  type: "mcts_ahd_evolution"
```

### Switching checklist (only when the user actively requests it)

1. **Replace `evolution` section** — remove the old method's specific params,
   write the new method's complete section from the templates above. Do NOT
   carry over params that don't exist in the new method's schema (e.g.
   `num_islands` is IGA-only).
2. **Update `planner.type`** — must match the new method.
3. **Keep everything else** — `providers`, `evaluator`, `coder`, `dataset`,
   `version_control` stay unchanged.
4. **Warn the user** — switching evolution type means a fresh run; you cannot
   resume an old checkpoint under a different method.
5. **Re-verify** — after editing `config.yaml`, re-run `debug_run.py` to
   confirm the new method loads correctly.
6. **Ask whether to continue** — after the switch is applied and verified, ask
   the user if they want to adjust any parameters further or are satisfied.

<!-- PLATFORM-USAGE-DRAFT: 以下"如何在平台上使用"为草稿，待产品同学修订 -->
## How the package is used on the LLM4AD platform

Once the package exists, the user runs it to evolve their algorithm. Two paths:

**CLI:**
```bash
llm4ad run path/to/config.yaml
# resume an interrupted run: pass a real checkpoint file written under
# ./runs/<project>/<run_id>/checkpoints/ (Island GA names them
# iga_checkpoint_gen_<N>_<hash>.json; MEoH names them meoh_<N>.json).
llm4ad run config.yaml --resume ./runs/<project>/<run_id>/checkpoints/iga_checkpoint_gen_5_a1b2c3d4.json
```
It launches the evolution pipeline and prints per-generation progress (best score
each generation).

**Web platform (this project's Web UI):**
1. Create or open a task (the package's `config.yaml` + files back it).
2. Configure an LLM provider (the user's own key/model — OpenAI-compatible or
   Anthropic). The package's `config.yaml` references providers by the platform's
   env placeholders; the platform injects the real credentials at run time.
3. Start the evolution run; monitor progress in the browser — current generation,
   best individual's score, metrics, and logs — in real time.
4. When it finishes, inspect the best evolved algorithm and its score.

**Tuning after generation (optional):** users often adjust `config.yaml` evolution
parameters to trade cost vs. quality — e.g. `max_generations`, `num_islands`,
`island_population_size`, `mutation_rate` / `crossover_rate`, `early_stop_patience`,
the provider `model`, and `evaluator.timeout`.

**Where results go:** each run writes to `./runs/{project}/{run_id}/`:
- `best/code/` — the evolved algorithm source (the main deliverable)
- `best/metadata.json`, `best/summary.txt` — score, generation, metrics
- `state/evolution_state.json` — full generation history (drives the Web UI dashboards)
- `logs/llm4ad.log` — full execution log for troubleshooting
- `checkpoints/` — snapshots to resume from
<!-- END PLATFORM-USAGE-DRAFT -->

## Completion criteria

The package is done only when `test_evaluator.py` loads the evaluator AND
`debug_run.py` runs without raising. Then summarize what was built (files + the
7 decisions) and the verification result.

After a successful build, always close by conveying this message (translate to the
user's language; do NOT proactively recommend a specific method — just mention
that the option exists):

> The task package has been built and verified. I can help you choose the
> evolution method (such as EoH, ReEvo, or MCTS-AHD), and adjust evolution
> parameters (such as max_generations, population size, etc.), or feel free
> to let me know if you have other needs!