nasde-benchmark-from-public-repos git:20260319.9c40b01A

Immutable. This exact content is served forever at /api/v1/blob/b71228b9a1df6beb.

---
name: nasde-benchmark-from-public-repos
description: |
  Build diverse benchmark task suites from public GitHub repositories for testing universal skills. Use this skill when the user wants to:
  - Create a benchmark that spans multiple public repositories and languages
  - Test a universal skill (refactoring, test writing, code review, etc.) across diverse codebases
  - Curate a representative set of repos and tasks for cross-codebase validation
  - Build an evaluation suite for a skill that should work in any repository
  Even if the user doesn't say "benchmark" — if they're building a skill meant to work everywhere and want to validate it across many different projects, this skill applies.
---

# NASDE Benchmark from Public Repos

Build a diverse NASDE benchmark by curating tasks from multiple public GitHub repositories. Designed for validating universal skills — skills that should work across different languages, frameworks, project sizes, and architectural styles.

## Prerequisites

- An existing NASDE benchmark project (run `nasde init` first, or use the `nasde-benchmark-creator` skill)
- A clear description of the skill being evaluated (what it does, what kinds of tasks it helps with)
- Internet access (to browse and clone public repositories)

## Step 1: Understand the skill under test

Ask the user:

- **What does the skill do?** (e.g., "helps agents refactor code", "guides test writing", "enforces DDD patterns")
- **What languages/frameworks should it support?** (e.g., "Python, TypeScript, and Go" or "any language")
- **What task types exercise the skill?** (e.g., "extract method, rename module, split class" for a refactoring skill)
- **Are there known weak spots?** (e.g., "seems to struggle with large files" or "not sure about Rust")

## Step 2: Design the diversity matrix

Based on the skill description, define axes of variation that the benchmark should cover. Present these to the user as a table:

### Example for a refactoring skill

| Axis | Values to cover | Why it matters |
|------|----------------|---------------|
| **Language** | Python, TypeScript, Go, Rust, C# | Refactoring idioms differ per language |
| **Project size** | Small (<5K LOC), Medium (5-50K), Large (>50K) | Large codebases stress navigation and context |
| **Test coverage** | Extensive tests, Minimal tests, No tests | Refactoring with no safety net is harder |
| **Architecture** | Monolith, Microservice, Library | Different refactoring patterns apply |
| **Difficulty** | Extract function, Split module, Restructure package | Increasing complexity |

Not every cell in the matrix needs a task. Aim for **8–15 tasks** that provide meaningful coverage across the axes. Ask the user which axes matter most — they may want to emphasize language diversity over project size, or vice versa.

## Step 3: Find candidate repositories

For each cell in the matrix that needs coverage, search for public repositories that fit.

**Good source repositories have:**
- A clear, active codebase (not abandoned, not a tutorial/toy project)
- A working build system and some test infrastructure
- A well-understood structure (README, organized directories)
- A permissive license (MIT, Apache 2.0, BSD — avoid GPL if the benchmark may be shared)
- Enough complexity to be a meaningful test (not a single-file script)

**Search strategies:**
1. **GitHub search** — search by language, stars, topic tags
2. **Known ecosystem repos** — well-known open source projects in each language (e.g., FastAPI for Python, Express for Node, Gin for Go)
3. **GitHub Trending** — find actively maintained repos with good structure
4. **User suggestions** — the user may know repos that represent their target audience

For each candidate repo, present:

```
[1] github.com/user/repo — "Description from GitHub"
    Language: Python | Size: ~15K LOC | Stars: 2.3K | License: MIT
    Tests: pytest suite, good coverage
    Why: Medium Python project, clean architecture, good refactoring target
    Proposed task: "Extract the database access layer into a repository pattern"

[2] github.com/user/repo2 — "Description from GitHub"
    Language: TypeScript | Size: ~40K LOC | Stars: 890 | License: Apache 2.0
    Tests: Jest, moderate coverage
    Why: Large TS project, component-heavy, tests UI refactoring
    Proposed task: "Split the UserDashboard component into focused sub-components"
```

Ask the user to select which repos and tasks to include.

## Step 4: Create tasks for each selected repo

For each approved repo+task pair, generate the full task directory. Work through each file with the user.

### 4a: Determine the "before" state

Unlike `nasde-benchmark-from-history` (which uses a specific commit), here you choose a state of the repo that presents the problem to solve:

- **Option A: Current main branch** — the repo as-is has the problem (e.g., a God class that should be split). Set `source.ref` to a specific commit hash on main for reproducibility.
- **Option B: A tagged release** — use a specific version. More stable for long-lived benchmarks.
- **Option C: Create a setup branch** — if the task requires introducing a specific problem into a clean codebase, create a branch that sets up the scenario. Push it to a fork or document the setup in the Dockerfile.

Always pin to a specific commit hash, not a branch name — branches move, hashes don't.

### 4b: task.json

```json
{
  "name": "<language>-<repo-slug>-<task-slug>",
  "description": "<What the agent must do>",
  "difficulty": "<easy|intermediate|hard>",
  "estimated_time_minutes": 30,
  "tags": ["<language>", "<skill-type>", "<project-size>"],
  "source": {
    "git": "https://github.com/<owner>/<repo>.git",
    "ref": "<pinned-commit-hash>"
  },
  "environment": {
    "type": "docker",
    "dockerfile": "./environment/Dockerfile"
  },
  "instruction": "./instruction.md",
  "evaluation": {
    "type": "script",
    "script": "./tests/test.sh",
    "timeout_seconds": 300
  },
  "metadata": {
    "language": "<language>",
    "framework": "<framework>",
    "source_repo": "https://github.com/<owner>/<repo>",
    "diversity_axes": ["<axis:value>", "<axis:value>"]
  }
}
```

The `diversity_axes` metadata helps track coverage across the matrix.

### 4b-bis: task.toml (required — Harbor config)

Harbor reads `task.toml`, not `task.json`. Every task directory MUST have both files. Generate `task.toml` alongside `task.json`:

```toml
version = "1.0"

[metadata]
name = "<language>-<repo-slug>-<task-slug>"
description = "<What the agent must do>"
difficulty = "<easy|intermediate|hard>"
language = "<language>"
framework = "<framework>"
source_repo = "https://github.com/<owner>/<repo>"

[agent]
timeout_sec = 1800

[verifier]
timeout_sec = 300
```

The `agent.timeout_sec` should be `estimated_time_minutes × 60`. The `verifier.timeout_sec` matches `evaluation.timeout_seconds` from task.json.

### 4c: instruction.md

Write a task instruction that:
- Describes the codebase context (what the project does, relevant directory structure)
- States the problem clearly (what needs to change and why)
- Defines success criteria the agent can verify
- Does NOT prescribe the implementation approach

```markdown
# Task: <Descriptive name>

## Context
You are working in `/app`, a <language> <framework> project that <brief description>.
The project structure relevant to this task:
<tree of relevant directories/files>

## Requirement
<What needs to change. Describe the problem, not the solution.
Example: "The UserService class handles authentication, authorization, profile management,
and notification dispatch. It's 800 lines and growing. Separate these concerns into
focused services.">

## Scope
- Focus on: <specific files/directories>
- Do NOT modify: <test files, configuration, unrelated modules>
- Preserve: <all existing public APIs, test behavior>

## Quality Expectations
- Follow <language> idioms and the project's existing style
- Maintain or improve test coverage
- Keep changes minimal — change what needs changing, nothing more

## Success Criteria
1. <Specific, testable criterion>
2. <Specific, testable criterion>
3. All existing tests continue to pass
```

**Important for universal skill benchmarks:** Write the instruction as if the agent has no special skill. The skill configuration is injected via the variant's `CLAUDE.md` — the instruction describes the raw problem.

### 4d: environment/Dockerfile

Each repo needs its own Dockerfile. Detect the stack and generate:

```dockerfile
FROM <base-image>

RUN apt-get update && apt-get install -y git curl wget ca-certificates && rm -rf /var/lib/apt/lists/*

WORKDIR /app
RUN git clone https://github.com/<owner>/<repo>.git . && git checkout <pinned-hash>

# Install dependencies
RUN <dependency-install-command>

# Verify build works at the "before" state
RUN <build-or-compile-command>

CMD ["/bin/bash"]
```

**Test the Dockerfile** before moving on:
```bash
docker build -t benchmark-test-<task-name> -f tasks/<task-name>/environment/Dockerfile .
```

### 4e: tests/test.sh

For public repos, the verifier typically:
1. Verifies the project still builds
2. Runs the existing test suite (all tests must still pass)
3. Checks for task-specific outcomes (new files exist, specific patterns in code, etc.)

```bash
#!/bin/bash
cd /app

echo "Step 1: Verifying build..."
if <build-command>; then
    echo "✓ Build succeeded"
else
    echo "✗ Build failed"
    echo 0 > /logs/verifier/reward.txt
    exit 1
fi

echo "Step 2: Running existing tests..."
if <test-command>; then
    echo "✓ Existing tests pass"
else
    echo "✗ Existing tests failed — regression detected"
    echo 0 > /logs/verifier/reward.txt
    exit 1
fi

echo "Step 3: Checking task-specific criteria..."
# Example: verify a specific file was created or a class was split
if <task-specific-check>; then
    echo "✓ Task criteria met"
else
    echo "✗ Task criteria not met"
    echo 0 > /logs/verifier/reward.txt
    exit 1
fi

echo "EVALUATION PASSED ✓"
echo 1 > /logs/verifier/reward.txt
exit 0
```

Task-specific checks depend on the task type:
- **Refactoring:** check that target files exist, original God class is smaller or removed
- **Test writing:** check that new test files exist and pass
- **Bug fix:** check that a specific test case now passes

### 4f: assessment_criteria.md

Use the benchmark project's `assessment_dimensions.json` and adapt each dimension to this specific task and language:

```markdown
# Assessment Criteria: <Task Name>

Evaluate the agent's solution across the following dimensions.
This task is in <language> using <framework> — apply language-specific standards.

## 1. <Dimension> (0–<max_score>)

| Score | Criteria |
|-------|----------|
| 0     | <worst — specific to this language and task> |
| ...   | ... |
| <max> | <best — specific to this language and task> |

**Key checks:**
- <Language-specific things to look for>
```

**Important:** Assessment criteria should reflect language idioms. "Good refactoring" looks different in Rust (ownership, lifetimes) vs Python (duck typing, protocols) vs Go (interfaces, embedding).

## Step 5: Review coverage

After all tasks are created, present the filled diversity matrix to the user:

```
Coverage matrix for "refactoring" skill benchmark:

| Language   | Small | Medium | Large |
|------------|-------|--------|-------|
| Python     |       | ✓ fastapi-extract | |
| TypeScript |       | | ✓ dashboard-split |
| Go         | ✓ cli-service-layer | | |
| Rust       | ✓ error-handling | | |
| C#         |       | ✓ ddd-aggregate | |

Gaps: No large Python project, no small TypeScript project.
```

Ask the user if they want to fill gaps or if current coverage is sufficient.

## Step 6: Verify the benchmark

1. **Build all Docker images** — every task must build successfully
2. **Run existing test suites** in each container — they must pass at the "before" state
3. **Dry run** on one or two tasks:
   ```bash
   nasde run --variant <skill-variant> --tasks <task-name> --without-eval -C <benchmark-dir>
   ```

## Tips

- **Pin commit hashes, not branches.** Branches move. A benchmark task that worked yesterday might break tomorrow if main advanced. Always use the full 40-char hash.
- **Diverse doesn't mean random.** Each task should test a different aspect of the skill. If three tasks all test "extract method in Python," replace two with different languages or problem types.
- **Test the Dockerfiles early.** The most common failure is a Dockerfile that doesn't build because a dependency changed or a repo restructured. Build images as you create tasks, not all at the end.
- **Keep instructions skill-agnostic.** The task instruction describes the problem. The skill is injected via the variant's `CLAUDE.md`. This separation lets you test the same tasks with and without the skill.
- **Start with 5–8 tasks.** You can always add more. A smaller, well-curated benchmark is better than a large, noisy one.
- **Combine with nasde-benchmark-creator.** This skill generates tasks from public repos; `nasde-benchmark-creator` handles project scaffold, dimensions, and variants. Use them together.