bench git:20260810.8075e72A

Immutable. This exact content is served forever at /api/v1/blob/660d4856ff38e63e.

---
name: bench
description: Run embedding benchmarks — R@5 code retrieval, timing, model comparison
disable-model-invocation: false
---

# Embedding Benchmarks

Run retrieval quality and performance benchmarks.

## When to Use

- Evaluating embedding model changes
- Performance regression testing
- Comparing model candidates
- User says "benchmark", "bench", "test embeddings"

## Steps

### Step 1: Check Prerequisites

```bash
# Verify benchmark data exists
ls benchmarks/data/ 2>/dev/null || echo "No benchmark data"

# Verify current model
python -c "from mempalace_code.storage import DEFAULT_EMBED_MODEL; print(f'Model: {DEFAULT_EMBED_MODEL}')"
```

### Step 2: Code Retrieval Benchmark

Run the standard code retrieval benchmark:

```bash
python benchmarks/code_retrieval_bench.py \
  --repo-dir . \
  --modes smart \
  --out benchmarks/results_$(date +%Y%m%d).json
```

Metrics collected:
- **R@5**: Recall at 5 (target: >= 0.95)
- **R@10**: Recall at 10 (target: 1.0)
- **Embed time**: Seconds to embed all chunks
- **Query time**: Milliseconds per query
- **Index size**: MB on disk

### Step 3: Category Breakdown

If benchmark supports categories:

| Category | Description |
|----------|-------------|
| architecture | High-level design questions |
| class_lookup | Find specific class definitions |
| cross_file | References spanning multiple files |
| function_lookup | Find specific functions |

### Step 4: Compare Models (optional)

If comparing multiple models:

```bash
python benchmarks/embed_ab_bench.py \
  --models minilm,mpnet \
  --skip-longmemeval \
  --out benchmarks/results_models_$(date +%Y%m%d).json
```

### Step 5: Text Retrieval Gate (if changing models)

Per project policy, any embedding model change must also pass text retrieval benchmarks:

```bash
test -f benchmarks/data/longmemeval_s_cleaned.json \
  || { echo "Missing LongMemEval fixture"; exit 1; }

python benchmarks/embed_ab_bench.py \
  --models minilm,mpnet \
  --longmemeval-data benchmarks/data/longmemeval_s_cleaned.json \
  --out benchmarks/results_models_with_text_gate_$(date +%Y%m%d).json
```

Target: Match or beat current model on LongMemEval R@5.

## Output Format

```
## Benchmark Results

Model: all-MiniLM-L6-v2
Dataset: mempalace code retrieval (20 queries, N chunks)

| Metric | Value | Target | Status |
|--------|-------|--------|--------|
| R@5 | 0.950 | >= 0.95 | PASS |
| R@10 | 1.000 | 1.0 | PASS |
| Embed time | 15.2s | < 60s | PASS |
| Query time | 15.9ms | < 100ms | PASS |
| Index size | 17.0 MB | < 50 MB | PASS |

Category R@5:
- architecture: 0.800
- class_lookup: 1.000
- cross_file: 1.000
- function_lookup: 1.000

**Verdict: PASS** — Model meets all targets.
```

## Model Comparison Table

When comparing models, produce:

```
| Model | R@5 | R@10 | Embed(s) | Query(ms) | Index(MB) |
|-------|-----|------|----------|-----------|-----------|
| all-MiniLM-L6-v2 | 0.950 | 1.000 | 15.2 | 15.9 | 17.0 |
| all-mpnet-base-v2 | 0.900 | 1.000 | 47.5 | 30.5 | 17.7 |

**Recommendation:** minilm remains default (better R@5, 3x faster).
```