tap-model-training git:20261108.a3ce67aA

Immutable. This exact content never changes and is served at /api/v1/blob/a96b314cab6b5a05.

---
name: tap-model-training
title: "Octomind-Tap Embed Model Training and Publishing"
description: "Train, evaluate and publish the octomind-embed model behind capability auto-activation. Use when capability triggers or skill semantic() phrases change."
license: Apache-2.0
compatibility: "octomind-tap repo with the model/ subdirectory. Requires Python 3.11+, uv, ANTHROPIC_API_KEY for LLM augmentation, HF_TOKEN with write scope for publishing. GPU optional."
capabilities: core
domains: octomind
rules:
  - file(model/scripts/train.py)
  - file(model/bin/train)
  - content(retrain)
  - content(octomind-tap)
  - content(octomind-embed)
  - content(auto-activation)
  - content(auto_activate)
  - match(\b(re)?train\s+(the\s+)?(tap|embedding|embed)\b)
  - match(\b(publish|ship)\s+(the\s+)?(tap|embedding|embed|model)\b)
  - match(\b(add|edit|update)\s+(a\s+)?(new\s+)?(capability|capabilities|trigger|triggers)\b)
  - match(\b(eval_gate|eval_real|build_dataset|build_eval_seed|calibrate_thresholds|model_soup|compare_bases)\b)
  - match(\bbin/(train|publish)\b)
  - match(\b(auto[\s-]?activation|capability\s+routing)\b)
  - semantic(retrain the tap embedding model)
  - semantic(I added a new capability — what do I need to do)
  - semantic(how do I republish octomind-embed)
  - semantic(my new triggers aren't being picked up by auto-activation)
  - semantic(training got disconnected, how do I resume)
  - semantic(how do I update the runtime threshold for capability auto-activation)
  - semantic(the model is missing intents for one of my capabilities)
---

## Overview

`muvon/octomind-embed` is the bi-encoder octomind uses to score a user message against capability triggers and skill `semantic()` phrases. Since 2026-09 it is `ibm-granite/granite-embedding-30m-english` (6 layers, 384-dim, CLS-pooled) given a short contrastive fine-tune and blended back into the base as a WiSE-FT soup; the runtime loads its int8 ONNX graph. Whenever triggers or semantic phrases change, the model is stale. This skill walks through retraining, calibration, publishing, and recovery.

The cross-encoder reranker is intentionally not trained or shipped: at the current capability count it does not improve routing over a well-calibrated bi-encoder.

## Mental model

The runtime decision is: intent → one installed capability, or abstain. It scores each capability as the mean of its top-3 trigger cosines and accepts the winner only if it clears an absolute threshold and beats the runner-up by a margin. The model, the trigger corpus, and those two constants are one system: retraining moves the cosine scale, so the constants must be recalibrated on the int8 graph and shipped together with the weights.

The fine-tune is a nudge. A full fine-tune forgets the base's generalization; interpolating fine-tuned and base weights beats both parents, so `bin/train` sweeps the soup alpha and ships the best gate.

## Rules

### When to retrain

| Change | Retrain? |
|--------|----------|
| Added or removed a capability (`capabilities/<name>/config.toml`) | yes |
| Edited `triggers = [...]` | yes |
| Added or edited `semantic(...)` rules in any `SKILL.md` | yes |
| Edited the `_oos` curated phrases in `build_dataset.py` | yes |
| Agent, skill body, or docs change with no trigger touched | no |
| Threshold/margin change in octomind `capability.rs` / `skill.rs` | no — rebuild octomind |

### Standard retrain flow

Run from `model/`, inside `tmux` or `nohup` so a dropped connection doesn't kill it:

```bash
cd model && uv sync
tmux new -s train
ANTHROPIC_API_KEY=... bin/train --llm --resume --iterations 2
```

What runs (`bin/train`):

1. `build_eval_seed.py` once, if `data/eval_real.jsonl` is missing.
2. `augment_llm.py --resume` — LLM paraphrases for new `(label, trigger)` pairs only, appended to `data/intents.jsonl`.
3. `build_dataset.py` — `pairs.jsonl`, `triplets.jsonl`, `holdout.jsonl`, and `raw_triggers.jsonl` (the un-augmented corpus the runtime actually embeds), with the `_oos` sink label, multi-turn surfaces, and hard-negative mining.
4. `train.py` — CachedMNRL (scale 10) + Matryoshka, one epoch capped by `train.max_steps`; with `--iterations N`, triplets are re-mined from the previous round's mistakes between rounds.
5. `model_soup.py` + `compare_bases.py` — sweep `soup.alphas` against the base and keep the best gate on `raw_triggers.jsonl`.
6. `eval.py` (holdout) and `eval_gate.py` (`eval_real`, informational here).
7. `export_onnx.py` (fp32 + int8), then `eval_gate.py` and `calibrate_thresholds.py --target-fpr 0.03` on the int8 graph.

Other flags: `--skip-build` (reuse `data/`), `--skip-export`, `--ft checkpoints/<dir>` (redo soup, eval, export, and calibration on an existing run), `--legacy` (diagnostic `model.fit` path).

### Calibrating the runtime gate

`bin/train` prints the recommended operating point. To rerun by hand:

```bash
uv run python scripts/calibrate_thresholds.py \
  --model checkpoints/<run>/onnx --onnx-file model_quantized.onnx --target-fpr 0.03
```

It sweeps τ × δ over `data/eval_real.jsonl` and recommends the highest gate accuracy with null false-positive rate under the target. Copy the values into:

- `octomind/src/mcp/runtime/capability.rs` — `AUTO_ACTIVATE_THRESHOLD`, `AUTO_ACTIVATE_MARGIN`
- `octomind/src/mcp/runtime/skill.rs` — `SEMANTIC_DEFAULT_THRESHOLD`, `SEMANTIC_MARGIN`
- `model/configs/default.yaml` — `eval.runtime_threshold`, `eval.runtime_margin` (the publish gate scores at these)

The shipped soup runs at 0.65 / 0.06. Calibrate on the int8 graph: quantization shifts cosines, and that graph is what users load.

### Publishing

```bash
HF_TOKEN=hf_... bin/publish            # latest checkpoint; --run <dir> to choose
```

`bin/publish` runs `eval_gate.py` against `eval_baselines.json` on the fp32 weights and again on the int8 graph, and aborts on regression (top-1 accuracy drop > 0.01, null FPR rise > 0.01, any per-capability recall drop > 0.05). It warns if `onnx/model_quantized.onnx` is missing, since the runtime prefers it. `--force`/`--skip-gate` bypass the gate for documented emergencies only; `--dry-run` prints without uploading.

New weights reach fresh installs only — a cached HF repo is not re-checked — so ship the recalibrated octomind constants in the same release. After the rollout is confirmed, record the baseline the next publish gates against:

```bash
uv run python scripts/eval_gate.py --model muvon/octomind-embed --write-baseline eval_baselines.json
```

### Resuming an interrupted run

The trainer checkpoints every epoch (`save_total_limit=2`). If `checkpoints/embed-*/checkpoint-N/` exists:

```bash
uv run python scripts/train.py --resume                 # latest run dir
uv run python scripts/train.py --resume-from <run-dir>  # explicit
```

If iteration 1 was interrupted: finish it with `--resume`, re-mine with `build_dataset.py --neg-embed-model <iter1-dir>`, train iteration 2 with `train.py`, then `bin/train --ft <iter2-dir>` for soup, eval, export, and calibration. `--resume` covers the modern `train.py` path only, not `--legacy`.

## Examples

### Example 1: new triggers not picked up

❌ `bin/train --skip-build` after editing triggers — the build is what turns triggers into training surfaces, so the new run inherits the old weaknesses.

✅ `bin/train --llm --resume --iterations 2` — rebuild, augment only the new triggers, retrain.

### Example 2: wrong capability wins but triggers look right

Retraining rarely fixes this. First run `calibrate_thresholds.py` to see whether the shipped τ/δ still fit the model; add the failing prompt to `data/eval_real.jsonl` with its correct label and rerun `eval_gate.py`; then compare the competing capabilities' triggers — shared generic vocabulary is fixed by differentiating triggers, not by more training.

### Example 3: publishing without a baseline update

Publishing and forgetting `--write-baseline` leaves the next publish gating against the previous model, so a later regression can slip through. Record the baseline once the rollout is confirmed.

## Checklist

- [ ] Trigger or semantic-phrase change → full rebuild with `--llm --resume`, not `--skip-build`?
- [ ] Training run inside tmux or nohup?
- [ ] Soup winner evaluated on holdout and `eval_real`, int8 graph exported and scored?
- [ ] τ/δ calibrated on the int8 graph and copied into `capability.rs`, `skill.rs`, and `configs/default.yaml`?
- [ ] `bin/publish` passed the gate without `--force`, and the octomind release carries the new constants?
- [ ] Baseline rewritten after the rollout was confirmed?

## Composition / References

- `model/README.md` — pipeline, dataset, loss stack, eval set
- `model/configs/default.yaml` — base model, training, soup alphas, export, runtime-mirror constants
- `model/bin/train`, `model/bin/publish` — entry points
- `tap-capability-authoring` skill — writing triggers