cipher-qlora-finetune git:20260908.e9ac98dA

Immutable. This exact content is served forever at /api/v1/blob/950b33a19b3b9d4b.

---
name: cipher-qlora-finetune
description: QLoRA / LoRA fine-tuning pipeline for the aide-cipher-4b in-house model (Qwen2.5-Coder 4B Q8 + LoRA adapter) on the GTX 1060 6GB. Use when preparing training data, running a fine-tune round, evaluating the new adapter against the capability battery, and promoting/archiving the result. Research-backed, no guessing.
---

# cipher-qlora-finetune — adapter fine-tune SOP for the in-house model

## 0. What this skill is for

The in-house model in AIDE is `aide-cipher-4b`:
- Base: `Qwen2.5-Coder-7B`-class lineage, actually 4B (4,022,468,096 params) at Q8_0
- Adapter: `frontier-lora.gguf` (v1, 132 MB) — operator-trained, currently the production adapter
- Runtime: `llama-server` (CPU build today, since Vulkan probe fails on this VMware SVGA stack; per `daemon-restart.err.log`)

The fine-tuning lane: produce a v2 adapter, evaluate against the same capability battery, promote only on positive delta, never regress the production path.

**Why this matters:** the user wants to "continue improving it, fine-tuning it, and testing it in AIDE-world workflows." The first step is a reproducible, measurable, gated fine-tuning round. Not LoRA on the 4B base at random — a documented procedure with research-backed numbers and verification gates.

## 1. Current verified state (2026-08-28)

| Item | Value | Source |
|---|---|---|
| Base model | `base.q8_0.gguf` (4.28 GB) | `E:\aide-sovereign-workbench\models\aide-house\` |
| Current adapter | `frontier-lora.gguf` (132 MB) | same dir |
| Baseline (raw engine, no harness) | **0.683 composite**, 11 PASS / 11 PARTIAL / 1 FAIL across 23 AIDE-world tasks | `E:\aide-sovereign-workbench\docs\evidence\capability-audit-cipher-4b.md` (re-run 2026-08-28) |
| Strong categories | A_code_gen (4/4), C_edit (3/3), G_longctx (2/2), F_openapi | battery report |
| Weak categories | D_tool (audit format mismatch — model uses AIDE's own grammar), E_reason, F_docstring, H_math (probability + two_sum) | battery report |
| Engine latency | prompt eval ~50 tok/s CPU, generation ~47 tok/s CPU (Vulkan not engaged) | `engine-aide-cipher-4b.err.log` recent runs |

## 2. Hard blockers on this device (2026-08-28)

The user wants fine-tuning, but two infrastructure issues block the standard Python-ML path:

1. **Python venv is broken.** `E:\felon_workspace\venv_trek\pyvenv.cfg` has `home = E:\Python310` — the venv's "home" is itself a Python install. Result: every `python.exe` (system + venv) dies with `Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding` because `sys.prefix` is the venv's `home` (a real Python install), not a copy. Even `-E` does not fix it because the failure is in the embedded Python startup, not in `PYTHONPATH`. The `failure-pythonpath-hijack` skill names this exact failure mode.
2. **AIDE `/api/chat` returns 500 "response violates the contract"** on both legacy (4777) and arch (4778) servers. Root cause: `openapi.json` has drifted from the runtime zod schema. Battery tests log `drift detected: regenerate with npm run contracts`. The model itself is fine; the route is broken. **This blocks "test in AIDE-world workflows" through the AIDE app** (engine-direct testing is unaffected).

**Until the venv is fixed and the contract is regenerated, the in-lane fine-tune path is blocked.** This skill documents the procedure that will run as soon as the venv is recreated.

## 3. Hardware constraints (GTX 1060 6GB, measured 8/7)

Per `model-scaling` skill (which is the device's hard limit):

| Config | Peak VRAM | Verdict |
|---|---|---|
| 4B BF16 base + LoRA r=16, target QKV+FFN | ~5.2 GB | tight, fits at the spill threshold |
| 4B 4-bit (NF4) base + LoRA r=16 | ~3.5 GB | fits with headroom |
| 4B FP32 base + LoRA | ~8 GB | **DOES NOT FIT** — fail |

**QLoRA (4-bit base) is the realistic path on this hardware.** `bitsandbytes` QLoRA on Pascal (CC 6.1) is dicey — no native 4-bit CUDA kernels, falls back to slower paths. If it fails, fall back to plain LoRA on BF16 base with aggressive gradient checkpointing and small batch.

## 4. Research base (verified 2026-08-28, sources named)

- `aide-cipher-house-model` skill: the lifecycle skill for the in-house model. Says "v1 = frontier_lora_full_f16.gguf (v1, operator-trained) + personal adapters (v2+, generated by sleep-time training)." So the v2+ adapter is *expected* to exist; this skill is the procedure to produce it.
- `model-scaling` skill: size ceiling, spill threshold, GQA divisibility. Hard size cap is 150M for Trek/queen-bee-v5 (NOT for cipher-4b). Cipher-4b is the existing 4B base; the cap doesn't apply.
- `device-training-1060` skill: hard rule #1 = FP32 ONLY. The cipher-4b base is Q8, so QLoRA 4-bit is a strict subset of the Q8 precision. Per `device-training-1060`, QLoRA is the only viable in-card training option.
- `training-sop` skill: AdamW(0.9, 0.95), wd 0.1, grad clip 1.0-5.0 calibrated by clip fraction, cosine schedule, effective batch ~32K tok/step, ONE epoch for SFT, replay buffer 30%.
- `aide-inference-control` skill: sampler profiles, presets, per-turn overrides. Sampler change is a config edit, not a training round; can be evaluated separately on the same battery.
- `golden-llama.cpp` repo + `transformers`, `peft`, `trl`, `datasets` are the standard stack. Quantization 4-bit NF4 is the bitsandbytes default; `bnb_4bit_quant_type="nf4"`, `bnb_4bit_compute_dtype=torch.bfloat16`.

## 5. SFT data generation (engine-direct, no Python needed)

**Approach:** use the working engine (raw, no AIDE route) to generate high-quality SFT pairs. The model is the teacher; the pairs are capacity-matched prompts that target the **weak categories** (D, E, F2, H) without breaking the **strong categories** (A, B, C, F, G).

Procedure:
1. Pick 20-30 prompts across the weak categories. Each prompt is short, unambiguous, has a verifiable correct answer.
2. Call the engine directly (`POST /v1/chat/completions`) with low temperature (0.0-0.2) to get deterministic outputs.
3. For prompts where the engine fails, write the CORRECT answer by hand (since the goal is the v2 adapter having better answers than v1, not the v2 adapter learning v1's failures).
4. Save as JSONL: `{"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}`.
5. Reject any pair where the assistant's output is empty, off-topic, or fails the must-contain check. Target ≥ 80% acceptance rate.

**Tool: `E:\pip_temp\opencode\sft_generate_cipher.mjs` (to be authored when this lane is unblocked).** Pattern: same as the capability_audit_cipher_4b.mjs script, but writes SFT pairs instead of scores.

## 6. Training procedure (LoRA on the 4B, BF16 base)

When the venv is fixed:

```python
# Pseudocode — full script written in step 7
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16,
                         bnb_4bit_use_double_quant=True)

base = AutoModelForCausalLM.from_pretrained("E:/aide-sovereign-workbench/models/aide-house/base.q8_0.gguf-base",
                                            quantization_config=bnb,
                                            device_map="auto",
                                            torch_dtype=torch.bfloat16)
base = prepare_model_for_kbit_training(base, use_gradient_checkpointing=True)

lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                  target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                                 "gate_proj", "up_proj", "down_proj"],
                  bias="none", task_type="CAUSAL_LM")
model = get_peft_model(base, lora)
# train
```

Hyperparameters (from `training-sop`):
- AdamW(0.9, 0.95), wd 0.1
- LR 2e-4 (LoRA standard, lower than full-FT)
- Cosine schedule, 100-step warmup, 1-2 epochs over the SFT data
- Effective batch 32K tokens/step (micro 1-2 with grad accum)
- Grad clip 1.0, monitored
- Replay buffer 30% from a general-instruction set (e.g., a small slice of the cipher-tokenizer or general SFT pool if available)

## 7. Convert LoRA → GGUF adapter (so llama.cpp can serve it)

Two paths:

A. **Adapter GGUF (preferred for AIDE's model-manager):** use `llama.cpp`'s `convert_lora_to_gguf.py`:
```
python E:/llama-cpp/convert_lora_to_gguf.py \
  --base E:/aide-sovereign-workbench/models/aide-house/base.q8_0.gguf \
  --outfile E:/aide-sovereign-workbench/models/aide-house/cipher_v2_lora.gguf \
  E:/path/to/trained/adapter
```
Then AIDE's `model-manager.mjs` already supports `--lora <path>` for hot-loading; the manifest's `lora_adapter` field points at the new file.

B. **Merged GGUF (alternative):** merge LoRA into base, convert HF model to GGUF, then quantize. Bigger single artifact; loses the ability to swap adapters. Not preferred for AIDE.

## 8. Evaluation gate (mandatory)

**No promotion without a passing evaluation.** Procedure:

1. Backup the current `frontier-lora.gguf` (production adapter) to `frontier-lora.v1.gguf`. This is the rollback.
2. Put the new `cipher_v2_lora.gguf` in place, OR temporarily update the manifest's `lora_adapter` field to point at the new file.
3. Restart the engine so it loads the new adapter.
4. Run `E:\pip_temp\opencode\capability_audit_cipher_4b.mjs` (the existing 23-task battery, raw engine).
5. Compare composite to baseline 0.683.
6. **Decision rule:**
   - Composite delta ≥ +0.02 AND no category regressed by more than 0.1 → **promote** (replace `frontier-lora.gguf` with v2, keep the v1 backup)
   - Composite delta ∈ [-0.02, +0.02] → **archive** (keep v1 as production, save v2 for analysis)
   - Composite delta < -0.02 → **rollback** (delete v2, v1 stays)
7. **Honest reporting:** write the verdict + raw numbers to `E:\aide-sovereign-workbench\docs\evidence\cipher-v2-eval.md`. No "looks better" — only "delta X on battery Y, with task scores Z."

## 9. Threat matrix (lane-level)

| Threat | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Python venv stays broken, training can't run | High | High | Report blocker; offer to do config-only work in parallel |
| QLoRA OOMs on 1060 despite planning | Medium | High | Fall back to plain LoRA on BF16; shrink batch to 1; enable grad checkpointing |
| New adapter overfits the SFT pairs (catastrophic forgetting) | Medium | High | Replay buffer 30% general data; ONE epoch; re-run baseline battery, not just narrow tasks |
| New adapter improves narrow tasks but regresses strong ones | Medium | Medium | Gate 0.02 composite + per-category cap; rollback if any strong category drops >0.1 |
| Engine can't load new adapter (format mismatch) | Low | Medium | Convert via the official `convert_lora_to_gguf.py`; smoke-test load before battery |
| GPU available during training conflicts with AIDE's serving | Medium | Medium | AIDE off during training; or train on a copy of base in CPU RAM (slow) |
| AIDE route 500 (contract drift) blocks "AIDE-world" tests | Certain | Medium | Test against engine directly (already done); fix AIDE separately (`npm run contracts`) |
| Public-facing copy rule violation in SFT-generated answers | Low | High | All SFT answers written by hand or filtered by must-contain; no auto-generated fallback |
| Frontend (UI) pretends v2 is "in production" before evaluation | Low | High | UI reads the manifest field; do not touch the manifest until the battery gate passes |

## 10. Dependencies

- Working Python venv with: `torch`, `transformers`, `peft`, `bitsandbytes`, `accelerate`, `trl`, `datasets`
- `llama.cpp`'s `convert_lora_to_gguf.py` (already on this machine at `E:\llama-cpp\`)
- A working `llama-server` binary (CPU build at `E:\llama-cpp\llama-server.exe` works)
- The 4B base model in HF format (or convertible from GGUF): need to convert `base.q8_0.gguf` → HF safetensors first via `llama.cpp`'s `convert_hf_to_gguf.py` in reverse, OR use `unsloth`/`ctranslate2` if the path is fragile
- The capability battery (already exists at `E:\pip_temp\opencode\capability_audit_cipher_4b.mjs`)

## 11. Sources (all verified 2026-08-28)

- AIDE baseline: `E:\aide-sovereign-workbench\docs\evidence\capability-audit-cipher-4b.md` (0.683 composite, 23 tasks, raw engine)
- Engine log: `E:\aide-sovereign-workbench\logs\engine-aide-cipher-4b.err.log`
- `aide-cipher-house-model` skill: `C:\Users\Grey_\.agents\skills\aide-cipher-house-model\SKILL.md` (lifecycle doctrine, v1→v2+ progression)
- `model-scaling` skill: `C:\Users\Grey_\.agents\skills\model-scaling\SKILL.md` (GQA divisibility, 5GB spill threshold)
- `device-training-1060` skill: `C:\Users\Grey_\.agents\skills\device-training-1060\SKILL.md` (FP32 only, QLoRA fallback)
- `training-sop` skill: `C:\Users\Grey_\.agents\skills\training-sop\SKILL.md` (AdamW, cosine, ONE epoch, replay 30%)
- `failure-pythonpath-hijack` skill: `C:\Users\Grey_\.agents\skills\failure-pythonpath-hijack\SKILL.md` (PYTHONPATH + broken venv failure mode)
- AIDE route 500: `E:\aide-sovereign-workbench\logs\proof-arch.log`, `run-arch.log` (contract drift messages)

## 12. When the venv is fixed, the exact next moves

1. `python -m venv E:\felon_workspace\venv_cipher` (fresh venv, no broken `home` setting)
2. `& E:\felon_workspace\venv_cipher\Scripts\python.exe -m pip install torch transformers peft bitsandbytes accelerate trl datasets`
3. Convert `base.q8_0.gguf` → HF safetensors (via `llama.cpp`'s reverse path or unsloth)
4. Write `E:\pip_temp\opencode\sft_generate_cipher.mjs` (Node.js, no Python) and produce `E:\pip_temp\opencode\sft_pairs_v1.jsonl` (~30 pairs, weak categories)
5. Write `E:\felon_workspace\train_cipher_v2.py` (QLoRA on the 4B)
6. Run training, save adapter to `E:\felon_workspace\cipher_v2_adapter\`
7. Convert to GGUF adapter via `convert_lora_to_gguf.py` → `E:\aide-sovereign-workbench\models\aide-house\cipher_v2_lora.gguf`
8. Restart AIDE's cipher engine, run the battery
9. Apply the gate (section 8) — promote / archive / rollback
10. Journal to `E:\aide-sovereign-workbench\AGENT_NOTES.md`

## 13. Out of scope for v1

- Fine-tuning a from-scratch 7B cipher (different lane, different skill, requires cloud GPU)
- Replacing the entire base model
- DPO/RLHF (would require preference pairs; not in scope)
- Multi-adapter orchestration (aLoRA, phase-specific personas)
- Real-time trajectory capture from AIDE agent loop (separate infrastructure work; not in this skill)
- Quantization-aware training (QAT) below 4-bit