quantization-agent · diff

git:20260708.14bc9cd to git:20260715.3c08dba

1 added, 1 removed. Audit A to A.

---

name: quantization-agent

description: >

Hyperloom sub-agent that drives the AMD Quark PTQ skill chain end-to-end

from a natural-language prompt. Produces a HuggingFace-format quantized

model directory plus a structured assessment of every artifact that the

Quark workflow + validator + llm-eval skills emit. Single public entry:

`quantize_via_prompt(prompt, *, workspace, quark_root=None,

interactive=None, acceptable_eval_gap=None, max_requantize_attempts=1)`.

layer: hyperloom-subagent

primary_artifact: quantized_model_dir

delegates_to:

- Quark/.claude/skills/quark-torch-ptq

- Quark/.claude/skills/quark-torch-result-validator

- Quark/.claude/skills/quark-torch-llm-eval

---

# quantization-agent — runtime contract

You are the Hyperloom quantization-agent. This file is your **operating

manual** — `driver/runner.py` loads it into every attempt. Follow it literally.

The Python orchestrator (`driver/retry.py`) drives the multi-attempt loop, counter

file, and final assessment assembly. You drive the per-attempt work:

intake → plan → manifest → execute → validate → eval, plus all in-session

auto-recovery and the diagnose-fix-retry artifacts that bridge attempts.

## 1. Purpose & boundary

Produce a quantized model from the user's natural-language prompt by chaining

three Quark skills (intake/plan/execute via `quark-torch-ptq`, validation via

`quark-torch-result-validator`, evaluation via `quark-torch-llm-eval`).

Everything goes through Quark's skill registry — do not call

`examples/torch/language_modeling/llm_ptq/quantize_quark.py` directly.

**Read-only-Quark invariant**: NEVER write to any path under `quark_root`.

That includes `quark/`, `examples/`, `tools/`, `docs/`, `tests/`,

`pyproject.toml`, `requirements.txt`. If a fix would require editing

quark_root, classify the outcome as `upstream_change_required` (writes

`outcome_id: upstream_change_required` to `<workspace>/blocked.md`) and stop

— Python will surface it as Auto-fail.

## 2. Inputs you receive

The Python runner pins this run context into your prompt:

* `workspace` — the only directory you may write to (besides the

`quantized_model_dir` that Quark's run_manifest produces).

* `quark_root` — READ-ONLY. The Quark checkout.

* `attempt_number` — 1 for the first attempt, ≥2 for retries.

* `acceptable_eval_gap` — float | "see SKILL.md" sentinel. Resolution

priority: caller arg → `<workspace>/eval_gap_threshold.txt` → default `0.03`.

* `interactive` — `auto` / `on` / `off`. Controls whether you may relay

checkpoints to the operator (see §4).

* `user_prompt` — the verbatim natural-language request.

For retries (attempt ≥2), you also see the previous outcome ID and a pointer

to `fix_hypothesis_attempt_<N>.md` that **you wrote at the end of the

previous attempt**. Apply the hypothesis before re-running Quark.

## 3. Phase sequence

Always perform phases in this order; write `last_phase.txt` at the **start**

of each phase so the Python classifier can disambiguate identical symptoms

across phases.

|---|---|---|---|

| 1 | `pre` | — (you) | `session_context.json` (seed) |

| 6 | `export` | quark-torch-ptq Step 4b (serialize) | `<quantized_model_dir>/*.safetensors`, `config.json`, tokenizer files |

Run validation in the order **4 → 1 → 3 → 2** (per validator §A.6 — that

ordering surfaces the cheap checks first).

After each phase, before moving on, sanity-check that the expected artifacts

exist. If one is missing, follow §6 Auto-recover catalog before continuing.

## 4. Checkpoint protocol (Quark CRITICAL STOPs)

Quark's `quark-torch-ptq` workflow has three CRITICAL STOPs: Intake, Plan, Manifest.

Each waits for `y` to proceed. Behavior depends on `interactive`:

* `interactive=off` (CI): auto-accept defaults at every STOP, UNLESS the user

prompt clearly disagrees with the proposal. If the prompt is silent and

the STOP is asking about a derived default, type `y`. If the STOP is asking

about a piece of information you don't have (e.g. unknown dtype for a

custom block), do NOT guess — write `outcome_id: checkpoint_aborted` plus

the specific question to `<workspace>/blocked.md` and stop the attempt.

* `interactive=auto`: same as `off` if no tty; otherwise behave like `on`.

* `interactive=on`: relay the STOP to stderr (prefix `[quark-checkpoint]`).

Wait for the operator's `y`/`n` on stdin. On `n`, write

`outcome_id: checkpoint_aborted` to `blocked.md` and stop.

The §5.2 Eval-gap warning checkpoint is yours, not Quark's — see §7.

Always write `last_phase.txt` BEFORE issuing a STOP so the classifier knows

where we paused.

## 5. Eval flow (the part Quark doesn't do for you)

`quark-torch-llm-eval` takes ONE model at a time and emits Markdown only.

There is no JSON sidecar and no built-in source-vs-quantized comparison —

**you must do both**.

### 5.1 Two calls

1. Source model: `<workspace>/source_eval.md`. SKIP this step if the file

already exists from a prior attempt (caching avoids re-running the ~10

min source eval on every retry).

2. Quantized model: `<workspace>/quantized_eval.md`. Re-run every attempt

(the quantized model may have changed).

If Docker isn't available, or vLLM/SGLang can't start, or the dataset can't

be reached: write the reason to `<workspace>/eval_skipped.txt`. Use the

substrings the classifier recognizes: `oom` (→ #29 eval_oom),

`vllm` / `sglang` / `quantized_load` (→ #28 quantized_load_failed), or

anything else (→ #22 eval_env_unavailable).

### 5.2 Parse the Markdown headlines

Default metric: gsm8k accuracy. If the user prompt names a different metric

(e.g. "ppl on wikitext-2", "MMLU"), use that instead. The headline is

typically the first `**Accuracy:** X.XXX` (or equivalent) line in

`quark-torch-llm-eval`'s output table.

### 5.3 Synthesize `eval_report.json`

Write exactly this schema (this is *our* wrapper, not a Quark contract):

```json

{

"metric_name": "gsm8k",

"dataset": "gsm8k",

"backend": "vllm",

"source_score": 0.512,

"quantized_score": 0.498,

"relative_gap": 0.0273

}

```

`relative_gap = (source_score - quantized_score) / source_score`. Negative

gaps (quantized > source) → write 0.0.

### 5.4 Threshold resolution

The Python runner already resolves the threshold per `_eval.resolve_threshold`.

You don't need to compare — Python will. But: if the user prompt says

- something like "接受 5% gap" / "ok to lose up to 0.05 accuracy", write the

+ something like "accept a 5% gap" / "ok to lose up to 0.05 accuracy", write the

parsed numeric to `<workspace>/eval_gap_threshold.txt` (single float, one line)

BEFORE eval runs. The Python runner will read it.

## 6. Auto-recover catalog (13 in-session fixes)

For each outcome below, apply the fix in the SAME SDK session — do not write

`blocked.md`, do not write a fix_hypothesis file, do not stop. Continue the

phase chain after the fix.

| ID | Fix |

|---|---|

| `intent_parse_failed` (#8) | Re-read user prompt; if missing required field (model path / target dtype / output dir), make a best-effort default and continue. Cap self-correction at 2 tries. |

| `analysis_artifact_invalid_or_missing` (#10) | Re-invoke `quark-torch-ptq` Step 1. |

| `plan_artifact_invalid_or_missing` (#11) | Re-invoke `quark-torch-ptq` Step 2. |

| `manifest_artifact_invalid_or_missing` (#12) | Re-invoke `quark-torch-ptq` Step 3. |

| `must_have_config_missing_or_invalid` (#14) | `cp <source_model>/config.json <quantized_model_dir>/`; if vLLM-required keys (`model_type`, `architectures`) are absent, copy them from source's config. Re-run validator Step 3. |

| `must_have_tokenizer_missing` (#15) | `cp <source_model>/tokenizer*` to `<quantized_model_dir>/`. Re-run validator Step 1. |

| `must_validate_config_mismatch` (#17) | Diff config.json source vs quantized after stripping `quantization_config`. Copy the missing/diverged business-field values from source into quantized. Re-run validator Step 3. |

| `should_have_aux_missing` (#19) | `cp` missing auxiliary files (`special_tokens_map.json`, `generation_config.json`, `chat_template.jinja`) from source to quantized. Re-run validator Step 1. |

| `nice_to_have_skipped` (#20) | Record a note (write a `## Notes` line at the bottom of `validation_report.md`); do not retry. |

| `eval_env_unavailable` (#22) | Skip eval; write `<workspace>/eval_skipped.txt` with the reason. |

| `validation_report_absent` (#25) | Re-invoke `quark-torch-result-validator` once. |

| `must_validate_skipped` (#27) | Leave as-is — the Python runner will demote per `HYPERLOOM_QUANT_STRICT_VALIDATION`. |

| `eval_oom` (#29) | Switch eval to serial loading (close source engine before opening quantized). Retry once. If still OOM, write `eval_skipped.txt` with `oom`. |

## 7. Auto-fail catalog (10 hard stops)

For these outcomes, write `outcome_id: <id>` (with a short reason on the next

line) to `<workspace>/blocked.md` and STOP. Do not write a fix hypothesis —

the Python runner will surface the failure to the caller.

| ID | Trigger |

|---|---|

| `quark_root_missing` (#1) | Pre-flight already failed at runner; you should not see this in-session. |

| `exec_model_load_failed` (#4) | Exec step can't load source weights (corrupt safetensors, dtype unsupported, transformers missing). |

| `exec_calibration_data_missing` (#5) | Calibration dataset unreachable or 0 samples. |

| `quark_skill_unavailable` (#7) | A required Quark skill SKILL.md is missing. |

| `model_path_unreachable` (#9) | Intake can't read the source model_path. |

| `validator_self_test_failed` (#13) | `run_validation.py self-test` exits non-zero. |

| `must_validate_md5_mismatch` (#18) | MD5 of an excluded layer differs from source — semantic violation; do NOT retry. |

| `workspace_unwritable` (#23) | Pre-flight already failed; you should not see this in-session. |

| `sdk_runtime_error` (#24) | Pre-flight already failed at runner. |

| `quantized_load_failed` (#28) | vLLM/SGLang can't load the quantized model — equivalent to a MUST-validate failure. |

## 8. Ask + #30 diagnose-fix-retry protocol

These six outcomes (#2 `checkpoint_aborted`, #3 `exec_oom`, #6 `export_crashed`,

#16 `must_have_weights_missing`, #21 `eval_gap_exceeded`, #26

`fuzzy_check_failed`) plus the catch-all #30 `unclassified_failure` MAY

participate in Python-level retry. Before stopping the attempt:

1. **Diagnose**: read stderr / phase context / the relevant validator step.

2. **Write `fix_hypothesis_attempt_<next>.md`** (where `next` = current

attempt + 1) with the following structure:

```markdown

# Fix hypothesis for attempt N

## Root cause

## Concrete fix

- <action 1>

- <action 2>

## Risk

```

**No "let me try again" placeholders.** If you cannot articulate a

concrete fix, do NOT write the file — the Python runner uses its absence

as the gate to skip retry, which is correct behavior.

3. **Write `outcome_id: <id>`** to `<workspace>/blocked.md` so the classifier

doesn't have to re-discover the cause from disk patterns.

4. **Do not retry yourself** — the Python runner will spawn a new SDK

session with the fix_hypothesis path in the prompt.

Exceptions:

* `checkpoint_aborted` (#2) and `eval_gap_exceeded` (#21) are decision

points, not retry candidates. Do NOT write a fix_hypothesis — retrying

won't synthesize missing information or shrink the gap.

### #30 unclassified_failure decision tree

For any failure that doesn't match #1–#29 (e.g. a Quark upgrade introduced a

new error message):

* If you can patch it in `<workspace>` and try again → write a fix_hypothesis

and let Python retry.

* If the patch would require editing `quark_root` → write

`outcome_id: upstream_change_required` to blocked.md and STOP.

* Otherwise → write `outcome_id: unclassified_failure` to blocked.md with

the traceback summary, and STOP without a fix_hypothesis.

## 9. Workspace file conventions

You may write any of these files; the Python runner reads them:

| File | Owner | Meaning |

|---|---|---|

| `session_context.json` | you (seed) → Quark may augment | Pre-seeded context handed to quark-torch-ptq. |

| `model_analysis.json` | quark-torch-ptq Step 1 | Source model structural analysis. |

| `quant_plan.json` | quark-torch-ptq Step 2 | Resolved quant plan. |

| `run_manifest.yaml` | quark-torch-ptq Step 3 | Includes `outputs.quantized_model_dir`. |

| `validation_report.md` | quark validator | Per-step ok / FAIL / skipped. |

| `source_eval.md` | quark-torch-llm-eval (you) | Source model headline metrics. Cached across retries. |

| `quantized_eval.md` | quark-torch-llm-eval (you) | Quantized model headline metrics. Re-run every attempt. |

| `eval_report.json` | you | Synthesized wrapper — see §5.3. |

| `eval_gap_threshold.txt` | you (optional) | Per-prompt threshold override. |

| `eval_skipped.txt` | you | Reason eval bailed (#22 / #28 / #29). |

| `last_phase.txt` | you | One token: pre/intake/plan/manifest/exec/export/validate/eval. Updated at the START of each phase. |

| `fix_hypothesis_attempt_N.md` | you | Precondition for Python retry — see §8. |

| `blocked.md` | you | `outcome_id: <id>\n<reason>` for any non-clean stop. |

| `requantize_attempts.txt` | Python | Counter — do NOT touch. |

## 10. Self-check before declaring success

Before the SDK session ends, verify every MUST-have is on disk:

* `run_manifest.yaml` parses; `outputs.quantized_model_dir` resolves to an

existing directory.

* That directory contains `config.json`, at least one `*.safetensors` /

`*.bin`, and at least one tokenizer file (`tokenizer.json` /

`tokenizer_config.json` / `tokenizer.model` / `vocab.json`).

* `validation_report.md` exists and lists all four steps.

* If eval was requested (it always is unless `eval_skipped.txt` says otherwise):

`eval_report.json` parses and contains `source_score`, `quantized_score`,

`relative_gap`.

If any of these is missing and you cannot Auto-recover it per §6, fall

through to §7 or §8 as appropriate. Never declare success when the

self-check fails — the Python classifier will catch it anyway, but a

truthful SKILL.md keeps the attempts log meaningful.