archify-tuning ยท diff
git:20261008.556fa52 to git:20261013.d5806a6
40 added, 53 removed. Audit A to A.
---
name: archify-tuning
- description: Tune Archify's generation quality from evidence. Generate real diagrams through subagents, read their execution traces, attribute each failure, fix the root cause and prove the change on a fixed benchmark of first drafts. Use when asked to improve how good, accurate or fast generated Archify diagrams are, to run a generation round, or to judge whether a renderer, diagnostic or documentation change made generation better.
+ description: Improve Archify diagram accuracy, visual quality and complete authoring speed with fewer repairs. Use for generation experiments or evaluating changes to layout, routing, diagnostics, authoring instructions and acceptance rules against actual diagrams and agent traces.
metadata:
internal: true
---
# Archify Tuning
- Users do not write diagram JSON; they describe a system and an agent authors it. Tune for that path: an agent's **first draft** should render as an accurate, readable diagram with few repairs. When the goals conflict, keep this order:
+ Users describe a system and an agent authors the diagram. Optimize that complete journey, from understanding the source to delivering a usable artifact. Keep this priority:
- 1. **Accurate**: no node, relationship, lane or condition is dropped or merged to pass a gate.
- 2. **Readable**: clear reading path, few bends and crossings, legible text.
- 3. **Fast**: few finalize failures and tool calls.
+ 1. **Accurate**: facts, relationships, directions, conditions, ownership and sources match the system.
+ 2. **Readable and attractive**: a clear reading path, distinguishable connections, legible text and balanced composition.
+ 3. **Efficient**: short complete delivery time, few failed repairs and little repeated work.
- A change that makes gates pass while the diagram reads worse, or that makes agents delete content more often, is a regression.
+ **Tests and validators serve these outcomes and are themselves candidates for improvement.** A passing check does not establish a good diagram; a failing check can expose an unsuitable rule. Judge the artifact and authoring experience before choosing what to change.
- ## Principles
+ ## Product requirements and adjustable rules
- - **The first draft is the unit of measure.** Bundled examples carry hand-tuned widths and routes, so they hide defects that every fresh draft hits.
- - Keep synthetic regressions in repository tests, separate from the frozen first-draft benchmark and its pass-rate denominator.
- - **Deleted content is the accuracy signal.** A pass reached by removing an error state or merging lanes is a failure that looks like a success.
- - **Separate route controls from placement.** A draft without route controls can still have authored positions, dimensions or a fixed canvas. Reproduce a failure with valid placement and constraints before attributing it to automatic layout.
- - **Early diagnostics must match the browser gate.** A compile-time check stricter than `browser-check` is a false failure; a looser one defers the failure to the most expensive stage.
- - **Traces explain more than outcomes.** Read where a diagnostic or document misled the agent, not only whether it passed.
+ - Protect source facts and supported user behavior. First drafts can be wrong: retain originals and record source-backed corrections or additions. Counts of deleted elements flag a semantic review; they do not decide whether an edit is correct.
+ - Treat spacing, preferred direction, bend counts and single-screen fit as design strategies to assess in context. An extra bend or readable scrolling can improve the whole diagram. Preserve explicit geometry and supported contracts unless an agreed change covers them.
+ - Before repairing a failed check, identify the user harm it is meant to prevent and whether that harm is present. For an unsuitable rule, revise its scope, threshold or severity; update or retire obsolete tests with the rationale and actual before/after evidence. Keep the original failure visible. Changing an expected value alone proves no benefit.
+ - Early diagnostics and browser checks should agree where they measure the same requirement and state their different scopes elsewhere. Neither is an unquestionable oracle. Taste that cannot be judged reliably can remain advice or visual review instead of a hard failure.
## Measures
- | Level | Measure | Source |
+ | Role | Measure | Evidence |
|---|---|---|
- | Primary | Benchmark first-draft render pass rate, per type | `replay.mjs`; excludes finalize and browser gates |
- | Primary | Content changes from first draft to final | `collect.mjs`, "lost" column, then semantic comparison |
- | Secondary | Finalize failures and tool calls per diagram | `collect.mjs` on a new round |
- | Visual | Crossings, bends, shared corridors and text size, then screenshots | Receipts, `shots.mjs` |
- | Guard | No benchmark draft that passed on base fails on head; every changed output inspected | `replay.mjs`, golden, Gallery |
+ | Outcome | Accurate, visually acceptable final delivery | Source audit and actual screenshots; separate machine pass from visual acceptance |
+ | Outcome | Complete elapsed time, failed repairs and repeated work | Timed trace from task start through reading, first draft, repairs, viewing and delivery |
+ | Process | First-draft quality and time to first usable result | Preserve the first attempt, every failure and the last verified usable artifact |
+ | Regression | Fixed-input pass/fail and changed output | `replay.mjs`, affected tests and visual comparisons; render pass excludes finalize/browser acceptance |
+ | Diagnostic | Bends, crossings, route length, text size and CLI time | Explain an observed benefit or tradeoff; none is a substitute for whole-diagram quality |
- A round on a new project discovers problems; it does not establish improvement or absence of regressions. Compare fixed inputs for renderer changes. Evaluate authoring-instruction changes with comparable new generations on the same project, preserving first attempts, repairs and failures.
+ Report delivered outcomes and failed/unfinished tasks separately. A failed run ending sooner is not a speed gain. Keep synthetic regressions separate from the frozen first-draft denominator; additions to the corpus form a versioned cohort. Repaired drafts never replace their original benchmark entries.
## Setup
- **Tuning home**: `$ARCHIFY_TUNING_HOME`, default `~/.local/share/archify-tuning`. It holds `journal.md`, `bench/`, `rounds/` and `replays/`. It stays outside the repository because drafts carry content from source projects, which may be private: never commit it.
- - **Journal**: read `<tuning-home>/journal.md` before starting. It records the branch under tuning, the latest benchmark result, the open problems with evidence and the decisions already taken. Update it at the end of every iteration.
- - **Head**: the worktree you change, in a durable directory. Follow [Local setup and verification](../../../CONTRIBUTING.md#local-setup-and-verification) for dependencies and test selection.
- - **Base**: a separate worktree of the comparison point, usually `dev`: `git worktree add <dir> origin/dev`. Pass its `archify` directory as `--base`.
+ - **Resume**: read the journal, available checkpoint and current goal state. Respect pauses and prior product decisions; continue an active round instead of duplicating it. Use the user's chosen diagram types; cover others when shared changes affect them.
+ - **Versions**: refresh the target remote revision and record source, base and candidate identities. Use separate durable worktrees or frozen snapshots. Record remote movement without overwriting a comparison baseline or a running author's runtime.
+ - **Evidence selection**: follow [Choose evidence by impact](../../../CONTRIBUTING.md#choose-evidence-by-impact). Reuse unchanged evidence with its original revision and scope.
- **Tools**: Node 22.13 or later and Chrome or Chromium for `shots.mjs` (`ARCHIFY_CHROME` overrides discovery). The collector accepts Devin CLI history or a [normalized trace export](references/trace-export.md) from another agent. It does not read other agents' private history databases.
- Run the commands below from the repository root. `round`, `collect` and `replay` print their usage when run without arguments; pass an explicit replay directory to `shots` when retaining an acceptance receipt.
-
## Loop
- 1. **Baseline.** `node .agents/skills/archify-tuning/scripts/replay.mjs --base <base>/archify` and record the per-type render pass rate and revision receipt in the journal.
- 2. **Generate.** Pick a project not used before (see the journal). `node .agents/skills/archify-tuning/scripts/round.mjs <project-path> [--types ...]` creates `<tuning-home>/rounds/<round>/` and `prompts.json`. Start each prompt as its own background subagent, unchanged. The Archify tree they use must stay untouched until every subagent finishes: experiment in a separate worktree meanwhile.
- 3. **Collect.** `node .agents/skills/archify-tuning/scripts/collect.mjs <round> --dump` reads Devin history; add `--trace-export <file.json>` for another agent's normalized export. It records tool calls, finalize outcomes, authored route controls, content-change signals and diagnostic codes; it also retains traces and first drafts. Cross-check counts with the subagents' reports. Compare labels, conditions, sources and meaning manually: a zero loss count is not proof of semantic preservation.
- 4. **Attribute** every failure to exactly one cause, using [Trace signals](references/trace-signals.md):
+ 1. **Bound the question.** Record one user-visible problem, the hypothesis, evidence that would support or refute it, and a time/attempt budget in the journal. Rank it by frequency, harm, likely benefit and ongoing complexity. Prefer existing evidence before starting another generation.
+ 2. **Choose the experiment.** Use fixed JSON for renderer/rule changes, independent comparable authors for instruction or complete-flow claims, and a new project to discover problems or check transfer. For author runs, timing or screenshot comparisons, read [Experiment design](references/experiment-design.md). Discovery is not proof of improvement.
+ 3. **Attribute.** Read the trace and inspect the complete diagram using [Trace signals](references/trace-signals.md). Record the primary cause, contributing factors and uncertainty. Distinguish source misunderstanding, placement, routing, diagnostics, instruction usability and unsuitable acceptance rules. Reproduce with valid placement before blaming the router; a rule being documented does not establish that it is useful or easy to apply.
+ 4. **Change the smallest responsible scope.** Prefer a minimal reproduction before a runtime fix. Trace other callers before changing a shared path. Evaluate main-path arrangement, interacting modules, branch/merge space and return paths together with routing. Keep experimental placements on copies; an automatic route fix must honor authored pins. Add guidance only for a demonstrated information gap, then evaluate whether authors benefit.
+ 5. **Verify the claim.** Select affected public-behavior tests and fixed inputs; add a regression for a demonstrated runtime defect when existing coverage does not capture it. Inspect changed outputs with the same viewport and state. Investigate each changed pass/fail result: distinguish real regressions, corrected false failures and intended rule changes with evidence. Unexplained failures remain open. A local improvement must survive the complete diagram and relevant contrasting cases before adoption. Rebuild generated outputs only when their authoritative inputs changed.
+ 6. **Conclude and consolidate.** Record confirmed facts, rejected approaches, remaining problems and one disposition: confirmed problem, verified local fix, experimental candidate, rejected approach or candidate ready for independent review. State limits on visual, speed and generalization claims. Prepare small reviewable changes once benefits are established; separate unrelated fixes and prototypes. Follow [CONTRIBUTING.md](../../../CONTRIBUTING.md) for integration; this skill grants no push, PR, merge, release or live-install authority.
- | Cause | Test | Response |
- |---|---|---|
- | Renderer defect | Valid placement and authored constraints reproduce an automatic layout failure | Fix the renderer |
- | Misleading diagnostic | Following the message fails or edits the wrong element | Fix its fix text or evidence |
- | Documentation gap | The agent needs a fact found only outside the required reads | Move the fact into a required read, with numbers |
- | Authoring error | The required reads already say it | Nothing, or one sentence |
+ ### When to stop an experiment
- 5. **Prioritise** by how many rounds and types show the problem, times its harm to accuracy, over its cost. A problem seen every round needs a systematic fix, not one patch per round. Record the ranking in the journal.
- 6. **Fix** the root cause once, in the shared path. Preserve authored geometry and pins; when the renderer cannot repair a draft itself, return a verified, actionable fix. Reject a change that only moves a metric, such as trading crossings for shared corridors.
- 7. **Verify.**
- - Add a minimal test that fails before the fix.
- - Replay: `node .agents/skills/archify-tuning/scripts/replay.mjs --base <base>/archify`. Any REGRESSED entry blocks the change; investigate every FIXED and changed entry. A render pass does not replace finalization or browser acceptance.
- - Run `node .agents/skills/archify-tuning/scripts/shots.mjs <replay-dir>` and inspect every changed output. Treat capture failures as missing evidence; compare identical inputs, viewport, theme and page state. Check that every connection remains distinguishable from the background, including beneath label masks and beside arrowheads, in the affected detail modes and themes. Use `--all` for a broader visual pass.
- - Find affected tests by searching for the changed functions and message text, not by file name, and run them. Regenerate examples and Gallery when output changes, then run golden.
- 8. **Record.** Commit code, regenerated outputs and documentation separately. Update the journal: benchmark result, problems closed and opened, approaches rejected. Integrate regularly through [CONTRIBUTING.md](../../../CONTRIBUTING.md) so the full suite and CI run. This skill does not authorise pushing, opening pull requests or installing Archify.
+ - At the budget limit, record the result and remaining uncertainty. Extend only for a concrete new hypothesis and a fresh bounded test.
+ - Stop or change focus when attempts repeat without new causal evidence, depend on project-specific coordinate search, improve selected lines while worsening the whole graph, or add complexity disproportionate to the benefit.
+ - Preserve the last verified artifact separately from an unsuccessful experiment. Ending an experiment does not complete the overall goal. Consolidate useful fixes before accumulating another batch.
- ## Diagnostics and documentation
+ ## Tools and reporting
- - A diagnostic names the element, the shortfall (target width, character budget or required gap) and one fix that works. It never points at a fix that cannot succeed.
- - A compact receipt lists how many diagnostics of each code it omitted.
- - Documentation answers questions traces show agents asking, in a required read, with measured numbers. Keep one paragraph per mode and every line under the 2,000-character read limit.
+ Run from the repository root; these tools supply evidence for the selected experiment, not a mandatory sequence every round. `round`, `collect` and `replay` print usage with no arguments.
- ## Anti-patterns
+ | Need | Command |
+ |---|---|
+ | Fixed-input comparison | `node .agents/skills/archify-tuning/scripts/replay.mjs --base <base>/archify` |
+ | Prepare author tasks | `node .agents/skills/archify-tuning/scripts/round.mjs <project-path> --types <selected-types> --archify <frozen-runtime>/archify` |
+ | Collect traces and content-change signals | `node .agents/skills/archify-tuning/scripts/collect.mjs <round> --dump` (add `--trace-export <file.json>` outside Devin) |
+ | Initial light-theme image comparison | `node .agents/skills/archify-tuning/scripts/shots.mjs <replay-dir> --height 900` |
- - Counting failures while ignoring deleted content.
- - Claiming an improvement from one round on a new project.
- - Patching the same problem a little every round.
- - Updating a pinned test value without checking that the diagram got better.
- - Growing documentation until agents cannot read it whole.
+ The collector does not establish semantic correctness or complete elapsed time; supplement it with source review and timing records. Preserve failure logs and unknown outcomes. Diagnostics should name the affected element and a verified repair, or explain why no repair was found. Keep authoring guidance concise and reachable in required reads; avoid accumulating advice that has no observed benefit.