git:20261013.d3df086 to git:20261016.984bedb

---

name: fused-render-ai

description: Use when page or .py calls fused.ai (text/image/video/speech/transcribe/embed/decide/vision), picks model/provider, or AI call rejects.

---

# fused.ai

Eight verbs, one options object each, one shared result frame. `fused.ai(prompt)` / `fused.ai.text("hi")` don't exist — always `fused.ai.text({prompt})`.

Result frame, every verb: `{<payload>, provider, finishReason, warnings, usage, response: {id, modelId, timestamp}, providerMetadata}`. Read `response.modelId` (no top-level `model`), `usage.inputTokens` (camelCase, or null), `providerMetadata.local` for seed/sizes/paths. Never echo own request as caption.

| Verb | Required | Payload |

|---|---|---|

| `text({prompt})` | prompt | `text` |

| `image({prompt})` | prompt | `images: [{path, url, mediaType}]` |

| `video({prompt})` | prompt | `videos: [...]` |

| `speech({text})` | text | `audio: [{path, url, mediaType: "audio/wav"}]` |

| `transcribe({path})` | path | `text`, `segments: [{text, startSecond, endSecond, speaker?, words?}]`, `language`, `durationInSeconds` |

| `embed({texts})` | texts or paths | `embeddings: number[][]` (unit-length → cosine = dot) |

| `decide({state, questions})` | state, questions | `answers: {[id]: {type, confidence, ...}}` (probabilities, zero output tokens) |

| `vision({image, task})` | image, task (+ prompt unless caption) | `task`, `imageSize`, then `text` (+`reasoning`) / `objects` / `points` (0–1 coords) |

Universal rules:

- **Options closed** — unknown key rejects `bad_request` before any request.

- `provider`, `model`, `abortSignal` everywhere; `onChunk` = streaming (text, transcribe); `onProgress(job)` = job row (image, video, speech, transcribe).

- Relative file paths resolve beside page.

- Rejections: Error with `.type` (+ `.jobId` when job existed) — table below.

- Calls run concurrent — disable buttons in flight (image/video/speech/transcribe serialize silently server-side).

- Needs a local runtime, so it depends on WHICH export. **Hosted** export refuses any page containing the literal text `fused.ai.text(` — `fused.env` guard does NOT help; other verbs pass the check, then fail served. Only there is a local-only companion page the answer. A **`.fused` app file** deliberately allows `fused.ai.text(` (D388): it opens inside the recipient's own fused-render, and one without claude or a local model just gets the `ai_unavailable` rejection you already handle.

## Provider / model

Three tiers. `provider: "local" | "apple" | "claude"` pins; omitted → model decides: pinned apple ids (`afm-text`, `afm-speech`, `afm-embedding`) → apple; id with `/` or `.gguf` → local; `"sonnet"`/`"opus"`/`claude-*`/omitted → Claude CLI (text only; default tier haiku unless configured). `{provider: "apple"}` with no model = the tier's one id for the verb; a pinned id under another provider = `bad_request`. image/video/speech = local-only (apple rejects `unavailable`: no programmatic image model on macOS). `afm-speech` is Apple's speech-TO-text id, not `fused.ai.speech`. transcribe = local or apple. embed = local (apple `unavailable` in this build). decide and vision = local-only (claude/apple `unavailable`). **Take local ids from `fused.ai.models.catalog()`, treat as opaque** — never hardcode, never `split("/")`; apple ids are the three literals above and never appear in the catalog.

**apple tier** = macOS 26+, Apple Silicon, Apple Intelligence ON (System Settings) — else `ai_unavailable`/`unavailable` with the reason; OS still downloading the model → `model_loading` + `err.jobId` to watch. Nothing to download, nothing leaves the Mac. Small model (~3B on 26), ~4k-token context incl. history/systemPrompt → keep prompts short or `ai_error` on overflow. Guardrails refuse arbitrarily → `finishReason: "content-filter"` with the text so far, not an error; reword and retry. `usage` null on macOS 26. `providerMetadata.apple: {os, modelGeneration, refusal?, restarted?}` (text), `{locale}` (speech, also in the transcript file).

`fused.ai.models`: `list()`, `load(id, {capability})` / `download(id, {capability})` → `{jobId}` (**always pass capability** — wrong runner otherwise), `unload({capability})` (by capability, not id), `fused.ai.cancel(capability?)` (default text-generation — name capability to stop anything else).

## Text and cold start

**text** options: `prompt, provider, model, systemPrompt, effort ("low"|"medium"|"high"|"xhigh", Claude only; default "low" = no extended thinking — ask for more explicitly), history, raw, images, temperature/maxTokens/topP, thinking, onChunk, abortSignal`. Tier split: `history`/`raw` local-only, **reject `bad_request` on Claude**; `images` works on local vision models AND Claude (PNG/JPEG/GIF/WebP, ≤3.75 MB each, HEIC → convert with `sips` first); apple honours `history`, rejects `raw`, rejects `images` (until macOS 27 image input). `temperature`/`maxTokens`/`topP` on Claude, `effort` on local, `effort`/`topP`/`thinking` on apple dropped with `warnings[]` entry, call succeeds. `history` = `[{role: "user"|"assistant", content}]` — push user+assistant pair after each turn. Feed aggregates, not datasets — compute in Python first. Structured output: demand strict JSON, strip fences, `JSON.parse` in try/catch. Vision: `images` on Claude (any model) or a local model with `acceptsImage`.

**`thinking`** (boolean, local text models only — mlx and llama.cpp/GGUF): tri-state, unset/`true`/`false`. **Local models default to thinking ON.** Most models don't need this touched. Set `thinking: false` when a model's own card says thinking must be off to get usable output — `mlx-community/S1-mini-MLX-4bit` (a transcript normalizer) is the live example: leave `thinking` unset and it degenerates into repeating filler; pass `{thinking: false}` and it answers correctly. Dropped with a `warnings[]` entry on Claude and Apple (they have no such flag) — the call still succeeds, it just did not honour the setting. **Neither local runner strips `<think>…</think>`** — a reasoning model's `result.text` now starts with a visible think block by default, and reasoning tokens count against `maxTokens` (1024 default), so a long think can hit `finishReason: "length"` before the answer. Strip the block, raise `maxTokens`, or pass `thinking: false`.

**Cold start**: first text/embed/decide call naming non-resident local model rejects `model_loading` — download already started, `err.jobId` = it. `fused.watchJob(err.jobId)`, then retry. Not a failure. Image/video/speech/transcribe load inside own job instead (`done`/`total` null while weights arrive — guard division).

## Images: `fused.ai.image({prompt, ...})`

Options: `width`/`height` (1024, 256–2048, snapped down to /16), `steps` (28), `guidance` (4.0), `seed` (random; always returned — re-render with it), `image` (page-relative edit base — only some engines: check `acceptsImage`, else `bad_request` naming engine; defaults become file's size, steps 4, guidance 1.0), `mask` (page-relative mask PNG, **white = repaint, black = keep**, any size — resized to the render; needs `image`; mflux + a FLUX.1 Fill model only — omit `model` and the route picks the catalog's Fill row, or name `mflux-community/flux-1-dev-fill-mflux-q4`; pixels outside the mask come back untouched, so no compositing on the page; defaults: base's size, steps 28, guidance 30), `onProgress`, `abortSignal`. No negative prompt / batch / LoRA / strength. A Fill model refuses a call with no `mask`; a Klein model refuses one with it.

Resolves `images: [{path, url, mediaType: "image/png"}]`, `usage: {imagesGenerated: 1}`, `response.id` = job id, actual render in `providerMetadata.local: {seed, width, height, steps, guidance, prompt, previewPath}` (+ `image` on edits, + `image`/`mask` on inpaints). **No live thumbnail on an inpaint**: the Fill model has no preview fit, so the file `onProgress`'s `previewUrl` names is never written — ignore `previewUrl` when you passed `mask` (an `<img>` on it 404s), read `done/total` only. Read sizes off reply. `onProgress` job carries `done/total` (denoising steps) + `previewUrl` (~32px thumb, null on last tick — swap to `images[0].url`). One PNG per call; 900 s cap; renders serialize silently.

## Video: `fused.ai.video({prompt, ...})`

Apple Silicon only, no fallback — before drawing UI check `catalog()` `text-to-video` row `available`, take limits from its `videoTraits`. Options: `width`/`height` (704/480, snapped /32), `frames` (97; `1+8n` grid, rounded **up**), `steps` (8), `seed`, `image` (conditions frame 0; only if `videoTraits.supportsImage`), `onProgress`, `abortSignal`. No guidance, no preview.

Resolves `videos: [{path, url, mediaType: "video/mp4"}]`, `usage: {videosGenerated: 1}`, `response.id` = job id, `providerMetadata.local: {seed, width, height, frames, steps, prompt, image?}`. Up to 2 h; serializes; `fused.ai.cancel("text-to-video")` keeps model warm.

## Speech: `fused.ai.speech({text, ...})`

Qwen3-TTS, Apple Silicon only; check the `text-to-speech` row's `available`. Options depend on the model's `voiceMode`: `preset` = `voice` (from `voices`; default and echo = first listed) + optional `instruct` style; `clone` = `refAudio` (page-relative or absolute, 10-30 s) + `refText` (its words), both required; `design` = `instruct` describing the voice, required. Wrong option → `bad_request`. `language`: Qwen names (`english`, `chinese`, ...), not ISO codes; default `"auto"`. Also `model`, `provider`, `onProgress`, `abortSignal`.

Resolves `audio: [{path, url, mediaType: "audio/wav"}]`, `usage: {audioGenerated: 1}`, `response.id` = job id, `providerMetadata.local: {text, language, voice?, instruct?, refAudio?, refText?}`. Long text is made in parts (paragraphs at blank lines, at most 600 characters each); `onProgress` `done/total` counts parts. Switching between models reloads, so group lines by model.

## Transcribe, embed and decide

**transcribe** — `language, task ("transcribe"|"translate"), initialPrompt, vad, diarize, speakers, words, onChunk (per segment), onProgress (seconds)`. Default model = smallest; pass bigger for accuracy. Output persisted under `~/.fused-render/ai/transcripts/` (paths in `providerMetadata.local`). Failed long run keeps `err.outputPartial` (.partial.jsonl) — rows there are RAW `{start, end, text}`, not the resolved `startSecond`/`endSecond` shape. Word timings ±200 ms — don't cut clips on them; `translate` returns no words. One at a time, second queues. **apple** (`provider: "apple"` / `model: "afm-speech"`): `language` = ISO code or BCP-47 tag, mapped to Apple's ~30 locales (unsupported → `bad_request` listing them; absent → system locale, no auto-detect); `task: "translate"` and `diarize` → `bad_request`; `initialPrompt`/`vad` → warnings; `words` honoured; first use of a locale downloads its model (row shows it). Segments are utterance-sized, often one per sentence. Fast: 2-3× Whisper.

**embed** — `texts` OR `paths` (≤64), `kind: "query"|"document"`. Prose encoder: `kind`, no paths. Dual encoder (CLIP/SigLIP): `paths`, no kind. EmbeddingGemma 2 (`mlx-embed`, Mac): both — `kind` on `texts`, `paths` without it, one shared 768-dim space — and its `paths` may be pictures, SOUND (wav/mp3/flac/ogg natively; m4a/aac/opus only with ffmpeg on PATH) or VIDEO (mp4/mov/mkv/webm…), any mix in one list, sorted by extension; so a phrase ranks recordings and clips, and a recording ranks pictures. Not PDF/docx — turn those into text or page images first. Other dual encoders refuse a sound/video path by name (400 names the file). Catalog flags: `acceptsPaths`, `acceptsAudio`, `acceptsVideo`. Index with `"document"`, search with `"query"` — mixing silently drops recall. **Store `response.modelId` with vectors, compare before searching** — cross-model cosine = plausible garbage; dimensions don't catch it.

**decide** — `fused.ai.decide({state, questions, model, provider, abortSignal})`. Not generative: Laya (typed-decision encoder, `laya-mlx`, Apple Silicon only) reads a `state` and answers typed questions with calibrated probabilities. `state` = string | object | array (a message, a JSON record, a conversation). `questions` = object keyed by YOUR ids, each `{type, instructions, criteria}`: `"choice"` — `criteria` array of labels OR object label→description, answer `choice` + `probabilities: {label: p}`; `"score"` — `criteria` ordered rubric array (low→high), answer `score` = expected ZERO-BASED level (float), `legend: {"0": ..}`, `probabilities: {"0": p}`; `"noul"` — no criteria, answer `noul` = P(true). Every answer: `type`, `confidence`, `action: {actProbability}`. `confidence` is NOT P(picked): it is 1 − normalized entropy of the whole distribution (1 = all mass on one option, 0 = flat over the k options) — show the picked option's own probability when you mean "how likely"; for `noul` it equals `max(p, 1−p)`. `usage.outputTokens` always 0; `providerMetadata.local` = `{runner, seconds}` (model time only); no `onChunk`, no `onProgress` — sync, ~ms per question once resident (cold = `model_loading`, watch `err.jobId`, retry). **Context is small**: 512 tokens (English) / 1024 (multilingual) SHARED by instructions + criteria + state — too many labels/levels for the budget → `ai_error` ("too many options for the token budget"); too long a state is silently cut from the TAIL, reported per question in `warnings[]` (`{type: "other", message: "…state was cut to N tokens…"}`) — `usage.inputTokens` at the ceiling is the other tell. Shortlist labels first; keep instructions one sentence; put the decisive part of the state FIRST. Not for arithmetic, counting, date compare, multi-hop lookup. Ids from `catalog()` under `text-classification`, never hardcode; multilingual is the bare default, English `recommended`.

## Vision

`fused.ai.vision({image, task, prompt, length, reasoning, maxObjects, model, provider, abortSignal})` — local, Apple Silicon, synchronous like decide (cold → `model_loading` + `err.jobId`). `task`: `caption` (no prompt; `length`), `query` (`prompt`; `reasoning`), `detect` (→ `objects: [{xMin, yMin, xMax, yMax, label, score}]`, score null if unscored), `point` (→ `points: [{x, y, label}]`). Coordinates are 0–1 of the upright image: draw with an SVG `viewBox="0 0 1 1" preserveAspectRatio="none"` over the `<img>`. Each model answers the `tasks` on its catalog entry; omit `model` and the first that can is used. Grounding DINO (detect, scored; comma-separate labels) or Moondream 2 (all four; follow-ups on one image ~0.3 s). Python: `fused_ai.vision(image, task, prompt, max_objects=, model=)`.

## From Python

`import fused_ai` works in any `runPython` `.py`, no path setup. A daemon (`main =` or `daemon =`) must add the path itself, see fused-render-background-apps. Same option names, blocking; all verbs except video (speech: `ref_audio=`, `ref_text=`). `fused_ai.text(prompt, ...)` → str; `stream(...)` yields; `decide(state, questions, model=, provider=)` and the rest return frame (dict). `wait=False`, `on_progress=`, `timeout=` on job-backed calls. Catch `ServerNotRunning`, `AiError` (`.type`). Outside server: `sys.path.insert(0, server.json["shared"])`. No live UI through it — `runPython` returns once, no streaming.

## Errors

| `.type` | Meaning |

|---|---|

| `model_loading` | Not resident; load started — watch `err.jobId`, retry |

| `model_slot_conflict` | TERMINAL — never retry. Two callers want different models of one capability; the supervisor stopped evicting rather than loop forever. `err.capability`/`.models`/`.resident`/`.held` name the standoff. Resolves only via the user (Models dock, or `POST /api/ai/runtime/conflicts/resolve`\|`/release`) |

Every caller is identified by `X-Fused-Requester`/`X-Fused-Requester-Label` (percent-encoded; apps go through `fused.ai.*`/`fused_ai.py`, which set these for you) — this is what lets the supervisor tell your own router↔generator alternation apart from two different callers actually fighting over a slot. A conflict is between CALLERS, not models: the same model id loaded twice by two different callers can still conflict.

| `ai_unavailable` | claude CLI missing / local worker won't start — friendly state, not overlay |

| `unavailable` | Machine can't (no runner, needs Apple Silicon, claude on non-text verb, apple on image/video/speech/embed/decide/vision or below macOS 26 / Apple Intelligence off) |

| `bad_request` | Call wrong — read `.message` |

| `ai_error` | Ran, failed (bad id, OOM, crash, render past cap) |

| `timeout` | Text: 600 s on Claude, 900 s local (image 900 s, video 2 h, transcribe far longer) — size a UI timeout off the tier you're calling, not off 600 |

| `cancelled` | abortSignal / cancel / row ✕ — not a failure |

Debug machine (not page): `which claude`; `GET /api/ai/runtime`; `POST /api/ai` with `X-Fused: 1` (required on every POST).

Full option semantics + edge cases: `fused_render/static/runtime.js` header comment (SPEC §40 refs); Python signatures: `fused_render/templates/shared/fused_ai.py`.