token-factory git:20260907.805c6d5A

Immutable. This exact content is served forever at /api/v1/blob/caa12f5a84398e1b.

---
name: token-factory
description: Use for zero-GPU hosted inference through Nebius Token Factory — captioning, batch text generation, and Cosmos physical-AI reasoning — including key setup, model selection, and the npa.workflow toolRefs that need no cluster.
---

# Token Factory (zero-GPU hosted inference)

Nebius Token Factory is an OpenAI-compatible hosted-inference API for open text
and vision models. It is the cheapest tier in the workbench that produces a real
artifact: **no cluster, no GPU, no provisioning**. Reach for it before standing up
anything, both for real work and to prove a toolchain end to end.

Full reference: `docs/workbench/token-factory.md`. Key setup only:
`docs/workbench/token-factory-key.md`.

## The credential is not a Nebius IAM token

This is the single most common failure. A Token Factory key is a separate
credential minted in the separate Token Factory console
(<https://tokenfactory.nebius.com/>). It is a long opaque token starting with
`v1.`, read from `NEBIUS_TOKEN_FACTORY_KEY` or `~/.npa/credentials.yaml`. Your
`nebius` CLI IAM token returns `403` here — having Nebius CLI access is not
enough, and no amount of re-authenticating the CLI will help.

Keys are shown once at creation. A project with no balance returns `402`/`403` on
inference even with a valid key.

```bash
npa workbench token-factory status    # connection settings, no network call
npa workbench token-factory verify    # live models call; non-zero on auth failure
npa workbench token-factory models    # what this key can actually reach
```

Run `verify` before a batch job and `models` before pinning a model name — model
availability is per-key, so a model in the docs may not be in your project.

Defaults: base URL `https://api.tokenfactory.nebius.com/v1/`, overridable with
`NEBIUS_TOKEN_FACTORY_BASE_URL`. Requests retry on 429 and 5xx.

## August 2026 migration

The [official notice](https://docs.tokenfactory.nebius.com/august-2026-deprecation-notice)
retires the old public text, vision, and reasoning defaults. The replacements
are Nemotron-3.5-Lightning for text and MiniMax-M3 for vision/reasoning.
See `docs/workbench/token-factory-deprecation-verification.md` for exact IDs,
live observations, API differences, vendor terms, and verification commands.
MiniMax-M3 is under the MiniMax Community License; hosted API access does not
establish the operator's commercial entitlement. No vendor weights are bundled.

Direct-output client calls disable thinking with model-specific template keys:
Lightning `enable_thinking=false`, MiniMax `thinking_mode=disabled`. Explicit
client `extra` values win. Agent reasoning turns enable thinking deliberately.

## Commands

### Current pricing and usage

When the task requires a price check, fetch the official public catalog at
<https://tokenfactory.nebius.com/api/public/models_info>. The field meanings
are documented at <https://tokenfactory.nebius.com/model-catalog.md>; the site
also exposes <https://tokenfactory.nebius.com/llms.txt> for discovery. An empty
rendered pricing page is not evidence that pricing is unavailable.

Match pricing to the exact catalog model/flavor. Record the retrieval date and
applicable input/output rates, including the serving mode. A public-price lookup
does not require credentials. When inference is requested, also verify that the
selected model is in the key-scoped model list. Public catalog presence does not
prove account access or account-specific billing.
If required pricing remains unresolved, stop before paid inference.

Report inference usage and finish reason only when the response exposes them.
Keep the coding agent's token usage separate. Requested output tokens are a
limit, not measured usage; missing provider usage means cost is unmeasured.

### Inference commands

Every command takes local paths or `s3://` URIs for both input and output, and
supports `--dry-run` (compute without writing the artifact) and
`--output text|json`.

**Caption images** — default model `MiniMaxAI/MiniMax-M3`:

```bash
npa workbench token-factory caption \
  --input-path s3://<bucket>/frames/ \
  --output-path s3://<bucket>/captions.json \
  --max-images 50 --max-tokens 512 --temperature 0.2 \
  --instruction "Describe the scene, objects, and any action."
```

**Batch text generation** over a JSONL/text prompt file — default model
`nvidia/Nemotron-3_5-Lightning`:

```bash
npa workbench token-factory generate \
  --input-path prompts.jsonl \
  --output-path s3://<bucket>/generations.jsonl \
  --max-prompts 0 --max-tokens 512 --temperature 0.7 \
  --system-prompt "<applied to every request>"
```

`--max-prompts 0` means all of them. Set a small non-zero value first: this is
the command that turns a typo into a large token bill.

**Batch text generation** — same prompt file, same `generations.jsonl`, batch
token rates, default model `openai/gpt-oss-120b`:

```bash
npa workbench token-factory batch-generate \
  --input-path prompts.jsonl \
  --output-path s3://<bucket>/generations.jsonl \
  --model openai/gpt-oss-120b --completion-window 24h

# or submit now, collect later
npa workbench token-factory batch-generate ... --no-wait
npa workbench token-factory batch-status --operation-id <id> --output-path <same> --wait
```

Reach for `batch-generate` over `generate` whenever nothing is waiting on the
answer, which is most bulk stages. Three properties are unique to it, and each
one has already cost real debugging time:

- **Batch routing is a per-model entitlement, unrelated to real-time chat.** Most
  models that serve `generate` are rejected for batch. Historical measurements across eight
  text models on one key, exactly one — `openai/gpt-oss-120b` — was batch
  routable; `meta-llama/Llama-3.3-70B-Instruct`, `Qwen/Qwen3-32B`,
  `Qwen/Qwen3-30B-A3B-Instruct-2507`, `Qwen/Qwen3-235B-A22B-Instruct-2507`,
  `google/gemma-3-27b-it`, `deepseek-ai/DeepSeek-V4-Flash`,
  `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B`, and `zai-org/GLM-5.1` were not. That
  is why `DEFAULT_BATCH_MODEL` is `openai/gpt-oss-120b` and not
  `DEFAULT_TEXT_MODEL`. Treat the routable set as per-key and verify on a couple
  of prompts before pointing a large run at a new model.
- **Batch is text-to-text only.** A vision model is rejected at submit with
  `Batch inference is only supported for text2text models`, so there is no batch
  captioning path; use `caption`, which is real-time.
- **The completion window is a deadline, not a latency.** Observed live: batches
  of one and three prompts sat `in_progress` with `completed: 0` for over an hour
  against a 24h window. Do not read a slow batch as a hung one, and never put
  `--wait` on a path that has its own timeout.

Where the failure reason actually lives matters. `GET /operations/{id}/errors`
returns a single empty string for a failed batch — useless. The real per-row
reason is in the batch record's error file
(`GET /batches/{id}` → `error_file_id` → `GET /files/{id}/content`, which
redirects, so redirects must be followed). `batch-generate` reads that file and
reports it, and also surfaces `request_counts` (`total`, `completed`, `failed`,
`invalid`) as the only genuine progress signal a pending batch offers.

**Distinguish a degraded platform from your own bug.** A batch that is accepted,
reports `in_progress` with rows validated (`total: 2, invalid: 0`), and holds
`completed: 0` is usually not your job's fault. Batch execution has been observed
unavailable while submissions were still accepted through the datasets/operations
route. The cheapest tell is `POST /v1/batches`, the OpenAI-compatible submit,
returning `403 Creating new batch job is temporarily unavailable`. Confirm it is a
server-side switch rather than your request by checking where the 403 lands: an
empty body returns `422` naming the missing fields, but a *valid* payload with a
genuinely uploaded `input_file_id` still returns `403`, so the gate sits ahead of
resource validation. Meanwhile the rest of the key stays healthy — real-time chat
on the same model, `POST /v1/files` with `purpose=batch`, `GET /v1/batches`, and
dataset create/delete all succeed — which rules out the key, the balance, the
model, and the payload. When you see this, stop debugging your spec, cancel what
you queued (`POST /batches/{id}/cancel`), and use `generate` until batch recovers.
Do not wait it out: the same 403 was still being returned eight days after it was
first seen, so "temporarily" can outlast any plausible stage timeout. Plan the
run on `generate` and re-probe later rather than leaving a stage parked.

**That 403 is not the quota rejection**, and conflating the two sends you down the
wrong path. The documented limits are 10 active batches per customer and 100
submissions per hour, a batch counts as active only until its processing
finishes, and rate limiting surfaces as `429`. So before blaming a limit, list
your batches (`GET /v1/batches?limit=100` — the default page is 10, which makes a
long history look artificially short) and count the non-terminal ones. All
terminal plus a 403 with no `x-ratelimit-*` headers means availability, not quota.

**Physical-AI reasoning over a scene** — default model
`MiniMaxAI/MiniMax-M3`. Point it at scene images and ask what a robot
should do:

```bash
npa workbench token-factory reason \
  --input-path s3://<bucket>/scene/ \
  --output-path s3://<bucket>/plan.json \
  --task "Describe this scene and give a step-by-step plan of action." \
  --max-images 8 --max-tokens 1024 --temperature 0.2
```

## In workflows

These run as CPU-only `npa.workflow` steps with no accelerator request. The
renderer injects `NEBIUS_TOKEN_FACTORY_KEY` for `workbench.token_factory.*`
steps, so pass it as a secret at submit time and never in the YAML:

```bash
npa workbench workflow submit <spec.yaml> --secret-env NEBIUS_TOKEN_FACTORY_KEY
```

toolRefs: `workbench.token_factory.caption`, `.generate`, `.batch_generate`,
`.reason`, `.triage` (digest a run's textual artifacts into a triage report).

`npa workbench token-factory workflow` prints exactly four:
`token-factory-caption.yaml`, `token-factory-generate.yaml`,
`token-factory-cosmos-reason.yaml`, and `vlm-eval-token-factory.yaml`. Several
more are checked in but not listed by that command, so do not treat its output as
the full inventory:

- `token-factory-batch-generate.yaml` — the batch-inference twin of
  `token-factory-generate.yaml`.
- `token-factory-parallel-fanout.yaml` — parallel batches.
- `token-factory-gate-loop.yaml`, `tokenfactory-cosmos-gate.yaml` — a hosted
  model as a gate that decides whether the pipeline continues.
- `tokenfactory-rollout-judge.yaml`, `tokenfactory-scene-to-rollout-judge.yaml` —
  reason about a scene, then judge a rollout against that plan.
- `tokenfactory-train-triage.yaml` — triage a training run's artifacts.

All live under `workflows/testing/`.

## Choosing between Token Factory and VLM eval

They overlap and are easy to confuse. `token-factory reason` **produces** an
analysis or plan. `vlm-eval` **scores** a rollout against a task and emits a
pass/fail gate with a threshold. When you want a judged number for a gate, use
`skills/tools/vlm-eval/SKILL.md` — and note it can consume a Token Factory
reasoning artifact directly through `--task-from`, so a judge scores against a
plan an earlier stage wrote rather than a hardcoded string.

## Gotchas

- **Canonical Sim2Real is scoring, not planning.** Stage 8 uses
  `MiniMaxAI/MiniMax-M3` as its only Stage 8 evaluator, on CPU with no
  self-hosted evaluator image. It sends a bounded, deterministic rollout-wide
  frame sample and requires event-local structured scores. Stage 9 compares the
  single evaluator result with the authoritative Stage 7 rollout set and rejects
  missing, duplicate, or extra evaluations before PPO. Preserve request
  IDs, token usage, latency, retries, and an authoritative returned cost or
  explicit null separately from model-agent tokens.
- **Sim2Real preflight is stronger than model listing.** Its submit and prepared
  action paths declare `NEBIUS_TOKEN_FACTORY_KEY` by name only, then require
  both key-scoped model availability and a minimal inference before provisioning.
- **Model availability is per-key.** Confirm with `models` before pinning a name
  in a spec; a spec that names an unavailable model fails at run time, not at
  validation.
- **`--max-images` and `--max-prompts` are cost controls, not correctness knobs.**
  Defaults are 50 images and unlimited prompts. Always bound the first run.
- **`--dry-run` still calls the model.** It skips writing the artifact, so it is
  not a free syntax check. For a free check, validate the spec instead.
- **Hosted inference is not a rendering or simulation path.** It has no access to
  your cluster, your PVCs, or a GPU; give it S3 or local inputs it can read.
- **The key belongs in credentials, not in a spec or a shell history.** Persist it
  with `npa configure --save-env-credentials` (atomic `0600` write, never
  printed).

## Verify

```bash
npa/.venv/bin/python -m pytest npa/tests/guardrails/test_skills_index.py -q
```