--- name: domino-llm-gateway description: Call Large Language Models through Domino LLM Gateway 2.0, the centrally governed gateway extension an administrator installs in front of external providers and Domino-hosted models. Covers finding and detecting the gateway, authentication with the in-run access token (fresh per call) or a Personal Access Token outside Domino, model aliases and model discovery, the OpenAI-compatible and Anthropic-compatible paths from the OpenAI and Anthropic SDKs, Pydantic AI, LangChain and plain HTTP, streaming, attribution headers, refused calls (login redirect, grants, budgets, guardrails), pointing a coding assistant such as Claude Code or Codex at the gateway, and what the administrator configures. Use when a user mentions LLM Gateway, Gateway 2.0, a gateway alias or URL, asks how to call a model through the Domino gateway, how to run a coding assistant through it, why a gateway call was refused, or what the gateway administrator must set up. compatibility: Domino 6.3 and Domino Cloud, only where an administrator has installed the LLM Gateway 2.0 extension; availability depends on the installation, not the Domino version. Inside a Domino execution the bearer is the access token from DOMINO_API_PROXY/access-token (localhost:8899), re-read per call; outside Domino a Personal Access Token. Legacy user API keys are not used. --- # Domino LLM Gateway 2.0 Applies to Domino 6.3 and Domino Cloud where an administrator has installed LLM Gateway 2.0. It is an **extension**: installed separately, updated out of band, so whether it exists depends on the installation, not on the Domino version. It runs as a Domino App. If nobody has installed it, say so and use the `domino-llm-access` skill (a Domino-hosted endpoint or a provider SDK); do not try to install it. ## What the gateway is One base URL in front of many providers (external LLM vendors and Domino-hosted model endpoints). Provider keys stay in the gateway; callers authenticate with their Domino identity. Every request is tracked (user, Project, model, tokens, latency, estimated cost); administrators grant access per **model alias**, set budgets, and screen prompts and responses with guardrails. ## Find and detect the gateway The extension injects **nothing** into a Workspace: no URL, no token, no `ANTHROPIC_*` or `OPENAI_*` variable. Use `LLM_GATEWAY_URL` if the user or an administrator created it as a Project or user variable (not a `DOMINO_`-prefixed name; Domino reserves that prefix), else ask for the gateway App's URL (`https:///apps/`). Confirm it **with the bearer**: `GET /v1/models` returns JSON and `GET /health` returns `{"status": "ok"}`. HTML proves nothing: with a bearer `/` is the gateway's UI (200); without one every route is a 302 to the Domino login page. | Family | Path | Use with | |---|---|---| | OpenAI-compatible | `/v1` (`/models`, `/chat/completions`, `/responses`; `/embeddings` where an embedding alias exists) | OpenAI SDK, Pydantic AI, LangChain, any OpenAI-style client | | Anthropic-compatible | `/anthropic` (`/v1/messages`) | Anthropic SDK, Claude Code | ## Authentication Send `Authorization: Bearer ` on every request. - **Inside a Domino Workspace, Job or App:** read the access token from `$DOMINO_API_PROXY/access-token` (by default `http://localhost:8899/access-token`). It is short-lived, so re-read it before every request instead of caching it in a long-running process. In an App it is the publisher's token: every viewer's calls use the publisher's grants and budget, so tag them with `X-LLM-Tag-` to attribute usage. - **Outside Domino (laptop, CI):** a Domino bearer such as a Personal Access Token in a variable the user creates, for example `LLM_GATEWAY_TOKEN`. A Domino access token works from a laptop; a PAT or one of the gateway's own tokens (prefix `dgw_`, issued by an administrator) may be answered with the App login redirect or 401, in which case use a Domino access token. - Legacy user API keys are deprecated and not used. The Anthropic path needs the bearer in `Authorization`; a credential sent only as `x-api-key` is treated as no credential. ```python import os import requests from openai import OpenAI # not in the Domino Standard Environment: pip install openai GATEWAY = os.environ["LLM_GATEWAY_URL"].rstrip("/") def token() -> str: proxy = os.environ.get("DOMINO_API_PROXY") if proxy: # inside Domino: fresh per call return requests.get(f"{proxy}/access-token", timeout=5).text.strip() return os.environ["LLM_GATEWAY_TOKEN"] # outside: a Domino bearer the user stored def chat(messages, model="domino/example-alias", **kwargs) -> str: # an alias from /v1/models client = OpenAI(base_url=f"{GATEWAY}/v1", api_key=token()) return client.chat.completions.create(model=model, messages=messages, **kwargs).choices[0].message.content ``` ## Models `model` is an alias from `GET /v1/models` (OpenAI list shape, `data[].id`). Aliases are administrator-defined and may carry a prefix such as `domino/`; an alias's vendor name may also be accepted, but only listed aliases are guaranteed. `GET /v1/whoami` shows how the gateway resolved your identity. The `openai`, `anthropic`, `pydantic-ai` and `langchain-openai` packages are **not** in the Domino Standard Environment: `pip install` them in a Workspace to try; add them to the Compute Environment's Dockerfile for repeat use. ## Call it - OpenAI SDK, Responses API, embeddings, Pydantic AI, LangChain, plain HTTP and streaming: [CALLING-MODELS.md](./CALLING-MODELS.md). - Anthropic SDK: `Anthropic(base_url=f"{GATEWAY}/anthropic", auth_token=token())`, then `client.messages.create(model="", ...)`; `api_key=` silently breaks (see errors). - Attribution: `X-LLM-Tag-: ` headers (for example `X-LLM-Tag-team: research`) slice usage; they are labels, not access control. - Tracing: OpenAI and Anthropic SDK calls are autologged by `mlflow.openai.autolog()` / `mlflow.anthropic.autolog()`, so the `domino-genai-tracing` skill works unchanged. ## When a call is refused A **302 to the Domino login page** (HTML) is a missing, malformed or expired bearer: re-read the access token per call. A 302 to `/app-consent?appId=...` means the user has not consented to the gateway App acting for them: open the gateway URL in a browser once and accept. The Anthropic SDK with `api_key=` gets that HTML as a string and fails with `AttributeError: 'str' object has no attribute 'content'`; use `auth_token=`. `404 "Model '...' not found"` is an unknown alias: list `/v1/models`. A `4xx` naming a grant, budget or guardrail rule means no access, a spent budget, or a denied prompt or response: ask the administrator, or change the prompt. A 403 such as `Only a gateway admin can manage gateway admins` is an administrator-only route. `413` from the App proxy means the body is too large. Details: [CALLING-MODELS.md](./CALLING-MODELS.md#errors-to-expect). ## Coding assistants through the gateway Offer the user the choice; never route by default. Claude Code and Codex through the gateway, and the rule: [CODING-ASSISTANTS.md](./CODING-ASSISTANTS.md). ## What the administrator configures Providers, model aliases, access grants and budgets, guardrails, gateway tokens, and the usage and audit views, all in the gateway UI: [ADMIN-OVERVIEW.md](./ADMIN-OVERVIEW.md). Installing and updating the extension is the administrator's domain. ## Documentation - LLM Gateway 2.0: https://docs.domino.ai/cloud/platform-capabilities/features/llms/llm-gateway - Set up LLM access: https://docs.domino.ai/cloud/platform-capabilities/features/llms (6.3: replace `cloud` with `6.3`) - Authentication: https://docs.domino.ai/cloud/reference/api/domino-api-authentication