git:20261001.d9242c3 to git:20261014.681e9c1

45 added, 42 removed. Audit A to A.

---

name: domino-model-serving

- description: Deploy, invoke, and retire Domino model APIs and registered models via REST. Covers modelServing lifecycle, registered-models v1 vs v2 paths, MLflow tracking vs registry API, inference URLs vs management API, and GenAI endpoint vanity URLs. Use when automating model deployment, predictions, registry updates, or debugging stop/archive/delete behavior.

+ description: Automate Domino endpoints (Model APIs) and registered models over REST. Covers the /api/modelServing/v1 lifecycle (create an endpoint from a file or a registered model, add versions, poll deployment status, list, update replicas or hardware tier, archive, build and instance logs), registered models v1 vs v2 routes, MLflow tracking vs the registry API, management URL vs prediction URL, and sync Basic vs async Bearer prediction auth. Use when scripting deployments from CI, a Job or a notebook, registering a model version by API, or debugging stop, archive and delete behaviour. For UI publishing, the predict function, scaling and health settings use domino-model-endpoints; for drift and quality monitoring use domino-model-monitoring; for exporting a model out of Domino (image to your registry, NVIDIA Fleet Command, Snowflake UDF, SageMaker) or batch scoring Jobs use domino-model-delivery.

+ compatibility: Domino 6.3 and Domino Cloud. Requires DOMINO_API_HOST; in a run use DOMINO_API_PROXY or the localhost:8899 access-token endpoint, outside a run a Personal Access Token or service-account token. The modelServing beta visibility and collaborator routes exist only on Domino Cloud.

---

# Domino model serving (REST)

- Programmatic model deployment and inference on Domino. UI-focused endpoint monitoring stays in [model-endpoints](../model-endpoints/SKILL.md). Shared REST field catalogs live in [API-MODEL-SERVING.md](../python-sdk/API-MODEL-SERVING.md) and [API-MODELS.md](../python-sdk/API-MODELS.md).

+ Applies to Domino 6.3 and Domino Cloud.

- Authentication: https://docs.domino.ai/cloud/reference/api/domino-api-authentication . **Do not use API keys.**

+ Programmatic lifecycle for Domino endpoints (**Model APIs** in the API) and registered models. UI publishing, the prediction function, scaling and endpoint health are in `domino-model-endpoints`. Worked requests for every step: [LIFECYCLE.md](./LIFECYCLE.md); route catalog: [API-MODEL-SERVING.md](../domino-python-sdk/API-MODEL-SERVING.md).

- ## Configuration

+ ## Authentication and base URL

- | Context | Base | Authorization |

- |---------|------|----------------|

- | In-run, `DOMINO_API_PROXY` set | `{DOMINO_API_PROXY}` | None |

- | In-run, no proxy | `{DOMINO_USER_HOST or DOMINO_API_HOST}` | Bearer from `http://localhost:8899/access-token` |

- | Outside run | Public deployment URL | Bearer PAT or SA |

+ [domino-api-intro](../domino-api-intro/SKILL.md#authentication). Legacy API keys are deprecated; do not use them.

- Management paths include `/api/modelServing/v1/...`, `/api/registeredmodels/v1|v2/...`. Confirm with `GET {base}/api/modelServing/v1/modelApis?limit=1` returning HTTP 200.

+ Connectivity check: `GET /api/modelServing/v1/modelApis?limit=1` returns 200.

- ```bash

- curl ${TOKEN:+-H "Authorization: Bearer $TOKEN"} "$BASE/api/modelServing/v1/modelApis?limit=1"

- ```

+ ## Three surfaces (do not mix them)

- ## Three surfaces (do not merge)

+ | Surface | Base path | Purpose |

+ |---------|-----------|---------|

+ | Model Serving | `/api/modelServing/v1/modelApis` | Endpoints: create, versions, list, update, archive, logs |

+ | Registered models | `/api/registeredmodels/v2` (list, register); `/api/registeredmodels/v1/{modelName}` (get, patch, versions) | Registry records; `modelName` is the name, not an id |

+ | MLflow tracking | `MLFLOW_TRACKING_URI` in a run; `/api/2.0/mlflow/...` on the deployment URL with a bearer token | Experiments, runs, artifacts, the MLflow registry client |

- | Surface | Purpose |

- |---------|---------|

- | **Model Serving REST** | `/api/modelServing/v1/modelApis` deploy, versions, lifecycle |

- | **Registered models REST** | `/api/registeredmodels/v2` list/register; `/api/registeredmodels/v1/{modelName}` get/update/versions |

- | **MLflow tracking** | `MLFLOW_TRACKING_URI` / runs UI; not the same as registered-models REST |

+ MLflow logging or registration deploys nothing; deployment is a Model Serving call.

- Logging a run to MLflow does not replace registry or modelServing calls for deployment automation.

+ ## Management URL vs prediction URL

- ## Management vs inference URL

+ Management calls go to the base above under `/api/...`. Predictions go to the deployment URL: `/models/{modelApiId}/latest/model` (sync, Basic auth) or `/api/modelApis/async/v1/{modelApiId}` (async, Bearer). The `ModelApi` object has no `url` field; build the prediction URL from its `id` and the deployment URL shown on the endpoint's Overview tab.

- - Create/list/update: platform base + `/api/modelServing/v1/...` (or registered-models paths).

- - **Predict:** use the **`url`** field on the model API or version (often `.../models/.../latest/model`), on the deployment ingress. That path is not routed through `DOMINO_API_HOST` sidecar the same way as `/api/`.

+ ## Lifecycle

- GenAI: management API vs `https://.../endpoints/{vanity}` split. See [model-endpoints](../model-endpoints/SKILL.md) and the GenAI section in [API-MODEL-SERVING.md](../python-sdk/API-MODEL-SERVING.md).

+ 1. Train and, optionally, register (`POST /api/registeredmodels/v2`, or the MLflow client).

+ 2. `POST /api/modelServing/v1/modelApis` with `name`, `description`, `environmentId`, `isAsync`, `strictNodeAntiAffinity`, `environmentVariables`, `version{projectId, source, logHttpRequestResponse, monitoringEnabled}`; `source.type` is `File` (`file`, `function`) or `Registry` (`registeredModelName`, `registeredModelVersion`). Optional `hardwareTierId`, `replicas`.

+ 3. Poll the version: `GET .../modelApis/{modelApiId}/versions/{modelApiVersionId}` until `deployment.status` is `Running` and `isPending` is false (absent `deployment` means pending), then confirm `activeVersion.id` on the Model API. `Running` is not health: fetch `.../instanceLogs?maxResults=3000` (no `tail`), keep `model-` container lines, expect `WSGI app 0 (mountpoint='') ready` (`no app loaded. GAME OVER` is a crash loop), then one test prediction. Sync or commit before step 2 and pin `version.commitId`; the head lags uploads by seconds.

+ 4. New code: `POST .../modelApis/{modelApiId}/versions` (same `version` fields plus `shouldDeploy: true`). Poll that version, not the Model API; the old one stays active until the new one runs.

+ 5. Change replicas, hardware tier, name or environment: `PUT .../modelApis/{modelApiId}` (`name`, `description`, `replicas`, `environmentId` required; send current values back, including `resourceQuotaId`). Scale changes need a restart: UI, or internal `stopModelDeployment` then `startModelDeployment` (LIFECYCLE.md).

+ 6. Retire: stop the running version first (UI **Versions** > **Actions**, or `POST /v4/models/{modelApiId}/{versionId}/stopModelDeployment` with `{}`; non-public route; may change in a future Domino version), then `DELETE .../modelApis/{modelApiId}`: 500 while a version runs; after success `GET` is 404 and the endpoint leaves all lists.

+ 7. Verify each mutation with a `GET`; 2xx means accepted, not applied.

- ## Lifecycle (model APIs)

+ Logs: `GET .../versions/{modelApiVersionId}/buildLogs`, `exportLogs`, `instanceLogs`.

- Typical order:

+ `/api/modelServing/v1/modelDeployments` (with `start`, `stop`, `credentials`, `logs`, `versions`) is a separate entity: a registered model deployed onto an admin-defined external deployment target, SageMaker only. It is not a Model API; see [domino-model-delivery](../domino-model-delivery/EXPORT-TARGETS.md#amazon-sagemaker).

- 1. Train / register (MLflow and/or `POST /api/registeredmodels/v2/...`).

- 2. `POST /api/modelServing/v1/modelApis` (or new version on existing API).

- 3. Poll `GET /api/modelServing/v1/modelApis/{modelApiId}` until `status` is `Running` (or `Failed`); then read **`url`** for invoke.

- 4. **Stop** vs **archive** vs **delete** are different product operations; names and reliability differ by route.

- 5. Treat **DELETE** on deployments/model APIs as best-effort; verify resource gone before assuming cleanup.

- 6. **Stop** may be long-running and not idempotent; poll status instead of fire-and-forget retry loops.

+ ## Registered models

- Details and route list: [API-MODEL-SERVING.md](../python-sdk/API-MODEL-SERVING.md).

+ - List: `GET /api/registeredmodels/v2?projectId=&searchPattern=&maxResults=` (token-paginated with `pageToken`). Register: `POST /api/registeredmodels/v2` (`modelName`, `modelSource{sourceType, mlflowSource|huggingFaceSource}`, `create`). `POST /api/registeredmodels/v1/{modelName}/versions` is deprecated.

+ - Get and patch: `GET|PATCH /api/registeredmodels/v1/{modelName}` (`PATCH` takes `description` and `discoverable`; `GET` returns neither `discoverable` nor aliases); versions at `/api/registeredmodels/v1/{modelName}/versions/{version}`; endpoints serving a version at `.../versions/{version}/modelapis`.

+ - Aliases and tags are MLflow features: the MLflow client on `MLFLOW_TRACKING_URI`, or `/api/2.0/mlflow/registered-models/...` on the deployment URL; see `domino-experiment-tracking`.

- ## Registered model path key

+ ## Prediction auth

- `modelName` in `/api/registeredmodels/v1/{modelName}` is the **registered model name string**, not an opaque UUID. List/register often use **v2**; get/update/versions use **v1** with that name in the path.

+ Restricted **sync** endpoints require HTTP Basic with the endpoint access token as username and password. **Async** endpoints require `Authorization: Bearer <token>`. Unrestricted endpoints need neither. Tokens: the user's, or create one (`domino-model-endpoints` DEPLOY-ENDPOINT.md); never print it. Source: https://docs.domino.ai/cloud/platform-capabilities/features/model-deployment/domino-endpoint-authorization.

- ## Invocation auth

+ ## Which path applies

- Use the model API token documented on https://docs.domino.ai for the model endpoint invoke path. Older examples may show a basic-auth pattern with the token as both user and password; treat that as legacy and confirm in [API-SPECS.md](../domino-api-intro/API-SPECS.md) before hardcoding.

+ ```bash

+ curl -s "$DOMINO_API_HOST/version" # "version": "6.3.x" self-managed; "6.4.x" or later on Domino Cloud

+ ```

- ## Related documentation

+ Domino Cloud adds `/api/modelServing/beta/modelApis/{modelApiId}/visibility` (`Public`, `Discoverable`, `Private`), `.../access` and `.../collaborators` (roles `Owner`, `Editor`, `Operator`, `Viewer`); not on 6.3, where you set them in the UI (**Settings** > **Access and Sharing**).

- - [API-MODEL-SERVING.md](../python-sdk/API-MODEL-SERVING.md) - REST lifecycle and routes

- - [API-MODELS.md](../python-sdk/API-MODELS.md) - broader models API catalog

- - [model-endpoints](../model-endpoints/SKILL.md) - Grafana, Triton, UI deploy

- - Model registry (product): https://docs.domino.ai/cloud/platform-capabilities/features/model-registry

+ ## Related

+ - [LIFECYCLE.md](./LIFECYCLE.md): request and response examples for each step

+ - [API-MODEL-SERVING.md](../domino-python-sdk/API-MODEL-SERVING.md), [API-MODELS.md](../domino-python-sdk/API-MODELS.md)

+ - [domino-model-endpoints](../domino-model-endpoints/SKILL.md): UI, predict code, scaling, health, logs

+ - [domino-model-delivery](../domino-model-delivery/SKILL.md): model export, SageMaker, batch scoring

+ - Route discovery: [API-SPECS.md](../domino-api-intro/API-SPECS.md)

+ - Model registry: https://docs.domino.ai/cloud/platform-capabilities/features/models/register-and-govern/model-registry