git:20260907.619a061 to git:20260918.06c6019

---

name: gemini-robotics

description: Build low-latency robot agents with Gemini Robotics ER, including streaming perception, function calls, and guarded execution.

---

# Gemini Robotics

Keep Gemini on the perception and planning side of an actuator boundary. The

model may choose a capability; deterministic robot software validates and

executes it.

## Choose the endpoint from the interaction

- Use `gemini-robotics-er-2-streaming-preview` for a stateful Live API session

that receives text, JPEG frames, or audio and orchestrates robot tools with

low latency.

- Use `gemini-robotics-er-2-preview` for discrete embodied-reasoning requests

such as spatial analysis or offline video work. The standard endpoint does

not provide the Live API.

- Treat both model IDs and feature support as preview surfaces. Re-check the

current [model overview](https://ai.google.dev/gemini-api/docs/robotics-overview)

before changing dependencies or deployment assumptions.

## Make the stream an observe-act-observe loop

- Keep one `client.aio.live.connect` session open for the task and run a receive

loop that handles both model content and tool calls.

- Declare physical actions with `behavior: BLOCKING`. Execute each call through

the robot adapter, then manually return a `FunctionResponse` with the call ID,

name, and structured result using `send_tool_response`.

- Stream raw 16-bit, 16 kHz, little-endian PCM for speech input and explicitly

end finite audio with `audio_stream_end=True`. Send JPEG camera frames at no

more than the endpoint's current one-frame-per-second limit.

- A camera frame alone updates context but does not trigger reasoning. Pair it

with user audio/text, or use an intentional heartbeat prompt. Heartbeats are

turns and can interrupt generation.

- The streaming endpoint returns text, not synthesized audio. Route speech

through an independently replaceable TTS adapter or expose speaking as a

bounded tool.

Use Google's current [robotics streaming guide](https://ai.google.dev/gemini-api/docs/robotics-streaming)

for the volatile SDK syntax. Read [FAILURES.md](FAILURES.md) when a session

stalls, ignores images, overlaps actions, or never finishes an audio turn.

## Guard the robot outside the model

- Expose semantic capabilities such as named-waypoint navigation, bounded

inspection, or grasping a currently grounded object. Do not expose raw motor

commands, arbitrary poses, or unrestricted coordinates merely because the

function schema can describe them.

- Validate the tool allowlist, exact arguments, ranges, named resources, and

current perception-issued object IDs in ordinary code. A system instruction

and JSON schema improve model behavior but are not the safety boundary.

- Return completion, rejection, and failure states to the model. After motion,

send a fresh observation so the next decision is based on the resulting

scene rather than the pre-action frame.

- Give every long-running action cancellation and a deadline. On session or

tool timeout, invoke the robot's stop/cancel path independently of the model.

- Prove the same semantic contract against a fake adapter, representative

simulation, and finally supervised hardware. Keep simulator- and robot-

specific motion details behind the adapter.

For the evidence behind these choices and their current validation limits, read

## Done

- A complete user turn can stream input, execute a blocking semantic action,

return its result, and reason from a fresh observation.

- An undeclared or invalid action is rejected before reaching the robot SDK.

- Timeout and cancellation behavior is proven without depending on a model

response.