diff --git a/docs/providers/google_ai_studio/realtime.md b/docs/providers/google_ai_studio/realtime.md index a89d69fe2..f491dba2e 100644 --- a/docs/providers/google_ai_studio/realtime.md +++ b/docs/providers/google_ai_studio/realtime.md @@ -192,8 +192,9 @@ model_list: api_key: os.environ/GEMINI_API_KEY litellm_settings: - # Required for tool calling with Gemini Live: - # defer setup until client sends session.update (with tools) + # Optional. session.update events sent before any content already build the Gemini + # setup (with tools); this buffers audio and text sent before session.update so that + # update still applies gemini_live_defer_setup: true ``` @@ -205,13 +206,13 @@ python test_realtime_tool_calling.py ## Limitations - Does not support audio transcription. -- Session config updates after the first `session.update` are ignored (Gemini setup is one-time per connection). +- Every `session.update` sent before the first content frame is merged into the Gemini setup; a `session.update` sent after content is ignored (Gemini setup is one-time per connection). ## Precaution - Tool calling will not work unless you send `session.update` first with your `tools`. - Send it as the first config message for that websocket session. -- `gemini_live_defer_setup` defaults to `false` for backward compatibility. +- `gemini_live_defer_setup` defaults to `false`. It is only needed when the client sends audio or text before its `session.update`. ## Supported OpenAI Realtime Events diff --git a/docs/providers/vertex_realtime.md b/docs/providers/vertex_realtime.md index 06505843a..8e852cdab 100644 --- a/docs/providers/vertex_realtime.md +++ b/docs/providers/vertex_realtime.md @@ -163,8 +163,10 @@ Key settings for audio: - Server VAD is enabled by default with 800 ms silence threshold ```python -# session.update with server VAD — the proxy ignores this for Vertex AI -# because VAD is already configured in the initial setup message. +# Send session.update before any content: LiteLLM merges every session.update it +# receives before the first conversation item or audio frame into the one Vertex AI +# setup, so they set the modalities, VAD, and tools for the whole session. A +# session.update sent after content is ignored. await ws.send(json.dumps({ "type": "session.update", "session": { @@ -320,7 +322,7 @@ python test_realtime_tool_calling.py |---|---| | `input_audio_buffer.append` | Forwarded as `realtime_input.audio` | | `conversation.item.create` | Forwarded as `realtime_input.text` | -| `session.update` | Silently ignored — Vertex AI does not support mid-session reconfiguration | +| `session.update` | Every one sent before the first content frame is merged into the Vertex AI `setup` and acknowledged with `session.updated`. Ones sent after content are ignored: Vertex AI accepts one setup per connection | | `response.create` | Silently ignored — Vertex AI responds automatically after each turn | **Vertex AI → Proxy (→ Client)** @@ -336,11 +338,13 @@ python test_realtime_tool_calling.py ## Limitations -- `session.update` is not forwarded (Vertex AI only accepts one setup message per connection). +- Session configuration is fixed once content flows. LiteLLM sends the Vertex AI setup with the client's first content frame (a conversation item, audio, or a tool result): every `session.update` received before it is merged into that setup, a session that sends content first gets the model's default (audio) setup, and a `session.update` sent after content is ignored because Vertex AI accepts one setup per connection. +- A client that streams audio before its `session.update` gets the default setup. Set `gemini_live_defer_setup: true` to buffer that audio until the `session.update` arrives. +- `output_modalities: ["text"]` on an audio-only model (for example `gemini-live-2.5-flash-native-audio`) is downgraded to audio, and the proxy logs a warning saying so. - Audio transcription requires `outputAudioTranscription: {}` to be set in the initial setup (done automatically by LiteLLM). ## Precaution - Tool calling depends on `session.update` with `tools`. - If you skip `session.update`, tool calls will not be triggered. -- `gemini_live_defer_setup` defaults to `false` for backward compatibility. +- `gemini_live_defer_setup` defaults to `false`. It is only needed when the client sends audio or text before its `session.update`. diff --git a/docs/proxy/config_settings.md b/docs/proxy/config_settings.md index 2712fb97a..18bfbfc24 100644 --- a/docs/proxy/config_settings.md +++ b/docs/proxy/config_settings.md @@ -1222,7 +1222,7 @@ router_settings: | LITELLM_TPM_TOKEN_RESERVATION_ENABLED | Default `true`. Set to `false` to disable pre-request TPM reservation in the v3 rate limiter and apply actual usage after each real-time request completes. This removes one Redis operation per request, but concurrent requests may temporarily exceed the TPM limit. This setting does not apply to `POST /v1/batches`, which uses a [separate input-file limiter](../batches#how-rate-limiting-for-batches-api-works). See [Estimated output tokens](./users#estimated-output-tokens-requests-without-max_tokens). | | LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES | When set to "true", routes OpenAI /v1/messages requests through chat/completions instead of the Responses API for Anthropic models. Can also be set via `litellm_settings.use_chat_completions_url_for_anthropic_messages` | LITELLM_ROUTE_ALL_CHAT_OPENAI_TO_RESPONSES | When set to "true", routes all OpenAI /chat/completions requests through the Responses API bridge. Recommended for OpenAI models. Can also be set via `litellm_settings.route_all_chat_openai_to_responses` -| LITELLM_GEMINI_LIVE_DEFER_SETUP | When set to "true", defers Gemini/Vertex Live setup until the client sends `session.update` (required for runtime tool injection). Default is "false" for backwards compatibility, which auto-sends setup on connect. Can also be set via `litellm.gemini_live_defer_setup` +| LITELLM_GEMINI_LIVE_DEFER_SETUP | When set to "true", buffers audio and text the client sends before its `session.update` so that update still builds the Gemini/Vertex Live setup. Default is "false": the setup is sent with the client's first content frame, built from the `session.update` events received before it and from the model defaults when there were none. Can also be set via `litellm.gemini_live_defer_setup` | LITELLM_USE_LEGACY_INTERACTIONS_SCHEMA | When set to "true", uses the legacy Google Interactions API schema (`outputs` array, `2026-05-07` revision) instead of the new schema (`steps` array, `2026-05-20` revision). The legacy schema will be sunset on June 8, 2026. Can also be set via `litellm_settings.use_legacy_interactions_schema` | LITELLM_USER_AGENT | Custom user agent string for LiteLLM API requests. Used for partner telemetry attribution | LITELLM_WORKER_STARTUP_HOOKS | Comma-separated list of `module.path:function_name` callables to run in each worker process during startup. Runs early in the worker lifecycle (before config/DB loading). Useful for re-initializing per-process state like [gflags](https://github.com/google/python-gflags). See [Worker Startup Hooks](/docs/proxy/worker_startup_hooks) for details