A low-latency voice agent built from open models that runs entirely on your computer. One command starts a conversation: no API key, no cloud service.
The default pipeline is Silero VAD → Parakeet TDT → Qwen3.5-4B → Qwen3-TTS, and every stage is swappable. The server speaks the OpenAI Realtime protocol over WebSocket and WebRTC, so apps built for OpenAI's Realtime API can point at your own machine instead.
This pipeline runs in production as the conversation backend for thousands of Reachy Mini robots.
Use Python 3.10+ (3.11 recommended):
python3 -m venv .venv
source .venv/bin/activate
pip install speech-to-speech
speech-to-speech localThen talk: speak, pause, and the assistant answers through your speakers. Stop with Ctrl+C. The first run downloads the models, so give it a few minutes. Use headphones so the assistant does not hear itself, or add --local_audio_block_mic_during_playback.
On Ubuntu, first install the audio libraries with sudo apt-get install libportaudio2 libsndfile1. The default Linux Qwen3-TTS wheel targets CUDA 12.8 and Ubuntu 24.04; see the CUDA note for other systems. On Windows, activate the environment with .venv\Scripts\Activate.ps1.
The command picks local models for your platform:
| Platform | LLM | Plan for |
|---|---|---|
| Apple Silicon Mac | Qwen3.5-4B 4-bit via MLX LM | 16 GB of unified memory |
| Linux / Windows with NVIDIA | Qwen3.5-4B via Transformers | 24 GB of VRAM |
| Other machines | Qwen3.5-4B via Transformers on CPU | Slow replies |
Speech recognition and synthesis run locally too. After the first run, everything works offline. To use another model, a llama.cpp server, or a hosted provider, see LLM backends.
local runs a server and a microphone client in one process. To run only the server:
speech-to-speech serveIt listens at ws://127.0.0.1:8765/v1/realtime. Connect with:
- the packaged audio client:
speech-to-speech talk --url ws://127.0.0.1:8765/v1/realtime - the browser demo
- any client using the core OpenAI Realtime events; the OpenAI Agents SDK is tested over both transports (see Realtime API)
- LiveKit Agents, as the agent's realtime model
- How it works
- Quickstart
- Installation
- YAML configuration API
- Offline operation
- Supported components
- Commands
- Realtime API
- LLM backends
- Multi-language support
- OmniVoice
- Pocket TTS
- CLI reference
- Contributing
- Star history
- Citations
The pipeline is a cascade of four components, each running in its own thread and connected by queues:
- Voice Activity Detection (VAD): Silero VAD v5 detects speech boundaries and turn-taking.
- Speech to Text (STT): transcribes the user's turn, with optional live partial transcripts.
- Language Model (LLM): generates the response, streaming text and tool calls.
- Text to Speech (TTS): synthesizes audio and streams it back to the client.
Every stage has multiple interchangeable backends, selected via CLI flags. The code is designed for easy modification, with a focus on models available through Transformers and the Hugging Face Hub.
Optional speech-to-viseme extraction adds timed mouth shapes for avatar and robot
clients after TTS. Enable it with --enable_visemes; see the
viseme guide for client events and timing.
pip install speech-to-speech (see the Quickstart) includes everything the default pipeline needs:
- Silero VAD and Parakeet TDT for listening
- Qwen3.5-4B for replies: MLX LM on macOS, Transformers elsewhere
- Qwen3-TTS for speech:
mlx-audioon Apple Silicon, GGML elsewhere - the local audio client and the Realtime server
Platform-specific dependencies are resolved automatically.
See the configuration guide for reusable blocks, named pipelines, environment references, and explicit Python startup. The Mac example runs fully locally on Apple Silicon. From a repository checkout, run its audio client and server together, or start only the server:
speech-to-speech local -f example_configs/mac.yaml
speech-to-speech serve -f example_configs/mac.yaml --name maclocal runs one selected pipeline with the packaged loopback audio client.
talk -f example_configs/mac.yaml uses only the file's client settings.
Configured server startup currently supports one named definition per invocation; its
num_pipelines controls the number of isolated instances.
Existing Python interfaces, CLI commands, and flat JSON configuration remain available.
On Linux, the Qwen3-TTS GGML backend comes from faster-qwen3-tts[ggml]. Its default qwentts-cpp-python wheel on PyPI targets CUDA 12.8 and manylinux_2_39 (for example, Ubuntu 24.04). If your CUDA runtime or glibc is older, install the matching wheel from the Hugging Face wheelhouse before installing speech-to-speech:
# CUDA 13.x
pip install "qwentts-cpp-python==0.3.1+cu130" \
-f https://huggingface.co/datasets/andito/qwentts-cpp-python-wheels/tree/main/whl/cu130
# CUDA 12.4
pip install "qwentts-cpp-python==0.3.1+cu124" \
-f https://huggingface.co/datasets/andito/qwentts-cpp-python-wheels/tree/main/whl/cu124
# CPU-only fallback
pip install "qwentts-cpp-python==0.3.1+cpu" \
-f https://huggingface.co/datasets/andito/qwentts-cpp-python-wheels/tree/main/whl/cpu
pip install speech-to-speechTo use the previous CUDA-graphs implementation instead of GGML, pass --qwen3_tts_backend torch.
Optional components are installed with pip extras:
pip install "speech-to-speech[kokoro]" # Kokoro-82M TTS on non-macOS
pip install "speech-to-speech[kitten]" # KittenTTS
pip install "speech-to-speech[pocket]" # Pocket TTS
pip install "speech-to-speech[chattts]" # ChatTTS
pip install "speech-to-speech[omnivoice]" # OmniVoice TTS (CUDA, Intel XPU, or Apple Silicon)
pip install "speech-to-speech[faster-whisper]" # Faster Whisper STT
pip install "speech-to-speech[whisper-mlx]" # Lightning Whisper MLX STT on macOS
pip install "speech-to-speech[paraformer]" # Paraformer STT through FunASR
pip install "speech-to-speech[fireredvad]" # FireRed streaming VAD
pip install "speech-to-speech[nemo]" # Parakeet Unified, Nemotron, and Orukeet STT through NeMo
pip install "speech-to-speech[mlx-lm]" # mlx-vlm support for vision models on macOSDeprecated implementations, including MeloTTS, live in archive/ and are no longer wired into the CLI.
Note on DeepFilterNet: DeepFilterNet, used for optional audio enhancement in VAD, requires numpy<2 and conflicts with Pocket TTS, which requires numpy>=2. Install it manually only in environments where you are not using Pocket TTS.
To work on the code, install Git and uv, then run:
git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech
uv sync --python 3.11
source .venv/bin/activateThis installs the package in editable mode. With the environment activated, use the same speech-to-speech commands shown above.
| Component | Backend | Platforms | Install |
|---|---|---|---|
| VAD | Silero VAD v5 | all | built-in |
| VAD | FireRed Stream-VAD | CUDA / CPU | fireredvad |
| STT | Parakeet TDT (default) | CUDA / CPU through nano-parakeet, Apple Silicon through MLX | built-in |
| STT | Parakeet Unified | CUDA / CPU | nemo |
| STT | Nemotron Speech Streaming | CUDA / CPU | nemo |
| STT | Nemotron Streaming Farsi, selected with --stt nemotron-streaming --nemotron_streaming_model_name mehdi-hf/nemotron-asr-streaming-farsi |
CUDA / CPU | nemo, NeMo >=3.0, Python >=3.11 |
| STT | Orukeet | CUDA / CPU | nemo |
| STT | Whisper through Transformers | CUDA / CPU | built-in |
| STT | Faster Whisper | CUDA / CPU | faster-whisper |
| STT | Lightning Whisper MLX | Apple Silicon | whisper-mlx |
| STT | MLX Audio Whisper | Apple Silicon | built-in on macOS |
| STT | Paraformer | CUDA / CPU | paraformer |
| STT | Qwen3-ASR through Transformers | CUDA / CPU, Apple Silicon | built-in |
| STT | OpenAI-compatible /v1/audio/transcriptions endpoint |
local or remote HTTP server | built-in |
| STT | OpenAI Realtime transcription | hosted or compatible WebSocket server | built-in |
| STT | vLLM Realtime transcription (experimental) | local or remote vLLM server | built-in |
| LLM | mlx-lm (default on macOS) | Apple Silicon | built-in on macOS |
| LLM | Transformers (default elsewhere) | supported accelerators / CPU | built-in |
| LLM | OpenAI-compatible API (responses-api, chat-completions) |
optional hosted providers or self-hosted servers | built-in |
| TTS | Qwen3-TTS (default) | GGML / CUDA on Linux, mlx-audio on macOS | built-in |
| TTS | Kokoro-82M | CUDA / CPU, Apple Silicon | kokoro on non-macOS; built-in on macOS |
| TTS | KittenTTS | CPU | kitten |
| TTS | Pocket TTS | CPU / CUDA | pocket |
| TTS | ChatTTS | CUDA / CPU | chattts |
| TTS | OmniVoice | CUDA / Intel XPU / Apple Silicon | omnivoice |
| TTS | MMS TTS | CUDA / CPU | built-in |
| TTS | OpenAI-compatible /v1/audio/speech endpoint |
local or remote HTTP server | built-in |
| TTS | MiniMax | hosted HTTP streaming | built-in |
Optional streaming speaker diarization
adds speaker labels to transcribed turns. Enable it with --diarization after
installing the supporting Transformers build; the linked guide has the current
model revision and setup while the merged Transformers PR #49056
is awaiting a package release.
Select implementations with --stt, --llm_backend, and --tts. The CLI constructs configuration only for the selected backends; known options for inactive backends remain accepted for compatibility but are ignored with a warning. JSON configuration may likewise include extra inactive-backend keys, which are ignored. Run speech-to-speech serve -h for the defaults, or pass selectors before -h to see another combination's backend-specific flags (for example, speech-to-speech serve --stt mlx-audio-whisper -h).
For client-only TTS serving with vLLM-Omni or another compatible server, see OpenAI-compatible TTS.
For client-only speech recognition with vLLM, OpenAI's hosted Transcription API, or another compatible server, see OpenAI-compatible STT. For native incremental audio and partial transcripts, see stateful streaming STT.
| Command | Behavior | Use it when |
|---|---|---|
serve |
Runs the pipeline server over OpenAI Realtime WebSocket and WebRTC. | You are building an app or device against the API. |
talk --url <full-realtime-url> |
Runs the packaged microphone/speaker client. | You want to talk to an existing Realtime server. |
local |
Composes serve and talk in-process over loopback. |
You want to run the server and talk to it from one command. |
serve binds to 127.0.0.1 by default; pass --host 0.0.0.0 explicitly for network exposure. local always binds to loopback and connects the same packaged client at ws://127.0.0.1:<port>/v1/realtime.
The command only decides how the server and client run. Where replies are generated is set by --llm_backend, which is local by default for every command.
The packaged local client buffers 196 ms of received audio when using the
OpenAI-compatible TTS backend, which absorbs short delivery gaps from HTTP
speech inference. Other local backends and talk start playback immediately
by default. Use --playback-buffer-ms <milliseconds> to override either
default: a larger value resists stuttering but delays the start of each
response, while a smaller value starts sooner but is more sensitive to jitter.
This setting only controls the packaged Python client's speakers; browser and
other Realtime clients manage their own playback buffers.
The packaged client can opt in to local Python tools with talk --tool-module <module> or local --tool-module <module>. The module contract, programmatic API, and web-search examples (Serper, or keyless You.com) are documented in Tool calling design.
--mode is deprecated and will stop working soon. During this migration window, speech-to-speech --mode realtime runs speech-to-speech serve, and speech-to-speech --mode local runs speech-to-speech local; both print a warning. All other mode values have been removed and exit with guidance to use the new commands.
On macOS, local and serve already run all three models through MLX. The optional --mac-optimal-settings preset also makes MPS the default device for supported components. It only sets defaults: explicit --device, component-device flags such as --qwen3_tts_device, and --stt, --llm_backend, --model_name, and --tts all win.
--tts pocket, --tts kokoro, --tts omnivoice, and --tts kitten are also valid on macOS.
To compare the MLX quantization variants locally:
python scripts/benchmark_tts.py \
--handlers qwen3 \
--iterations 3 \
--qwen3_mlx_quantizations bf16 4bit 6bit 8bitInstall the NVIDIA Container Toolkit, then:
docker compose upThe compose file starts a llama.cpp server with Gemma 4 and the Realtime server, exposing ports 8080 and 8765.
The server logs per-response STT, LLM, first TTS audio, and speech-to-audio durations for supported backends, including vLLM-backed STT and TTS. See the response latency guide for the coverage matrix and measurement boundaries.
Realtime mode supports the OpenAI Realtime protocol over WebSocket and WebRTC, with live transcription and low-latency turn-taking. WebSocket clients connect at /v1/realtime:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8765/v1",
websocket_base_url="ws://localhost:8765/v1",
api_key="not-needed",
)
with client.realtime.connect(model="local") as conn:
conn.send(
{
"type": "session.update",
"session": {
"type": "realtime",
"instructions": "You are a helpful assistant.",
"audio": {
"input": {
"turn_detection": {
"type": "server_vad",
"interrupt_response": True,
}
}
},
},
}
)
for event in conn:
print(event.type)The server implements the core Realtime event set: input_audio_buffer.append, session.update, conversation.item.create, conversation.item.truncate, response.create, and response.cancel inbound; speech start/stop, streaming transcription, audio deltas, tool calls, and response.done outbound. CI connects pinned @openai/agents RealtimeSession instances through the SDK's stock WebSocket and WebRTC transports. This is a tested core subset, not a claim of full OpenAI Realtime API equivalence. The event matrix, architecture, and design details live in the Realtime Engine README.
With --enable_llm_proxy, the realtime server also exposes the remote LLM it is configured with as a plain OpenAI compatible endpoint, so a client can run side tasks (summaries, titles, background agents) with tools and streaming, fully concurrent with the voice conversation and never interrupted by new speech:
POST /v1/chat/completionswhen running--llm_backend chat-completionsPOST /v1/responseswhen running--llm_backend responses-api
The server performs no authentication and no throttling of its own. Enable the proxy only on a trusted network, or deploy the server behind a gateway that owns access control. The s2s-endpoint compute replica is such a gateway: it opens these paths only to clients that created their session with an HF token, checks the API key against that token, and applies a rate limit per user. Point the stock OpenAI SDK at whichever host you talk to; this server ignores the API key (a gateway in front decides what it must be):
from openai import OpenAI
llm = OpenAI(base_url="http://localhost:8765/v1", api_key="unused")
completion = llm.chat.completions.create(
model="anything", # ignored: the server forces its configured --model_name
messages=[{"role": "user", "content": "Summarize the conversation so far: ..."}],
)Requests are stateless (send the full message list each time) and are proxied to the configured upstream with the key held by the server, which never reaches clients. The model field is always overwritten with the server configured --model_name. The proxy is off by default, requires a remote backend (chat-completions or responses-api), and answers 501 with the reason otherwise.
By default the LLM runs inside the speech process. You can swap the model, run it in a separate server, or hand it to a hosted provider:
| Where the LLM runs | Backend |
|---|---|
| In process on Apple Silicon (default) | mlx-lm |
| In process elsewhere (default) | transformers |
| A separate server on your hardware (llama.cpp, vLLM, ...) | responses-api or chat-completions |
| A hosted provider | responses-api or chat-completions |
Pass --model_name, using an MLX conversion on Mac and a Transformers checkpoint elsewhere:
speech-to-speech local --model_name mlx-community/Qwen3.5-9B-4bit # Apple Silicon
speech-to-speech local --model_name Qwen/Qwen3.5-9B # Linux / WindowsTransformers picks an available accelerator and falls back to CPU. --llm_device overrides the LLM's device; --device overrides every stage. Generation options such as --llm_gen_max_new_tokens are in the LLM guide.
Serve the LLM with llama.cpp and connect speech-to-speech through its local Responses API. This works on Apple Silicon or Linux/NVIDIA. Install llama.cpp with brew install llama.cpp on macOS or use a CUDA build on NVIDIA.
With the default speech models, budget 24 GB of unified memory on Mac or 16 GB of VRAM on NVIDIA; about 8 GB of that goes to the speech pipeline.
Terminal 1 — start the LLM and leave it running:
llama-server \
-hf ggml-org/gemma-4-E4B-it-GGUF:Q4_0 \
--alias local-gemma \
--host 127.0.0.1 --port 8080 \
-ngl all -np 1 -c 8192 -fa on \
--no-mmproj \
--reasoning offTerminal 2 — activate your Python environment and connect the speech pipeline to that LLM:
speech-to-speech local \
--llm_backend responses-api \
--model_name local-gemma \
--responses_api_base_url http://127.0.0.1:8080/v1 \
--responses_api_api_key ""The Q4_0 weights are about 4.6 GB. The example uses one 8k context and disables reasoning for faster replies; --no-mmproj skips the image/audio projector because STT supplies text. To try another GGUF model, change -hf and keep --alias and --model_name matched.
Select an API backend to move only the LLM to a provider. For OpenAI:
export OPENAI_API_KEY=...
speech-to-speech local --llm_backend responses-apiThis uses gpt-5.6-terra with reasoning effort none. Speech recognition and synthesis stay local, so your audio never leaves your computer; transcribed text and conversation history go to the provider.
Two API backends share the same --responses_api_* connection flags:
--llm_backend responses-apitargets/v1/responses.--llm_backend chat-completionstargets/v1/chat/completions.
Works with any provider or server that implements the OpenAI Responses API. Point --responses_api_base_url at the endpoint and set --model_name accordingly:
| Provider / server | --responses_api_base_url |
--responses_api_api_key |
|---|---|---|
| OpenAI | omit, uses OpenAI default | $OPENAI_API_KEY |
| HF Inference Providers | https://router.huggingface.co/v1 |
$HF_TOKEN |
| OpenRouter | https://openrouter.ai/api/v1 |
$OPENROUTER_API_KEY |
| vLLM | http://localhost:8000/v1 |
omit or any string |
| llama.cpp | http://127.0.0.1:8080/v1 |
empty string |
# OpenAI
speech-to-speech local \
--llm_backend responses-api \
--model_name "gpt-4o-mini" \
--responses_api_api_key "$OPENAI_API_KEY" \
--responses_api_stream \
--enable_live_transcription# HF Inference Providers: Qwen3.5-9B via Together
speech-to-speech local \
--llm_backend responses-api \
--model_name "Qwen/Qwen3.5-9B:together" \
--responses_api_base_url "https://router.huggingface.co/v1" \
--responses_api_api_key "$HF_TOKEN" \
--responses_api_stream \
--enable_live_transcription# HF Inference Providers: GPT-oss-20B via Groq
speech-to-speech serve \
--llm_backend responses-api \
--model_name "openai/gpt-oss-20b:groq" \
--responses_api_base_url "https://router.huggingface.co/v1" \
--responses_api_api_key "$HF_TOKEN" \
--responses_api_stream \
--enable_live_transcriptionIdentical configuration to responses-api, reusing the same --responses_api_* connection flags, but talks to /v1/chat/completions instead of /v1/responses. Prefer it when:
- the provider ignores
chat_template_kwargs.enable_thinkingon the Responses path and needs areasoning_effortknob to suppress reasoning, or - the server's Responses streaming tool-call path is unreliable, while its Chat Completions tool-call streaming is solid. This is useful for some vLLM builds; see #312.
Add --responses_api_reasoning_effort none to disable reasoning on providers where the chat-template flag has no effect:
# vLLM serving a Qwen model with tool calling
speech-to-speech serve \
--llm_backend chat-completions \
--model_name "Qwen/Qwen3.5-4B" \
--responses_api_base_url "http://localhost:8000/v1" \
--responses_api_stream# Gemma 4 31B via the HF router on Cerebras, with reasoning disabled for low voice latency
speech-to-speech serve \
--llm_backend chat-completions \
--model_name "google/gemma-4-31B-it:cerebras" \
--responses_api_base_url "https://router.huggingface.co/v1" \
--responses_api_api_key "$HF_TOKEN" \
--responses_api_reasoning_effort none \
--responses_api_streamUse --stt none --llm_backend chat-completions to send each completed VAD
audio segment directly to an audio-input model. Direct audio mode is not
supported with --llm_backend responses-api: a model may accept audio through
/v1/chat/completions without supporting /v1/responses, including OpenAI's
gpt-audio-1.5.
You must explicitly set --model_name to a model that accepts audio: the
API backends' default gpt-5.6-terra accepts text and image input, but not audio. Check the
provider's model documentation and endpoint support before enabling this mode.
For OpenAI, see the
GPT-5.6 Terra model card
and audio-input guide.
speech-to-speech serve \
--stt none \
--llm_backend chat-completions \
--model_name "YOUR_AUDIO_CAPABLE_MODEL" \
--responses_api_base_url "https://provider.example/v1" \
--responses_api_api_key "$PROVIDER_API_KEY"OpenAI-compatible servers represent input audio differently. Use
--responses_api_audio_content_type input_audio (the default) for embedded
WAV base64, or --responses_api_audio_content_type audio_url for a base64
data URL.
For a fully local audio-input model on Apple Silicon, see the Gemma 4 12B example.
The pipeline can run without internet access after the dependencies and model assets for the selected components are installed locally. Before disconnecting, start the exact configuration once while online so it can cache the STT, LLM, TTS, Silero VAD, NLTK, and Smart Turn resources it needs.
After that first run, set HF_HUB_OFFLINE=1 to skip Hugging Face Hub requests:
HF_HUB_OFFLINE=1 speech-to-speech localFor the llama.cpp configuration, keep the LLM server running and pass the same LLM flags:
HF_HUB_OFFLINE=1 speech-to-speech serve \
--llm_backend responses-api \
--model_name local-gemma \
--responses_api_base_url "http://127.0.0.1:8080/v1" \
--responses_api_api_key ""Hosted LLMs still need a connection.
Smart Turn uses a separate ONNX checkpoint. A cached checkpoint works with HF_HUB_OFFLINE=1; for an explicit,
cache-independent setup, pass --smart_turn_model_path /path/to/smart-turn-v3.2-cpu.onnx. If the checkpoint is not
available, pass --no_smart_turn to disable Smart Turn.
Language coverage depends on the STT and TTS backends you pick, not on the pipeline itself:
| Component | Backend | Languages |
|---|---|---|
| STT | Parakeet TDT (default) | 25 European languages |
| STT | Whisper / Whisper MLX / Faster Whisper | Broad multilingual coverage, depending on the selected Whisper checkpoint |
| STT | Paraformer | Depends on the selected FunASR checkpoint; the default is Chinese-oriented |
| TTS | Qwen3-TTS (default) | Multilingual, with --qwen3_tts_language auto by default |
| TTS | Kokoro | Multiple language/voice mappings, depending on backend availability |
| TTS | KittenTTS | English |
| TTS | ChatTTS | English and Chinese |
| TTS | MMS TTS | Broad multilingual coverage through MMS checkpoints |
| TTS | OmniVoice | 600+ languages; voice cloning, design, and automatic voice selection |
| TTS | Supertonic | 32 languages |
| TTS | Pocket TTS | English, French, German, Portuguese, Italian, Spanish |
| TTS | OpenAI-compatible /v1/audio/speech endpoint |
Depends on the connected TTS server/model |
Make sure the STT, LLM, and TTS you pair all cover your target language(s). By default, the language code sent to TTS comes from the user's transcription. Qwen3-TTS keeps its configured auto behavior by default. Add --detect_llm_output_language to detect the language of each assistant text chunk instead and send that code to TTS. This helps when the assistant replies in a different language from the user. The Lingua detector is loaded and warmed when the pipeline starts, so detection does not wait for the full reply. Short text (under 20 characters for most languages, or under four CJK characters) and ambiguous text use the last detected assistant language for that response. If there is none yet, TTS receives no language code and uses its own automatic or default behavior; this avoids delaying speech to collect more text. A TTS backend must support the detected language to use it. When a Realtime session selects a language with session.audio.input.transcription.language, STT keeps that language, but TTS receives the confidently detected assistant language instead. The first spoken text of each response decides; if it is too short or ambiguous, the whole response uses the session's selected language. If the active TTS backend cannot speak the detected language, that response uses the session's selected language.
For Parakeet TDT, the decoder chooses the transcription language automatically. --parakeet_tdt_language remains accepted for compatibility with existing commands, but setting it to a code such as de does not constrain decoding. The reported language is inferred with Lingua from the finished transcription when possible. Text shorter than 20 characters is not classified, and inconclusive or failed detection reports an unknown language rather than the configured or previous code. --language belongs to the Whisper backends; it does not control Parakeet. For Whisper and Whisper MLX, use --language auto to detect each turn, or a code such as --language zh to fix the language. Other STT backends have their own language flags; see the STT component guide.
To encourage replies in the user's detected language, add --enable_lang_prompt. It appends a per-turn instruction such as "Please reply to my message in French." The flag is off by default and works independently of --detect_llm_output_language. Qwen3-TTS uses --qwen3_tts_language auto by default: it infers language from text unless assistant language detection or a session selection supplies a supported code. Set --qwen3_tts_language german, for example, to force synthesis in German.
Automatic detection with Parakeet and a language instruction for the LLM:
speech-to-speech serve \
--stt parakeet-tdt \
--enable_lang_prompt \
--detect_llm_output_language \
--qwen3_tts_language autoA single non-English language, Chinese in this example:
speech-to-speech serve \
--stt whisper-mlx \
--stt_model_name large-v3 \
--language zhBoth commands also work with --mac-optimal-settings; explicit --stt flags override the defaults it sets.
OmniVoice provides voice cloning, voice design, and automatic voice selection across 600+ languages. Install its opt-in dependencies and provide a reference clip plus its transcript for voice cloning. This example uses CUDA on Linux or Windows; use --omnivoice_device mps on Apple Silicon, --omnivoice_device xpu with an Intel XPU-enabled PyTorch installation, or --omnivoice_device npu with an Ascend torch_npu installation:
pip install "speech-to-speech[omnivoice]"
speech-to-speech serve \
--tts omnivoice \
--omnivoice_device cuda \
--omnivoice_ref_audio /path/to/reference.wav \
--omnivoice_ref_text "Transcript of the reference clip."The handler converts OmniVoice's completed 24 kHz float output into the pipeline's 16 kHz int16 blocks. OmniVoice does not currently expose incremental audio through generate(), so the first block is available only after the full utterance has been synthesized. See the TTS component guide for saved prompts, voice design, devices, latency, and all backend flags.
The omnivoice extra is supported on Linux, Windows, and macOS. On non-macOS platforms, both OmniVoice and the built-in Qwen3 backend share Transformers 5 through faster-qwen3-tts>=0.4.0, so installing this extra keeps the default Qwen3 path available. Linux uses Qwen3's GGML extra by default; see the CUDA note if its CUDA 12.8 / manylinux_2_39 native wheel does not match your host.
Warning
OmniVoice's code is Apache-2.0, but its pretrained weights are CC-BY-NC and are not licensed for commercial use. Use voice cloning only with authorization and consent; do not use it for impersonation, fraud, scams, or other illegal or unethical activity.
Pocket TTS from Kyutai Labs provides streaming TTS with voice cloning:
speech-to-speech serve \
--tts pocket \
--pocket_tts_voice jean \
--pocket_tts_device cpuAvailable voice presets: alba, marius, javert, jean, fantine, cosette, eponine, azelma. Custom voice files and Hugging Face paths also work.
Select mehdi-hf/pocket-tts-farsi-v2
with --pocket_tts_model_name. The handler normalizes Persian spelling, expands
numbers, and converts assistant text to phonemes with
mehdi-hf/Homo-GE2PE-Persian-HF before synthesis. Both models download on first use.
The released Pocket TTS package does not support this model's phoneme frontend
flags yet. Install the author's fork after the pocket extra. This command pins
an inspected commit for reproducibility:
pip install "speech-to-speech[pocket]"
pip install "pocket-tts @ git+https://github.com/mallahyari/pocket-tts@3807c204babb5fe54be8fe18a362a58315e870d6"
speech-to-speech serve \
--tts pocket \
--pocket_tts_model_name mehdi-hf/pocket-tts-farsi-v2 \
--pocket_tts_voice /voices/persian-reference.wav \
--pocket_tts_device cpuUse a Persian reference recording as a local audio file or Hugging Face audio
URL. Preset voice states such as jean are incompatible with these weights.
The handler uses the first five seconds of the recording. It splits text at
sentence boundaries before phoneme conversion, adds a 0.25-second pause between
sentences, and limits synthesis chunks to 18 tokens. It keeps ezafe-linked words
together and rejects a linked phrase that exceeds the budget. Oversized G2P inputs
raise an error rather than lose text.
Configure the LLM to respond in Persian. This model choice stays fixed for the
session and does not switch languages based on STT detection.
Farsi synthesis bypasses the fork's orthographic sentence splitter so that ?
remains a glottal-stop phoneme. It uses an EOS threshold of -2 and zero extra
frames after EOS. --pocket_tts_temperature overrides the model's default of
0.3; --pocket_tts_eos_threshold overrides -2. Each phoneme chunk is buffered
before playback. Silent, non-finite, or length-capped output is retried once,
then reported as an error. This adds one chunk's generation time before the
first audio block, but avoids playing a failed attempt. PCM conversion clips
peaks instead of wrapping, and downsampling preserves filter history across
playback blocks.
To check the handler with actual weights from a source checkout, run:
python scripts/check_pocket_tts_farsi.py --output-dir /tmp/pocket-tts-farsi-checkThe check downloads the TTS and G2P models and the author's Persian reference
samples/prompt_short_sentence.wav. It uses seed 42 and temperature 0.3.
It synthesizes a greeting, a sentence with a Persian number, and a
two-sentence reply. It checks PCM block format, non-silent output, and response
completion, then saves WAVs and timing measurements. Listen to the WAVs to assess
pronunciation. Use --voice /voices/reference.wav to test your own recording.
The check saves native 24 kHz audio by default. Pass --sample-rate 16000 to
compare with the pipeline's current 16 kHz playback output. The server and local
playback path still assume 16 kHz, so changing only --pocket_tts_sample_rate
to 24000 does not enable native-rate playback.
The model weights use CC-BY-NC-4.0 and are for non-commercial use.
References for pipeline CLI arguments live in the arguments classes and in speech-to-speech serve -h. Client arguments are listed by speech-to-speech talk -h.
See ModuleArguments. It allows setting:
- a common
--device, if every part should run on the same device - macOS model/device defaults (
--mac-optimal-settings) - STT implementation (
--stt) - LLM backend (
--llm_backend:transformers,mlx-lm,responses-api, orchat-completions) - TTS implementation (
--tts) - logging level
- transcript logging (
--log_transcripts) - realtime pipeline pool size (
--num_pipelines) - PyTorch CPU thread count (
--torch_num_threads)
Logs are content-free by default: transcript-bearing records report a character count rather
than the text, because logs are commonly retained by service managers, containers and hosted
log aggregators. Pass --log_transcripts to include full user and assistant transcripts when
debugging STT, LLM, TTS or Realtime behaviour; it logs a warning at startup because enabling
it writes conversation content wherever those logs are collected. The talk client takes the
same flag, as --log-transcripts or --log_transcripts.
--torch_num_threads N sets PyTorch's CPU intra-op thread count (torch.set_num_threads) for
the whole process. It is unset by default, which keeps the current behaviour: Silero VAD sets
PyTorch to a single thread when it loads, and that single thread also applies to every other
PyTorch model on the CPU. If your STT runs on the CPU, you can trade some Silero speed for
faster transcription:
speech-to-speech serve --stt parakeet-tdt --device cpu --torch_num_threads 8The setting is shared: Silero and the CPU STT use the same thread count, and more threads can
slow Silero down, so measure the complete pipeline when choosing a value. It only affects
PyTorch; ONNX Runtime, MLX and other non-PyTorch backends keep their own thread settings. In a
YAML configuration, set it under runtime as torch_num_threads. See the
torch.set_num_threads documentation.
See VADHandlerArguments. Notable options:
--thresh: threshold value to trigger voice activity detection.--min_speech_ms: minimum duration of detected voice activity to be considered speech.--min_speech_continuation_ms: sustain-bar hysteresis threshold for speech that continues a reopenable soft-ended, uncommitted turn within the reopen window. The default and recommended pairing is--min_speech_ms 384 --min_speech_continuation_ms 192.--min_silence_ms: minimum length of silence intervals for segmenting speech. Default is 64 ms.--short_segment_merge_ms: optional merge window for stitching adjacent VAD segments that are each shorter than--min_speech_ms.--speculative_reopen_ms: delay response commitment for 800 ms after a soft-ended turn so immediately resumed speech can reopen it.--unanswered_reopen_ms: sanity cap on how long a soft-ended speculative turn that has not yet received any assistant output stays reopenable. With Smart Turn enabled, this is clamped to at least--smart_turn_max_wait_msso a turn remains reopenable for its full grace.
Smart Turn v3.2 can validate Silero's end-of-speech
decisions using the content and prosody of the current turn. Silero finalizes the segment
and STT/LLM work may begin speculatively. Complete turns start processing immediately and use
--speculative_reopen_ms (800 ms by default) before committing output. Incomplete turns wait
--smart_turn_incomplete_delay_ms (600 ms by default) before starting STT/LLM work, while their output remains
gated by --smart_turn_max_wait_ms (2 seconds by default). If speech resumes during either delay, the existing turn is
reopened as a newer revision, the accumulated audio is re-emitted, and work from the previous revision is
discarded before it reaches the user.
The turn tracker owns the conversation order and the LISTENING, SOFT_ENDED,
ANSWERING, and CLOSED states. VAD supplies speech boundaries and Smart Turn timing;
the tracker decides whether resumed speech reopens the current turn. The unanswered
reopen cap uses streamed-audio time, so a push-to-talk pause with no audio does not
advance it. Processing and output holds use wall-clock deadlines. Starting a newer
turn drops older work that has not committed; accepted output can finish.
Each response belongs to the input supplied to generation. The server records that ownership on conversation items and accepted tool calls. A tool follow-up keeps its originating input's turn. If a user message follows the tool call, the follow-up answers that message's turn; if a newer turn closed without one, it moves to that turn, so a late tool result is still spoken. Client input without a speech-turn association stays untagged. A response cannot borrow the identity of speech still being recorded. These input records do not control turn state, reopening, or deadlines.
The server holds input_audio_buffer.speech_stopped and the final transcription while a turn can still
reopen. Resumed speech keeps the same open item and live transcription deltas
continue. Once the turn commits, the client receives one stop, an input-buffer commitment, the created
user item, and one final transcript, so its user-turn history matches the model's.
If transcription fails, the server sends the stop and failure after the reopen grace ends.
The base package includes the quantized CPU runtime and enables Smart Turn by default:
pip install speech-to-speech
speech-to-speech serveThe latest supported v3.2 CPU checkpoint downloads from the Hugging Face Hub on first use. Pass
--smart_turn_model_path /path/to/model.onnx to use a local model, or --no_smart_turn to disable Smart Turn.
Smart Turn is enabled by default for server sessions and the packaged local client.
Tune the completion cutoff with --smart_turn_threshold (default 0.5). A higher threshold makes ambiguous
pauses more likely to use the longer speculative response grace.
model_name, torch_dtype, and device are exposed for each STT, LLM, and TTS implementation. STT and TTS parameters use the handler prefix, for example --stt_model_name or --qwen3_tts_device. LLM model selection and chat settings are shared across backends via unprefixed flags, for example --model_name and --chat_size; backend-specific flags use the responses_api_ prefix for the responses-api and chat-completions backends and the llm_ prefix for local backends.
For example:
# Local transformers/mlx-lm backend
--model_name google/gemma-2b-it
# OpenAI-compatible backend
--llm_backend responses-api --model_name deepseek-chat --responses_api_base_url https://api.deepseek.comOther generation parameters can be set using the handler prefix plus _gen_, for example --stt_gen_max_new_tokens 128 or --llm_gen_temperature 0.7. Parameters not yet exposed can be added to the relevant arguments class.
Run a pinned 40-question audio sample or all 1,000 Big Bench Audio questions
through VAD, STT, the LLM, and TTS, with
per-category answer accuracy, audio latency, and protocol failures in JSON reports.
src/speech_to_speech/evals/big_bench_audio/Dockerfile.eval builds an image for
local GPU runs or Hugging Face Jobs;
scripts/prepare_eval_space.py optionally builds the image in a private HF Space.
Start with four questions, then compare reports across revisions. See
Big Bench Audio system tests.
Issues and PRs are welcome. Good starting points are the open issues. For larger changes, open an issue first to discuss the approach.
For local development:
uv sync --group dev
uv run pytest tests -q
uv run ruff check src tests
uv run ruff format --check src tests
uv run mypy srcTo check turn ordering, Smart Turn timing, and Realtime routing on CPU:
CUDA_VISIBLE_DEVICES='' uv run pytest -q \
tests/test_speculative_turns.py \
tests/test_smart_turn.py tests/test_stt_stale_filter.py \
tests/test_audio_input_notifier.py tests/test_lm_output_processor.py \
tests/openai_realtime/test_response_input_identity.py \
tests/openai_realtime/test_realtime_service.py \
tests/openai_realtime/test_speculative_turn_protocol.pyThe response-input tests include a delayed tool completing during synthetic speech, using the packaged client coordinator, service, and VAD handler with mocked model output and VAD probabilities. Run that reproduction alone with:
CUDA_VISIBLE_DEVICES='' uv run pytest tests/openai_realtime/test_response_input_identity.py -q -s
If you use this pipeline, please also cite the component models you run. The defaults are:
@misc{SileroVAD,
author = {Silero Team},
title = {Silero VAD: pre-trained enterprise-grade Voice Activity Detector (VAD), Number Detector and Language Classifier},
year = {2021},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/snakers4/silero-vad}},
email = {hello@silero.ai}
}@misc{parakeet-tdt,
author = {NVIDIA},
title = {Parakeet TDT 0.6B v3},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3}}
}@misc{qwen3-tts,
author = {Qwen Team},
title = {Qwen3-TTS},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice}}
}Citations for optional backends such as Kokoro, KittenTTS, Pocket TTS, ChatTTS, Whisper variants, Paraformer, and MMS live in the respective component READMEs.