Supporting Multi-Language Voice Conversations with Automatic Language Detection in Speech-to-Speech

The Speech-to-Speech pipeline achieves multi-language voice conversations with automatic language detection by processing audio through STT handlers that detect ISO-639-1 language codes using either the lingua library or Whisper's native tokens, then propagating these codes through the Transcription message to coordinate multilingual responses across the LLM and TTS stages.

The huggingface/speech-to-speech repository implements a modular, full-duplex voice agent architecture designed for seamless multi-language voice conversations with automatic language detection. The system decomposes the pipeline into four interchangeable components—Voice Activity Detection (VAD), Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS)—each running in isolated threads and communicating via queues. Language detection occurs exclusively in the STT stage, creating a single source of truth that flows downstream to ensure consistent multilingual dialogue without manual intervention.

STT-Based Language Detection Architecture

Automatic language detection is implemented within the STT handlers, which analyze incoming audio streams and attach detected language codes to transcription objects. The repository provides two primary backends for this functionality, each handling detection differently while emitting a standardized Transcription message.

Parakeet TDT with Lingua Library

The default STT handler utilizes the Parakeet TDT model combined with the lingua language detection library. Located in src/speech_to_speech/STT/parakeet_tdt_handler.py (lines 30-80), the ParakeetTDTSTTHandler class loads the multilingual model at startup and initializes the lingua detector to avoid latency on first use. After each transcription, the handler runs the detector on the predicted text, producing an ISO-639-1 language code that is attached to the Transcription message via the language_code field.

Whisper Language Token Validation

Alternatively, the Whisper STT handler leverages the model's built-in language token generation. As implemented in src/speech_to_speech/STT/whisper_stt_handler.py (lines 20-33), the WhisperSTTHandler class extracts the language token from Whisper's output and validates it against an internal whitelist. If the detected language is unsupported, the system falls back to the last known language code, ensuring continuity in multi-turn conversations.

Language Propagation Through the Pipeline

Both STT handlers emit a Transcription object defined in src/speech_to_speech/pipeline/messages.py. This dataclass carries the detected language downstream:

Transcription(
    text=pred_text,
    language_code=language_code,   # e.g. "en", "fr", "es‑auto"

    turn_id=...,
    turn_revision=...,
    speech_stopped_at_s=...,
)

The pipeline orchestrator in src/speech_to_speech/s2s_pipeline.py forwards this message to the LLM component. Through LanguageModelHandlerArguments (defined in src/speech_to_speech/arguments_classes/language_model_arguments.py), the LLM can be instructed to honor the language_code, generating responses in the same language as the user's utterance. The TTS handler subsequently selects the appropriate voice model—such as Qwen3-TTS, Kokoro-82M, or Pocket-TTS—based on the propagated language code, completing the multilingual loop.

Implementing Multi-Language Support with Automatic Detection

Server Deployment with Auto-Detect

To launch a multilingual conversation server using the default Parakeet TDT backend with automatic detection enabled:

# Install the library (includes Parakeet TDT and lingua)

pip install speech-to-speech

# Start the realtime server (auto-detect enabled)

speech-to-speech serve \
    --stt parakeet-tdt   # language auto-detect is default

To force a specific starting language while retaining auto-detection for subsequent turns:

speech-to-speech serve \
    --stt parakeet-tdt \
    --stt_language de    # sets starting language; model will still auto-detect later turns

Switching to Whisper STT

For deployments preferring Whisper's detection mechanism:

speech-to-speech serve \
    --stt whisper \
    --stt_language auto

Programmatic Access to Detected Languages

For custom integrations, instantiate the handler directly to retrieve language codes:

from speech_to_speech.pipeline.messages import Transcription
from speech_to_speech.STT.parakeet_tdt_handler import ParakeetTDTSTTHandler

stt = ParakeetTDTSTTHandler()
stt.setup(language="auto")          # enable auto-detect

# Assume `audio_chunk` is a NumPy array captured from VAD

for result in stt.process(audio_chunk):
    if isinstance(result, Transcription):
        print(f"User said: {result.text}")
        print(f"Detected language: {result.language_code}")   # e.g. "fr"

Summary

• Automatic language detection occurs exclusively in the STT stage via ParakeetTDTSTTHandler or WhisperSTTHandler, preventing latency by pre-loading detectors at startup.
• The lingua library powers detection for Parakeet TDT, while Whisper uses native language tokens with whitelist validation and fallback logic.
• Language information flows through the Transcription dataclass in messages.py, creating a single source of truth that coordinates the LLM and TTS components.
• The system supports dynamic language switching mid-conversation without manual configuration, automatically selecting appropriate TTS voice models based on detected ISO-639-1 codes.
• Deployment requires only the --stt_language auto flag (default for Parakeet TDT) or explicit language codes for targeted localization.

Frequently Asked Questions

How does the system handle language switching during an active conversation?

The STT handlers continuously analyze each audio chunk for language changes. When using Parakeet TDT, the lingua detector runs on every transcription, updating the language_code field if the user switches languages. The Whisper handler similarly updates its language token each turn, with fallback to the previous language only if the new detection fails whitelist validation. This updated code propagates through the pipeline, causing the LLM to generate responses in the new language and the TTS to switch voice models accordingly.

What languages are supported by the automatic detection system?

The Parakeet TDT handler supports all languages detected by the lingua library, with specific supported languages listed in parakeet_tdt_handler.py (lines 30-80). The Whisper handler maintains a curated whitelist defined in whisper_stt_handler.py (lines 20-33). Both systems output standard ISO-639-1 codes (e.g., "en", "fr", "de") that downstream components use for model selection.

Can I disable automatic detection and force a specific language?

Yes. Pass the --stt_language argument with a specific ISO-639-1 code (e.g., --stt_language de for German) when starting the server. While this sets the initial language, the Parakeet TDT backend continues monitoring for language changes unless explicitly configured otherwise. For strict language locking, modify the handler initialization in parakeet_tdt_arguments.py to bypass the detection logic.

How does detected language affect TTS voice selection?

The language_code field in the Transcription message instructs the TTS handler to select voice models trained on that specific language. According to the pipeline implementation in s2s_pipeline.py, the system maps language codes to appropriate backends such as Qwen3-TTS, Kokoro-82M, or Pocket-TTS, ensuring that the spoken response matches the language of the user's original utterance without requiring manual voice switching.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}