# Supporting Multi-Language Voice Conversations with Automatic Language Detection in Speech-to-Speech

> Enable multi-language voice conversations with automatic language detection in speech-to-speech. Our pipeline supports ISO-639-1 codes for seamless LLM and TTS coordination.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: deep-dive
- Published: 2026-08-09

---

**The Speech-to-Speech pipeline achieves multi-language voice conversations with automatic language detection by processing audio through STT handlers that detect ISO-639-1 language codes using either the lingua library or Whisper's native tokens, then propagating these codes through the Transcription message to coordinate multilingual responses across the LLM and TTS stages.**

The huggingface/speech-to-speech repository implements a modular, full-duplex voice agent architecture designed for seamless multi-language voice conversations with automatic language detection. The system decomposes the pipeline into four interchangeable components—Voice Activity Detection (VAD), Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS)—each running in isolated threads and communicating via queues. Language detection occurs exclusively in the STT stage, creating a single source of truth that flows downstream to ensure consistent multilingual dialogue without manual intervention.

## STT-Based Language Detection Architecture

Automatic language detection is implemented within the STT handlers, which analyze incoming audio streams and attach detected language codes to transcription objects. The repository provides two primary backends for this functionality, each handling detection differently while emitting a standardized `Transcription` message.

### Parakeet TDT with Lingua Library

The default STT handler utilizes the **Parakeet TDT** model combined with the **lingua** language detection library. Located in [`src/speech_to_speech/STT/parakeet_tdt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/parakeet_tdt_handler.py) (lines 30-80), the `ParakeetTDTSTTHandler` class loads the multilingual model at startup and initializes the lingua detector to avoid latency on first use. After each transcription, the handler runs the detector on the predicted text, producing an ISO-639-1 language code that is attached to the `Transcription` message via the `language_code` field.

### Whisper Language Token Validation

Alternatively, the **Whisper STT** handler leverages the model's built-in language token generation. As implemented in [`src/speech_to_speech/STT/whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/whisper_stt_handler.py) (lines 20-33), the `WhisperSTTHandler` class extracts the language token from Whisper's output and validates it against an internal whitelist. If the detected language is unsupported, the system falls back to the last known language code, ensuring continuity in multi-turn conversations.

## Language Propagation Through the Pipeline

Both STT handlers emit a `Transcription` object defined in [`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py). This dataclass carries the detected language downstream:

```python
Transcription(
    text=pred_text,
    language_code=language_code,   # e.g. "en", "fr", "es‑auto"

    turn_id=...,
    turn_revision=...,
    speech_stopped_at_s=...,
)

```

The pipeline orchestrator in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) forwards this message to the LLM component. Through `LanguageModelHandlerArguments` (defined in [`src/speech_to_speech/arguments_classes/language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/language_model_arguments.py)), the LLM can be instructed to honor the `language_code`, generating responses in the same language as the user's utterance. The TTS handler subsequently selects the appropriate voice model—such as Qwen3-TTS, Kokoro-82M, or Pocket-TTS—based on the propagated language code, completing the multilingual loop.

## Implementing Multi-Language Support with Automatic Detection

### Server Deployment with Auto-Detect

To launch a multilingual conversation server using the default Parakeet TDT backend with automatic detection enabled:

```bash

# Install the library (includes Parakeet TDT and lingua)

pip install speech-to-speech

# Start the realtime server (auto-detect enabled)

speech-to-speech serve \
    --stt parakeet-tdt   # language auto-detect is default

```

To force a specific starting language while retaining auto-detection for subsequent turns:

```bash
speech-to-speech serve \
    --stt parakeet-tdt \
    --stt_language de    # sets starting language; model will still auto-detect later turns

```

### Switching to Whisper STT

For deployments preferring Whisper's detection mechanism:

```bash
speech-to-speech serve \
    --stt whisper \
    --stt_language auto

```

### Programmatic Access to Detected Languages

For custom integrations, instantiate the handler directly to retrieve language codes:

```python
from speech_to_speech.pipeline.messages import Transcription
from speech_to_speech.STT.parakeet_tdt_handler import ParakeetTDTSTTHandler

stt = ParakeetTDTSTTHandler()
stt.setup(language="auto")          # enable auto-detect

# Assume `audio_chunk` is a NumPy array captured from VAD

for result in stt.process(audio_chunk):
    if isinstance(result, Transcription):
        print(f"User said: {result.text}")
        print(f"Detected language: {result.language_code}")   # e.g. "fr"

```

## Summary

- **Automatic language detection** occurs exclusively in the STT stage via `ParakeetTDTSTTHandler` or `WhisperSTTHandler`, preventing latency by pre-loading detectors at startup.
- The **lingua library** powers detection for Parakeet TDT, while **Whisper** uses native language tokens with whitelist validation and fallback logic.
- Language information flows through the `Transcription` dataclass in [`messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/messages.py), creating a single source of truth that coordinates the LLM and TTS components.
- The system supports **dynamic language switching** mid-conversation without manual configuration, automatically selecting appropriate TTS voice models based on detected ISO-639-1 codes.
- Deployment requires only the `--stt_language auto` flag (default for Parakeet TDT) or explicit language codes for targeted localization.

## Frequently Asked Questions

### How does the system handle language switching during an active conversation?

The STT handlers continuously analyze each audio chunk for language changes. When using **Parakeet TDT**, the lingua detector runs on every transcription, updating the `language_code` field if the user switches languages. The **Whisper** handler similarly updates its language token each turn, with fallback to the previous language only if the new detection fails whitelist validation. This updated code propagates through the pipeline, causing the LLM to generate responses in the new language and the TTS to switch voice models accordingly.

### What languages are supported by the automatic detection system?

The **Parakeet TDT handler** supports all languages detected by the lingua library, with specific supported languages listed in [`parakeet_tdt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/parakeet_tdt_handler.py) (lines 30-80). The **Whisper handler** maintains a curated whitelist defined in [`whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/whisper_stt_handler.py) (lines 20-33). Both systems output standard ISO-639-1 codes (e.g., "en", "fr", "de") that downstream components use for model selection.

### Can I disable automatic detection and force a specific language?

Yes. Pass the `--stt_language` argument with a specific ISO-639-1 code (e.g., `--stt_language de` for German) when starting the server. While this sets the initial language, the Parakeet TDT backend continues monitoring for language changes unless explicitly configured otherwise. For strict language locking, modify the handler initialization in [`parakeet_tdt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/parakeet_tdt_arguments.py) to bypass the detection logic.

### How does detected language affect TTS voice selection?

The `language_code` field in the `Transcription` message instructs the TTS handler to select voice models trained on that specific language. According to the pipeline implementation in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py), the system maps language codes to appropriate backends such as Qwen3-TTS, Kokoro-82M, or Pocket-TTS, ensuring that the spoken response matches the language of the user's original utterance without requiring manual voice switching.