# How to Configure Multi-Language Support with Automatic Language Detection in Hugging Face Speech-to-Speech

> Easily configure multi-language support with automatic language detection in Hugging Face Speech-to-Speech. Install lingua-py and set language to None for 25+ languages.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-07

---

**Install the optional `lingua-py` dependency and initialize your STT handler with `language=None` to enable real-time automatic language detection across 25+ supported languages.**

The Hugging Face `speech-to-speech` repository provides a modular architecture for speech-to-text (STT) and text-to-speech (TTS) pipelines. Configuring multi-language support with automatic language detection leverages the optional `lingua-py` integration within specific STT handlers, allowing the system to dynamically identify spoken languages and route them through appropriate translation and voice synthesis pathways.

## Architecture Overview

The library implements language detection through a **handler-based architecture** where STT handlers expose a `language_code` field in their transcription results. According to the source code in [`src/speech_to_speech/STT/parakeet_tdt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/parakeet_tdt_handler.py), the detection workflow follows three distinct stages:

- **STT Handler** (e.g., `ParakeetTDTSTTHandler`): Performs initial transcription and optionally invokes language detection
- **Language Detector** (`lingua.LanguageDetector`): Analyzes transcribed text to infer ISO-639-1 language codes
- **Pipeline Router** (`S2SPipeline`): Propagates the detected `language_code` to downstream LLM and TTS components

The core detection logic resides in the private method `_detect_language_from_text`, which maps Lingua's output (e.g., converting `"nb"` to `"no"` for Norwegian) to the library's internal conventions. This method is invoked automatically when the handler's `language` parameter is set to `None` and the `lingua-py` package is available.

Other handlers implementing this pattern include [`whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/whisper_stt_handler.py), [`mlx_audio_whisper_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/mlx_audio_whisper_handler.py), and [`lightning_whisper_mlx_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/lightning_whisper_mlx_handler.py), each exposing the same `language_code` interface in their `Transcription` results.

## Prerequisites: Installing the Language Detector

Automatic detection requires the optional `lingua-py` dependency. Without this package, handlers default to the user-specified language or fall back to English (`"en"`).

```bash
pip install lingua-py

```

The detector builds at handler initialization through the `_build_lingua_detector` method, which pre-loads language models for the codes defined in the `SUPPORTED_LANGUAGES` constant (25 European languages by default).

## Configuration Methods

### Programmatic Setup

To enable detection in Python, instantiate your pipeline without specifying a static language code:

```python
from speech_to_speech.s2s_pipeline import S2SPipeline
from speech_to_speech.STT.parakeet_tdt_handler import ParakeetTDTSTTHandler

# Initialize pipeline with auto-detection enabled

pipeline = S2SPipeline(
    stt_handler=ParakeetTDTSTTHandler(),
    tts_handler=None,
    llm_handler=None,
)

# Configure handler with language=None for automatic detection

pipeline.stt_handler.setup(
    device="auto",
    compute_type="float16",
    language=None,  # Triggers _detect_language_from_text

)

# Process audio

transcription = pipeline.transcribe(audio_path="sample.wav")
print(f"Detected: {transcription.language_code}")
print(f"Text: {transcription.text}")

```

**Key implementation detail:** The handler checks `if self.language is None` before invoking the detector. When detection succeeds, the `Transcription` object carries the ISO-639-1 code; when it fails, the system logs a warning and defaults to `"en"`.

### CLI Usage

From the command line, simply omit the `--language` flag:

```bash
speech-to-speech \
    --stt parakeet-tdt \
    --device auto \
    --audio-file samples/multi_lang.wav

```

The CLI entry point in [`src/speech_to_speech/cli.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/cli.py) forwards arguments to the handler's `setup` method. Without an explicit language argument, the handler automatically engages the detection routine.

## Customizing Language Support

The default `SUPPORTED_LANGUAGES` list in [`parakeet_tdt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/parakeet_tdt_handler.py) covers major European languages. To add support for additional languages (e.g., Chinese or Japanese), modify the constant before handler initialization:

```python

# In src/speech_to_speech/STT/parakeet_tdt_handler.py

SUPPORTED_LANGUAGES = [
    "en", "de", "fr", "es", "it", "pt", "nl", "pl", "ru",
    "zh",  # Added Chinese support

    "ja",  # Added Japanese support

]

```

After modifying the list, the `_build_lingua_detector` method will include these languages in its detection model. This approach applies to any STT handler implementing the detection interface.

## Pipeline Integration

Once detected, the `language_code` propagates through the system:

- **LLM Prompting**: The `VoicePrompt` and `TextPrompt` classes in [`src/speech_to_speech/LLM/voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/voice_prompt.py) and [`text_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/text_prompt.py) receive the language code and can switch system prompts or conversation contexts accordingly
- **TTS Selection**: TTS handlers like [`kokoro_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/kokoro_handler.py) map `language_code` to specific voice models using internal dictionaries (e.g., `kokoro_lang_code`), automatically selecting the appropriate accent and pronunciation
- **Logging**: Handlers emit `logger.info` messages showing detection results, enabling debugging of misidentified languages

## Summary

- **Install** `lingua-py` to enable the detection backend
- **Set** `language=None` in your STT handler configuration to trigger automatic detection
- **Modify** `SUPPORTED_LANGUAGES` in handlers like [`parakeet_tdt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/parakeet_tdt_handler.py) to customize the detection pool
- **Access** the detected code via `transcription.language_code` in your application logic
- **Propagate** language settings downstream to LLM and TTS handlers for end-to-end multi-language support

## Frequently Asked Questions

### What languages are supported by default?

The `ParakeetTDTSTTHandler` includes 25 European languages in its `SUPPORTED_LANGUAGES` constant, including English, German, French, Spanish, Italian, Portuguese, Dutch, Polish, and Russian. You can extend this list by editing the constant in [`src/speech_to_speech/STT/parakeet_tdt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/parakeet_tdt_handler.py) before runtime.

### Can I use automatic detection with Whisper-based handlers?

Yes. The [`whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/whisper_stt_handler.py), [`mlx_audio_whisper_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/mlx_audio_whisper_handler.py), and [`lightning_whisper_mlx_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/lightning_whisper_mlx_handler.py) files implement the same detection interface. They all check for the `lingua-py` dependency and expose the `language_code` field when `language=None` is specified during setup.

### How does the detected language affect voice output?

The `S2SPipeline` passes the `language_code` from the STT handler to the TTS handler. For example, [`kokoro_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/kokoro_handler.py) uses this code to select the appropriate voice model via its internal `kokoro_lang_code` mapping, ensuring the synthesized speech matches the detected input language.

### What happens if language detection fails?

If `lingua-py` cannot determine the language with sufficient confidence, the handler logs a warning and defaults to `"en"` (English). This fallback ensures the pipeline continues processing rather than raising an exception, though you should monitor logs in production to identify frequent misclassifications.