How to Configure Multi-Language Support with Automatic Language Detection in Speech-to-Speech

Install the optional lingua-py package and initialize any supported STT handler (such as ParakeetTDTSTTHandler) with language=None to enable real-time automatic language detection across 25+ European languages.

The Hugging Face speech-to-speech repository provides a modular pipeline for real-time speech-to-text-to-speech conversion. Configuring multi-language support with automatic language detection allows the system to dynamically identify spoken languages from audio input and route transcriptions to appropriate TTS voices and LLM prompts without manual configuration.

Architecture Overview

The library implements multi-language support through STT handlers that optionally detect the input language using the lingua library. When enabled, the detection pipeline follows this flow:

• The STT Handler (e.g., ParakeetTDTSTTHandler) transcribes audio and runs _detect_language_from_text on the result.
• The Language Detector (lingua.LanguageDetector) infers the ISO-639-1 language code from the transcription text.
• The Pipeline (S2SPipeline) receives a Transcription object containing the language_code and propagates it to downstream components.

The core implementation resides in src/speech_to_speech/STT/parakeet_tdt_handler.py. This handler defines SUPPORTED_LANGUAGES covering 25 European languages and implements _build_lingua_detector to initialize the detector at startup. The private method _detect_language_from_text handles edge cases such as mapping Lingua's "nb" (Norwegian Bokmål) to the library's internal "no" code.

Other compatible handlers include whisper_stt_handler.py, mlx_audio_whisper_handler.py, and lightning_whisper_mlx_handler.py, all of which expose a language_code field in their transcription results.

Installing the Language Detector

Automatic detection requires the optional lingua-py dependency. Without this package, handlers default to the user-specified language or "en".

pip install lingua-py

Enabling Automatic Detection

To activate multi-language support, configure your STT handler without specifying a static language code.

Programmatic Configuration

Instantiate S2SPipeline with an STT handler and set language=None during setup:

from speech_to_speech.s2s_pipeline import S2SPipeline
from speech_to_speech.STT.parakeet_tdt_handler import ParakeetTDTSTTHandler

# Initialize pipeline

pipeline = S2SPipeline(
    stt_handler=ParakeetTDTSTTHandler(),
    tts_handler=None,
    llm_handler=None,
)

# Configure for automatic detection

pipeline.stt_handler.setup(
    device="auto",
    compute_type="float16",
    language=None,  # Triggers automatic detection

)

# Process audio

transcription = pipeline.transcribe(audio_path="sample.wav")
print(f"Detected: {transcription.language_code}")
print(f"Text: {transcription.text}")

When language=None, the handler invokes _detect_language_from_text on each transcription, loading the Lingua detector at startup via _build_lingua_detector.

CLI Configuration

Run the pipeline from the command line without the --language flag:

speech-to-speech \
    --stt parakeet-tdt \
    --device auto \
    --audio-file samples/multi_lang.wav

The CLI forwards arguments to ParakeetTDTSTTHandler.setup, and the omission of --language enables the detection routine.

Customizing Language Support

The default SUPPORTED_LANGUAGES list in parakeet_tdt_handler.py covers common European languages. To add support for additional languages, modify the constant in the source file:

# In src/speech_to_speech/STT/parakeet_tdt_handler.py

SUPPORTED_LANGUAGES = [
    "en", "de", "fr", "es", "it", "pt", "nl", "pl", "ru",
    "zh",  # Added Chinese support

    # ... additional ISO-639-1 codes

]

After editing, restart the pipeline. The detector will consider the new language during inference.

Integration with Pipeline Components

The detected language_code propagates through the entire pipeline:

• LLM Prompting: The VoicePrompt and TextPrompt classes in speech_to_speech/LLM/voice_prompt.py and speech_to_speech/LLM/text_prompt.py adapt prompts based on the detected language.
• TTS Selection: TTS handlers such as kokoro_handler.py use language-specific mappings (e.g., kokoro_lang_code) to select appropriate voices for the detected language.
• Logging: Handlers log detection events via logger.info and logger.warning, enabling debugging of detection failures.

Summary

• Install lingua-py to enable the automatic language detection capability.
• Set language=None in ParakeetTDTSTTHandler.setup() or omit --language from CLI commands to trigger detection.
• The system supports 25 European languages by default via the SUPPORTED_LANGUAGES constant in parakeet_tdt_handler.py.
• Detected language codes flow through S2SPipeline to influence TTS voice selection and LLM prompting automatically.
• Extend support to non-European languages by editing the SUPPORTED_LANGUAGES list in the handler source code.

Frequently Asked Questions

Which STT handlers support automatic language detection?

The ParakeetTDTSTTHandler in parakeet_tdt_handler.py provides built-in detection via the _detect_language_from_text method. The whisper_stt_handler.py, mlx_audio_whisper_handler.py, and lightning_whisper_mlx_handler.py handlers also support automatic detection when lingua-py is installed and language is set to None.

What happens if I don't install lingua-py?

Without the lingua-py package installed, the STT handlers skip the detection routine entirely. The system defaults to the language specified by the user during setup, or falls back to "en" (English) if no language is provided.

Can I restrict automatic detection to specific languages only?

Yes. Edit the SUPPORTED_LANGUAGES list in your chosen handler file (e.g., src/speech_to_speech/STT/parakeet_tdt_handler.py) to include only the ISO-639-1 codes you want to detect. The lingua.LanguageDetector will only consider these languages during inference, improving accuracy and reducing latency.

How does the detected language affect the TTS output?

The language_code field in the Transcription object is passed through S2SPipeline to the TTS handler. For example, kokoro_handler.py maps this code to a specific kokoro_lang_code and voice, ensuring the response is synthesized in the same language as the input audio.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}