How to Configure Multi-Language Support with Automatic Language Detection in Hugging Face Speech-to-Speech

Install the optional lingua-py dependency and initialize your STT handler with language=None to enable real-time automatic language detection across 25+ supported languages.

The Hugging Face speech-to-speech repository provides a modular architecture for speech-to-text (STT) and text-to-speech (TTS) pipelines. Configuring multi-language support with automatic language detection leverages the optional lingua-py integration within specific STT handlers, allowing the system to dynamically identify spoken languages and route them through appropriate translation and voice synthesis pathways.

Architecture Overview

The library implements language detection through a handler-based architecture where STT handlers expose a language_code field in their transcription results. According to the source code in src/speech_to_speech/STT/parakeet_tdt_handler.py, the detection workflow follows three distinct stages:

• STT Handler (e.g., ParakeetTDTSTTHandler): Performs initial transcription and optionally invokes language detection
• Language Detector (lingua.LanguageDetector): Analyzes transcribed text to infer ISO-639-1 language codes
• Pipeline Router (S2SPipeline): Propagates the detected language_code to downstream LLM and TTS components

The core detection logic resides in the private method _detect_language_from_text, which maps Lingua's output (e.g., converting "nb" to "no" for Norwegian) to the library's internal conventions. This method is invoked automatically when the handler's language parameter is set to None and the lingua-py package is available.

Other handlers implementing this pattern include whisper_stt_handler.py, mlx_audio_whisper_handler.py, and lightning_whisper_mlx_handler.py, each exposing the same language_code interface in their Transcription results.

Prerequisites: Installing the Language Detector

Automatic detection requires the optional lingua-py dependency. Without this package, handlers default to the user-specified language or fall back to English ("en").

pip install lingua-py

The detector builds at handler initialization through the _build_lingua_detector method, which pre-loads language models for the codes defined in the SUPPORTED_LANGUAGES constant (25 European languages by default).

Configuration Methods

Programmatic Setup

To enable detection in Python, instantiate your pipeline without specifying a static language code:

from speech_to_speech.s2s_pipeline import S2SPipeline
from speech_to_speech.STT.parakeet_tdt_handler import ParakeetTDTSTTHandler

# Initialize pipeline with auto-detection enabled

pipeline = S2SPipeline(
    stt_handler=ParakeetTDTSTTHandler(),
    tts_handler=None,
    llm_handler=None,
)

# Configure handler with language=None for automatic detection

pipeline.stt_handler.setup(
    device="auto",
    compute_type="float16",
    language=None,  # Triggers _detect_language_from_text

)

# Process audio

transcription = pipeline.transcribe(audio_path="sample.wav")
print(f"Detected: {transcription.language_code}")
print(f"Text: {transcription.text}")

Key implementation detail: The handler checks if self.language is None before invoking the detector. When detection succeeds, the Transcription object carries the ISO-639-1 code; when it fails, the system logs a warning and defaults to "en".

CLI Usage

From the command line, simply omit the --language flag:

speech-to-speech \
    --stt parakeet-tdt \
    --device auto \
    --audio-file samples/multi_lang.wav

The CLI entry point in src/speech_to_speech/cli.py forwards arguments to the handler's setup method. Without an explicit language argument, the handler automatically engages the detection routine.

Customizing Language Support

The default SUPPORTED_LANGUAGES list in parakeet_tdt_handler.py covers major European languages. To add support for additional languages (e.g., Chinese or Japanese), modify the constant before handler initialization:

# In src/speech_to_speech/STT/parakeet_tdt_handler.py

SUPPORTED_LANGUAGES = [
    "en", "de", "fr", "es", "it", "pt", "nl", "pl", "ru",
    "zh",  # Added Chinese support

    "ja",  # Added Japanese support

]

After modifying the list, the _build_lingua_detector method will include these languages in its detection model. This approach applies to any STT handler implementing the detection interface.

Pipeline Integration

Once detected, the language_code propagates through the system:

• LLM Prompting: The VoicePrompt and TextPrompt classes in src/speech_to_speech/LLM/voice_prompt.py and text_prompt.py receive the language code and can switch system prompts or conversation contexts accordingly
• TTS Selection: TTS handlers like kokoro_handler.py map language_code to specific voice models using internal dictionaries (e.g., kokoro_lang_code), automatically selecting the appropriate accent and pronunciation
• Logging: Handlers emit logger.info messages showing detection results, enabling debugging of misidentified languages

Summary

• Install lingua-py to enable the detection backend
• Set language=None in your STT handler configuration to trigger automatic detection
• Modify SUPPORTED_LANGUAGES in handlers like parakeet_tdt_handler.py to customize the detection pool
• Access the detected code via transcription.language_code in your application logic
• Propagate language settings downstream to LLM and TTS handlers for end-to-end multi-language support

Frequently Asked Questions

What languages are supported by default?

The ParakeetTDTSTTHandler includes 25 European languages in its SUPPORTED_LANGUAGES constant, including English, German, French, Spanish, Italian, Portuguese, Dutch, Polish, and Russian. You can extend this list by editing the constant in src/speech_to_speech/STT/parakeet_tdt_handler.py before runtime.

Can I use automatic detection with Whisper-based handlers?

Yes. The whisper_stt_handler.py, mlx_audio_whisper_handler.py, and lightning_whisper_mlx_handler.py files implement the same detection interface. They all check for the lingua-py dependency and expose the language_code field when language=None is specified during setup.

How does the detected language affect voice output?

The S2SPipeline passes the language_code from the STT handler to the TTS handler. For example, kokoro_handler.py uses this code to select the appropriate voice model via its internal kokoro_lang_code mapping, ensuring the synthesized speech matches the detected input language.

What happens if language detection fails?

If lingua-py cannot determine the language with sufficient confidence, the handler logs a warning and defaults to "en" (English). This fallback ensures the pipeline continues processing rather than raising an exception, though you should monitor logs in production to identify frequent misclassifications.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}