How Multi-Language Support Works in the Hugging Face Speech-to-Speech Pipeline

The Speech-to-Speech pipeline handles multi-language support by propagating language codes from the STT stage through LLM processing to TTS generation, with normalization layers at each step to ensure components speak the same language.

The huggingface/speech-to-speech repository implements a fully multilingual speech-to-speech system that automatically detects spoken languages and maintains language consistency across three pipeline stages: automatic speech recognition (STT), large language model (LLM) inference, and text-to-speech synthesis (TTS). Understanding this flow is essential for developers building localized voice applications or extending the pipeline with new language models.

How Language Codes Flow Through the Pipeline

Multi-language support in the speech-to-speech pipeline relies on a unified language propagation system built on Transcription and handler-specific request/response objects defined in src/speech_to_speech/pipeline/messages.py.

The pipeline uses three distinct language handling phases:

• STT Detection — STT handlers return language_code fields (e.g., "en", "fr-auto", "zh-cn")
• LLM Normalization — The resolve_auto_language utility strips -auto suffixes and maps to human-readable names
• TTS Adaptation — Model-specific handlers like Qwen3TTSHandler convert ISO codes to model-native identifiers

STT Stage: Language Detection and Transcription Objects

STT handlers in the speech-to-speech pipeline return Transcription objects that carry both recognized text and detected language information.

from src.speech_to_speech.pipeline.messages import Transcription

transcript = Transcription(
    text="Bonjour, comment ça va?",
    language_code="fr-auto",           # Auto-detected French

    speech_stopped_at_s=1.23,
)

The language_code field uses Whisper-style conventions with optional -auto suffixes indicating automatic detection rather than user-specified language. STT backends like Parakeet and Moonshine populate this field internally before emitting the transcription event.

Key implementation files for this stage:

• src/speech_to_speech/pipeline/messages.py — Defines Transcription dataclass with language_code: str
• src/speech_to_speech/pipeline/events.py — Event routing infrastructure
• src/speech_to_speech/pipeline/handler_types.py — Type definitions for cross-handler communication

LLM Stage: Language Resolution and Prompt Injection

The LLM handler receives language information through GenerateResponseRequest objects and processes them using utilities in src/speech_to_speech/LLM/utils.py.

The resolve_auto_language Function

This utility strips the -auto suffix and maps language codes to human-readable names:

from src.speech_to_speech.LLM.utils import resolve_auto_language

language_code, lang_name = resolve_auto_language("fr-auto")

# Returns: ("fr", "french")

language_code, lang_name = resolve_auto_language("zh-auto")

# Returns: ("zh", "chinese")

The mapping is powered by WHISPER_LANGUAGE_TO_LLM_LANGUAGE, which bridges Whisper's language codes with names suitable for LLM prompts.

Optional Language Prompt Mode

When --enable_lang_prompt is enabled, the LLM handler injects a system message instructing the model to respond in the detected language. The prompt construction happens in two locations:

• src/speech_to_speech/LLM/voice_prompt.py — build_voice_system_prompt()
• src/speech_to_speech/LLM/text_prompt.py — build_text_system_prompt()

# Simplified flow from src/speech_to_speech/LLM/language_model.py

if enable_lang_prompt:
    _, lang_name = resolve_auto_language(request.language_code)
    system_content = f"Please reply to my message in {lang_name}."
    active_chat.add_item(make_user_message(system_content))

This ensures the LLM generates responses in the same language as the user's spoken input, even for multilingual models that might otherwise default to English.

TTS Stage: Model-Specific Language Normalization

TTS handlers perform the final language adaptation, converting standard ISO codes to model-specific identifiers. The Qwen3TTSHandler in src/speech_to_speech/TTS/qwen3_tts_handler.py demonstrates this pattern.

QWEN3_LANGUAGE_ALIASES and _normalize_language

The handler maintains an alias mapping and normalization method (lines 56-78):

# From src/speech_to_speech/TTS/qwen3_tts_handler.py

QWEN3_LANGUAGE_ALIASES = {
    "en": "english",
    "en-us": "english",
    "en-gb": "english",
    "fr": "french",
    "fr-fr": "french",
    "zh": "chinese",
    "zh-cn": "chinese",
    "zh-tw": "chinese",
    # ... additional mappings

}

def _normalize_language(self, lang_code: str) -> str:
    """Convert various ISO-style codes to Qwen-3 native identifiers."""
    normalized = lang_code.lower().strip()
    return QWEN3_LANGUAGE_ALIASES.get(normalized, "english")

The normalized value is stored as self.language and passed to generation:

handler = Qwen3TTSHandler()
handler.setup(language="fr-fr")  # Accepts flexible input

# Internal normalization

normalized = handler._normalize_language("fr-fr")  # "french"

# Used in generation call

model.generate_voice_clone_streaming(
    ...,
    language=handler.language,  # Model-native "french"

    ...
)

Language Propagation Through Handler Architecture

The pipeline's BaseHandler subclass architecture ensures language fields persist across stage boundaries. Key type definitions in src/speech_to_speech/pipeline/handler_types.py include:

• LLMIn / LLMOut — Carry language_code from transcription to response
• TTSIn / TTSOut — Receive normalized language for synthesis

This design decouples language handling from business logic, allowing new STT, LLM, or TTS implementations to plug into the existing multi-language framework by adhering to the message contract.

Summary

Multi-language support in the Hugging Face speech-to-speech pipeline works through:

• Automatic detection at the STT stage via Transcription.language_code
• Flexible normalization via resolve_auto_language() for LLM prompts and _normalize_language() for TTS models
• Optional language steering with --enable_lang_prompt to constrain LLM output language
• Model-specific adaptation through handler-implemented alias mappings like QWEN3_LANGUAGE_ALIASES
• Type-safe propagation through messages.py dataclasses and handler_types.py protocols

Frequently Asked Questions

Does the speech-to-speech pipeline support real-time language switching?

The pipeline detects language per-utterance based on the STT handler's output, so consecutive utterances in different languages will be processed correctly. However, there is no explicit language-change event—each Transcription carries its own language_code independently.

What happens if a language code isn't recognized by the TTS handler?

Handlers fall back to a default language. For Qwen3TTSHandler, the _normalize_language method defaults to "english" for unknown codes, ensuring the pipeline continues functioning rather than failing.

Can I disable automatic language detection and force a specific language?

Yes—by omitting the -auto suffix and providing a bare language code (e.g., "en" instead of "en-auto"), the resolve_auto_language utility passes the code through unchanged, effectively bypassing auto-detection.

Which STT backends support language detection in this pipeline?

The bundled Parakeet and Moonshine handlers support language detection and populate language_code accordingly. Custom STT handlers must implement the same Transcription output contract to participate in multi-language support.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}