# How Multi-Language Support Works in the Hugging Face Speech-to-Speech Pipeline

> Discover how the Hugging Face speech-to-speech pipeline masterfully handles multi-language support by seamlessly passing language codes through STT, LLM, and TTS components.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: internals
- Published: 2026-08-05

---

**The Speech-to-Speech pipeline handles multi-language support by propagating language codes from the STT stage through LLM processing to TTS generation, with normalization layers at each step to ensure components speak the same language.**

The `huggingface/speech-to-speech` repository implements a fully multilingual speech-to-speech system that automatically detects spoken languages and maintains language consistency across three pipeline stages: **automatic speech recognition (STT)**, **large language model (LLM) inference**, and **text-to-speech synthesis (TTS)**. Understanding this flow is essential for developers building localized voice applications or extending the pipeline with new language models.

## How Language Codes Flow Through the Pipeline

Multi-language support in the speech-to-speech pipeline relies on a **unified language propagation system** built on `Transcription` and handler-specific request/response objects defined in [`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py).

The pipeline uses three distinct language handling phases:

- **STT Detection** — STT handlers return `language_code` fields (e.g., `"en"`, `"fr-auto"`, `"zh-cn"`)
- **LLM Normalization** — The `resolve_auto_language` utility strips `-auto` suffixes and maps to human-readable names
- **TTS Adaptation** — Model-specific handlers like `Qwen3TTSHandler` convert ISO codes to model-native identifiers

## STT Stage: Language Detection and `Transcription` Objects

STT handlers in the speech-to-speech pipeline return `Transcription` objects that carry both recognized text and detected language information.

```python
from src.speech_to_speech.pipeline.messages import Transcription

transcript = Transcription(
    text="Bonjour, comment ça va?",
    language_code="fr-auto",           # Auto-detected French

    speech_stopped_at_s=1.23,
)

```

The `language_code` field uses **Whisper-style conventions** with optional `-auto` suffixes indicating automatic detection rather than user-specified language. STT backends like Parakeet and Moonshine populate this field internally before emitting the transcription event.

Key implementation files for this stage:

- [`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py) — Defines `Transcription` dataclass with `language_code: str`
- [`src/speech_to_speech/pipeline/events.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/events.py) — Event routing infrastructure
- [`src/speech_to_speech/pipeline/handler_types.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/handler_types.py) — Type definitions for cross-handler communication

## LLM Stage: Language Resolution and Prompt Injection

The LLM handler receives language information through `GenerateResponseRequest` objects and processes them using utilities in [`src/speech_to_speech/LLM/utils.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/utils.py).

### The `resolve_auto_language` Function

This utility strips the `-auto` suffix and maps language codes to human-readable names:

```python
from src.speech_to_speech.LLM.utils import resolve_auto_language

language_code, lang_name = resolve_auto_language("fr-auto")

# Returns: ("fr", "french")

language_code, lang_name = resolve_auto_language("zh-auto")

# Returns: ("zh", "chinese")

```

The mapping is powered by `WHISPER_LANGUAGE_TO_LLM_LANGUAGE`, which bridges Whisper's language codes with names suitable for LLM prompts.

### Optional Language Prompt Mode

When `--enable_lang_prompt` is enabled, the LLM handler injects a system message instructing the model to respond in the detected language. The prompt construction happens in two locations:

- [`src/speech_to_speech/LLM/voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/voice_prompt.py) — `build_voice_system_prompt()`
- [`src/speech_to_speech/LLM/text_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/text_prompt.py) — `build_text_system_prompt()`

```python

# Simplified flow from src/speech_to_speech/LLM/language_model.py

if enable_lang_prompt:
    _, lang_name = resolve_auto_language(request.language_code)
    system_content = f"Please reply to my message in {lang_name}."
    active_chat.add_item(make_user_message(system_content))

```

This ensures the LLM generates responses in the same language as the user's spoken input, even for multilingual models that might otherwise default to English.

## TTS Stage: Model-Specific Language Normalization

TTS handlers perform the final language adaptation, converting standard ISO codes to model-specific identifiers. The `Qwen3TTSHandler` in [`src/speech_to_speech/TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py) demonstrates this pattern.

### `QWEN3_LANGUAGE_ALIASES` and `_normalize_language`

The handler maintains an alias mapping and normalization method (lines 56-78):

```python

# From src/speech_to_speech/TTS/qwen3_tts_handler.py

QWEN3_LANGUAGE_ALIASES = {
    "en": "english",
    "en-us": "english",
    "en-gb": "english",
    "fr": "french",
    "fr-fr": "french",
    "zh": "chinese",
    "zh-cn": "chinese",
    "zh-tw": "chinese",
    # ... additional mappings

}

def _normalize_language(self, lang_code: str) -> str:
    """Convert various ISO-style codes to Qwen-3 native identifiers."""
    normalized = lang_code.lower().strip()
    return QWEN3_LANGUAGE_ALIASES.get(normalized, "english")

```

The normalized value is stored as `self.language` and passed to generation:

```python
handler = Qwen3TTSHandler()
handler.setup(language="fr-fr")  # Accepts flexible input

# Internal normalization

normalized = handler._normalize_language("fr-fr")  # "french"

# Used in generation call

model.generate_voice_clone_streaming(
    ...,
    language=handler.language,  # Model-native "french"

    ...
)

```

## Language Propagation Through Handler Architecture

The pipeline's `BaseHandler` subclass architecture ensures language fields persist across stage boundaries. Key type definitions in [`src/speech_to_speech/pipeline/handler_types.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/handler_types.py) include:

- `LLMIn` / `LLMOut` — Carry `language_code` from transcription to response
- `TTSIn` / `TTSOut` — Receive normalized language for synthesis

This design decouples language handling from business logic, allowing new STT, LLM, or TTS implementations to plug into the existing multi-language framework by adhering to the message contract.

## Summary

Multi-language support in the Hugging Face speech-to-speech pipeline works through:

- **Automatic detection** at the STT stage via `Transcription.language_code`
- **Flexible normalization** via `resolve_auto_language()` for LLM prompts and `_normalize_language()` for TTS models
- **Optional language steering** with `--enable_lang_prompt` to constrain LLM output language
- **Model-specific adaptation** through handler-implemented alias mappings like `QWEN3_LANGUAGE_ALIASES`
- **Type-safe propagation** through [`messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/messages.py) dataclasses and [`handler_types.py`](https://github.com/huggingface/speech-to-speech/blob/main/handler_types.py) protocols

## Frequently Asked Questions

### Does the speech-to-speech pipeline support real-time language switching?

The pipeline detects language per-utterance based on the STT handler's output, so consecutive utterances in different languages will be processed correctly. However, there is no explicit language-change event—each `Transcription` carries its own `language_code` independently.

### What happens if a language code isn't recognized by the TTS handler?

Handlers fall back to a default language. For `Qwen3TTSHandler`, the `_normalize_language` method defaults to `"english"` for unknown codes, ensuring the pipeline continues functioning rather than failing.

### Can I disable automatic language detection and force a specific language?

Yes—by omitting the `-auto` suffix and providing a bare language code (e.g., `"en"` instead of `"en-auto"`), the `resolve_auto_language` utility passes the code through unchanged, effectively bypassing auto-detection.

### Which STT backends support language detection in this pipeline?

The bundled Parakeet and Moonshine handlers support language detection and populate `language_code` accordingly. Custom STT handlers must implement the same `Transcription` output contract to participate in multi-language support.