# Configuring Multi-Language Detection and Switching in HuggingFace Speech-to-Speech

> Enable multi language detection and switching for HuggingFace speech-to-speech. Set language to auto in STT and TTS handlers for seamless multilingual conversations.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-03

---

**To enable automatic language detection and dynamic switching in the speech-to-speech pipeline, set `language="auto"` in `WhisperSTTHandlerArguments` and `"auto"` (or a specific ISO-639-1 code) in your TTS handler arguments—the pipeline propagates detected language codes from STT through LLM to TTS for seamless multilingual conversations.**

The HuggingFace **speech-to-speech** library provides built-in support for multi-language detection and dynamic language switching across its processing pipeline. By configuring the STT component to auto-detect spoken language and ensuring downstream handlers respect the detected `language_code`, you can build applications that automatically respond in the same language as the user's input—even when languages change mid-conversation.

## Understanding Language Propagation in the Pipeline

The speech-to-speech architecture uses a message-passing system where language metadata travels alongside audio and text data. This propagation happens through three distinct stages: detection in STT, optional transformation in LLM, and synthesis in TTS.

### STT Language Detection with Whisper

The `WhisperSTTHandlerArguments` dataclass in [`src/speech_to_speech/arguments_classes/whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/whisper_stt_arguments.py) controls how the Whisper model handles incoming speech:

```python
@dataclass
class WhisperSTTHandlerArguments:
    stt_model_name: str = "distil-whisper/distil-large-v3"
    language: str = "en"  # Set to "auto" for detection

```

When `language="auto"`, Whisper runs internal language detection on each utterance and returns an **ISO-639-1 language code** (e.g., `"en"`, `"fr"`, `"es"`). The STT handler packages this as `language_code` in the transcription result.

The detection happens per utterance, so switching languages mid-conversation requires no additional configuration—Whisper automatically identifies the new language on the next speech segment.

### Message Types and Language Code Propagation

The pipeline's message types in [`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py) and [`src/speech_to_speech/pipeline/events.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/events.py) define the `language_code` field that carries language metadata:

- `RecognizeEvent` — carries STT output including detected language
- `UserSpeechEvent` — represents user input with language annotation
- Pipeline tuples — structured as `(text, language_code, tools)`

When the STT handler emits a transcription, the router forwards the complete payload preserving this metadata. No manual intervention is required to pass language information between pipeline stages.

## Configuring LLM and TTS for Multi-Language Output

### LLM Language Control

According to [`src/speech_to_speech/LLM/README.md`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/README.md), LLM handlers can use the incoming `language_code` to prepend control instructions that guide the model's response language. For example, detecting `"fr"` might prepend "Please answer in French." to the prompt.

This approach ensures the language model generates text in the same language as the detected speech, maintaining conversational coherence without requiring separate model instances per language.

### TTS Language Handling

All TTS handlers accept language configuration through their argument classes. The `Qwen3TTSHandler` in [`src/speech_to_speech/TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py) demonstrates this pattern:

```python

# From qwen3_tts_handler.py - language normalization and passing

def _normalize_language(self, lang_code: str) -> str:
    # Maps ISO-639-1 codes to model-specific format

    ...

def handle(self, event):
    lang = event.language_code if self.language == "auto" else self.language
    # Pass to synthesis model

```

When `qwen3_tts_language="auto"`, the handler extracts `language_code` from incoming events. Setting a concrete value like `"es"` forces Spanish synthesis regardless of detection results.

## Complete Implementation Example

This minimal configuration enables end-to-end automatic language detection and response:

```python
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline
from speech_to_speech.arguments_classes.whisper_stt_arguments import WhisperSTTHandlerArguments
from speech_to_speech.arguments_classes.qwen3_tts_arguments import Qwen3TTSArguments

# STT: Enable automatic language detection

stt_args = WhisperSTTHandlerArguments(
    stt_model_name="distil-whisper/distil-large-v3",
    language="auto",  # Key: triggers Whisper detection

)

# TTS: Accept detected language from pipeline

tts_args = Qwen3TTSArguments(
    qwen3_tts_language="auto",  # Uses incoming language_code

)

# Build and run pipeline

pipeline = SpeechToSpeechPipeline(
    stt_handler_kwargs=stt_args,
    tts_handler_kwargs=tts_args,
)

pipeline.run()  # Processes microphone input with auto language switching

```

**Execution flow:**
1. User speaks in French → Whisper detects `language_code="fr"`
2. Pipeline propagates `"fr"` through to TTS handler
3. Qwen-3-TTS synthesizes response in French
4. User switches to Spanish → Whisper detects `language_code="es"` on next utterance
5. TTS automatically switches to Spanish synthesis

## Advanced Configuration Patterns

### Force Fixed TTS Language

To override detection and always synthesize in a specific language:

```python
tts_args = Qwen3TTSArguments(
    qwen3_tts_language="de",  # Always German, ignores STT detection

)

```

### Runtime Language Switching

Call handler-specific methods to change language mid-session:

```python

# Force TTS to new language while STT continues auto-detection

pipeline.set_tts_language("it")  # Italian synthesis

```

The STT handler continues detecting actual spoken language, but responses synthesize in Italian until reconfigured.

### Per-Handler Language Overrides

Different handlers in the same pipeline can use different language strategies:

| Component | Configuration | Behavior |
|-----------|-------------|----------|
| STT | `language="auto"` | Detect actual spoken language |
| LLM | `language="auto"` (default) | Follow STT detection with instruction prepending |
| TTS | `language="fr"` | Force French output regardless |

## Key Source Files for Multi-Language Configuration

| File Path | Purpose |
|-----------|---------|
| [`src/speech_to_speech/arguments_classes/whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/whisper_stt_arguments.py) | STT language detection settings |
| [`src/speech_to_speech/TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py) | TTS language acceptance and normalization |
| [`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py) | Message type definitions with `language_code` |
| [`src/speech_to_speech/LLM/README.md`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/README.md) | LLM language control documentation |

## Summary

- **Enable detection**: Set `language="auto"` in `WhisperSTTHandlerArguments` to activate Whisper's per-utterance language detection
- **Propagate codes**: The pipeline automatically forwards `language_code` from STT through `RecognizeEvent` and related message types
- **Configure TTS**: Use `"auto"` to follow detection or specify ISO-639-1 codes for forced output languages
- **Switch dynamically**: Detection runs on every utterance; TTS can be reconfigured at runtime via handler methods
- **Control LLM output**: The LLM handler prepends language instructions based on incoming codes to ensure response language matching

## Frequently Asked Questions

### How does automatic language detection work in the speech-to-speech pipeline?

Whisper STT runs language identification on each audio segment when `language="auto"` is configured, returning ISO-639-1 codes like `"en"` or `"fr"`. The pipeline's message system carries this `language_code` through to downstream components without requiring manual handling.

### Can I force a specific output language while keeping input detection?

Yes. Configure STT with `language="auto"` for detection, but set your TTS handler arguments to a concrete language code like `qwen3_tts_language="es"`. The system detects the input language for logging or LLM context, but always synthesizes in the specified TTS language.

### What happens if someone switches languages mid-conversation?

Whisper detects the new language on the next utterance and the pipeline propagates the updated `language_code`. If TTS is configured with `"auto"`, synthesis switches automatically. No pipeline restart or code changes are required.

### Which TTS handlers support automatic language switching?

The `Qwen3TTSHandler` explicitly implements this pattern with its `qwen3_tts_language` argument. Other TTS handlers in the repository follow similar conventions—check their respective argument classes for `language` or `language_code` configuration options.