# How to Implement Bilingual Voice Conversations with Language Prompts

> Implement bilingual voice conversations using language prompts. Pass language codes via TTSInput to enable automatic language switching in real-time speech applications.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-07-30

---

**Pass a `language_code` parameter through the `TTSInput` message class to route language metadata from speech-to-text (STT) detection through the LLM to the text-to-speech (TTS) handler, enabling automatic language switching during real-time conversations.**

The `huggingface/speech-to-speech` library enables real-time voice conversations using a modular pipeline architecture. By leveraging the `language_code` field in message dataclasses, developers can build **bilingual voice conversations with language prompts** that automatically detect, propagate, and synthesize speech in multiple languages within a single session.

## How Language Flows Through the Pipeline

The pipeline transports language metadata across three stages using specific message types. Understanding this flow is essential for implementing robust multilingual support.

When audio enters the system, the **STT handler** (such as Whisper or Parakeet) detects the spoken language and creates a `Transcription` object containing the text and `language_code`. This code propagates through the pipeline until it reaches the **TTS handler** (such as Qwen‑3‑TTS or Pocket‑TTS), which uses it to select the appropriate synthesis model.

According to the source code in [`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py), the `TTSInput` dataclass carries the language hint:

```python

# src/speech_to_speech/pipeline/messages.py

class TTSInput(PipelineMessage):
    text: str
    language_code: Optional[str] = None   # Language hint for the TTS backend

```

If the `language_code` field is `None`, the TTS handler falls back to the last-used language or defaults to `"en"`.

## The Three Core Message Types

### Transcription Messages

**Transcription** objects are produced by STT handlers and carry the detected `language_code` from the audio stream. In [`src/speech_to_speech/STT/whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/whisper_stt_handler.py), the Whisper handler normalizes language codes and appends an `-auto` suffix when detection is automatic (e.g., `"en-auto"`).

### TTSInput Messages

**TTSInput** messages hold the text to be spoken and the optional `language_code` that instructs the TTS model which language to synthesize. The pipeline's transcription notifier forwards the language from the STT output to the next turn.

### LLMResponse Messages

**LLMResponse** objects may embed a *voice-system prompt* that guides the model to prefer spoken output. The `build_voice_system_prompt()` function in [`src/speech_to_speech/LLM/voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/voice_prompt.py) includes rules such as *"Speak naturally"* and *"If unsure whether a tool is needed, just speak"*, ensuring the model produces spoken replies even when language switching occurs.

## Step-by-Step Implementation

### 1. Configure Explicit Language Codes

For controlled bilingual conversations, initialize `TTSInput` objects with explicit language codes:

```python
from speech_to_speech.pipeline.s2s_pipeline import SpeechToSpeechPipeline
from speech_to_speech.pipeline.messages import TTSInput

# Initialize the pipeline

pipeline = SpeechToSpeechPipeline()

# English turn

english_input = TTSInput(
    text="Hello, how are you?", 
    language_code="en"
)
pipeline.send_user_input(english_input)

# Spanish turn  

spanish_input = TTSInput(
    text="¿Cómo estás?", 
    language_code="es"
)
pipeline.send_user_input(spanish_input)

```

The `send_user_input()` method in [`src/speech_to_speech/pipeline/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/s2s_pipeline.py) wraps the `TTSInput` into a `UserAudioChunk`, processes it through the STT (which may verify or override the language), and forwards the `language_code` to the LLM. The TTS handler then reads this code and switches the synthesis model accordingly.

### 2. Enable Automatic Language Detection

To let the system detect language on the fly, set the session configuration to `"auto"`:

```python
pipeline.set_session_config({"audio": {"input": {"language": "auto"}}})

```

When using automatic detection, the STT handler emits codes like `"es-auto"`. The helper function `resolve_auto_language()` in [`src/speech_to_speech/LLM/utils.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/utils.py) strips the `-auto` suffix and maps Whisper language codes to the LLM's internal identifiers:

```python
from speech_to_speech.LLM.utils import resolve_auto_language

lang, llm_lang = resolve_auto_language("es-auto")

# Returns: lang == "es", llm_lang == "es"

```

This approach allows the pipeline to detect the spoken language via STT, propagate the detected code to the TTS handler, and fall back to the last-known language if detection fails.

## Complete Working Example

Below is a full implementation demonstrating automatic bilingual conversation handling:

```python

# demo/full_bilingual.py

from speech_to_speech.pipeline.s2s_pipeline import SpeechToSpeechPipeline
from speech_to_speech.pipeline.messages import TTSInput

# Initialize with auto-detection enabled

pipeline = SpeechToSpeechPipeline()
pipeline.set_session_config({"audio": {"input": {"language": "auto"}}})

# Turn 1 – English (detected automatically)

pipeline.send_user_input(
    TTSInput(text="Hey, what's the weather?", language_code=None)
)

# Turn 2 – Spanish (detected automatically)  

pipeline.send_user_input(
    TTSInput(text="¿Cuál es la temperatura en Madrid?", language_code=None)
)

```

Under the hood, this script:
1. Detects `"en"` for the first turn and `"es"` for the second via the Whisper STT handler
2. Maintains the voice-system prompt from [`voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/voice_prompt.py) to keep responses spoken-only
3. Switches the TTS language on-the-fly (e.g., Qwen‑3‑TTS with `language="es"`)

## Key Source Files and Functions

| File | Purpose |
|------|---------|
| [`src/speech_to_speech/LLM/voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/voice_prompt.py) | Defines `build_voice_system_prompt()` for spoken-only output guidance |
| [`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py) | Contains `TTSInput` dataclass with `language_code` field |
| [`src/speech_to_speech/LLM/utils.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/utils.py) | Implements `resolve_auto_language()` for normalizing auto-detected codes |
| [`src/speech_to_speech/STT/whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/whisper_stt_handler.py) | Performs language detection and manages the `-auto` suffix logic |
| [`src/speech_to_speech/TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py) | Reads `language_code` to select multilingual synthesis models |
| [`src/speech_to_speech/pipeline/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/s2s_pipeline.py) | Orchestrates the end-to-end flow via `send_user_input()` |

## Summary

- **Use `TTSInput.language_code`** to explicitly set or propagate language metadata through the pipeline
- **Enable `"auto"` detection** via session configuration to let STT handlers automatically identify spoken languages
- **Handle `-auto` suffixes** with `resolve_auto_language()` in [`src/speech_to_speech/LLM/utils.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/utils.py) when processing automatic detections
- **Leverage voice-system prompts** from [`voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/voice_prompt.py) to ensure the LLM produces spoken output suitable for voice synthesis
- **Configure fallback behavior** by understanding that TTS handlers default to `"en"` or the last-used language when `language_code` is `None`

## Frequently Asked Questions

### How does the pipeline handle language switching mid-conversation?

The pipeline treats each turn independently while maintaining a language state. When `send_user_input()` receives a new `TTSInput`, it processes the contained `language_code` (or detects it via STT if set to `"auto"`) and passes this metadata to the TTS handler. The TTS backend in [`qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/qwen3_tts_handler.py) (or equivalent) then switches synthesis models or voice parameters accordingly, allowing seamless transitions between languages within the same session.

### What happens if language detection fails?

According to the logic in [`src/speech_to_speech/STT/whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/whisper_stt_handler.py), the system falls back to the last-known language code when detection is uncertain. If no previous language exists, the TTS handler defaults to `"en"`. This ensures the conversation continues without interruption, though developers can override this behavior by explicitly setting `language_code` in `TTSInput` messages.

### Can I force a specific language instead of using auto-detection?

Yes. Instead of setting the session configuration to `"auto"`, omit the configuration or set explicit `language_code` values in your `TTSInput` objects (e.g., `"en"`, `"es"`, `"fr"`). When provided, these codes bypass automatic detection and instruct the TTS handler to use the specified language model directly, as implemented in the message handling logic of [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py).

### Which TTS handlers support multilingual output?

The reference implementation includes support in [`src/speech_to_speech/TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py), which reads the `language_code` field and passes it to the underlying Qwen‑3‑TTS model. Other handlers like [`pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/pocket_tts_handler.py) implement similar logic, reading the `language_code` from `TTSInput` to select appropriate synthesis parameters. Check your specific TTS handler implementation to confirm multilingual support.