# How to Integrate the Speech-to-Speech Pipeline with Hugging Face Inference Providers

> Integrate the speech-to-speech pipeline with Hugging Face Inference Providers by setting the llm_backend and configuring your API key and base URL for seamless routing.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-05

---

**Set `--llm_backend responses-api` and configure `base_url` plus `api_key` to route the language model step through any Hugging Face Inference endpoint.**

The **speech-to-speech** pipeline from Hugging Face is a modular real-time system connecting voice activity detection, speech-to-text, language model inference, and text-to-speech synthesis. Integrating it with **Hugging Face Inference Providers** requires selecting the correct backend and configuring endpoint credentials—no changes to other pipeline stages needed.

---

## Architecture Overview

The pipeline routes text prompts through a pluggable language model layer. When you select the `responses-api` backend, the system swaps the local LM handler for `ResponsesApiModelHandler`, which communicates with any OpenAI-compatible HTTP endpoint—including Hugging Face's Inference API.

Three components cooperate to enable this integration:

- **`ResponsesApiLanguageModelHandlerArguments`** — defines CLI flags for endpoint configuration
- **`get_llm_handler()`** — instantiates the correct handler based on backend selection
- **`ResponsesApiModelHandler`** — executes HTTP requests to the remote inference endpoint

All queue wiring remains identical regardless of backend choice, ensuring clean separation between transport and inference logic.

---

## Step 1: Configure the Responses API Backend

The argument dataclass in [`src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py) exposes these fields:

| Field | Purpose | Default |
|-------|---------|---------|
| `base_url` | Full URL of the HF Inference endpoint | `None` (required) |
| `api_key` | Hugging Face API token | `None` |
| `model_name` | Model identifier for the payload | `""` |
| `chat_size` | Maximum conversation turns | `16` |
| `init_chat_prompt` | System prompt for the session | `""` |

Example CLI invocation targeting a public endpoint:

```bash
speech-to-speech \
  --llm_backend responses-api \
  --responses_api_base_url https://api-inference.huggingface.co/models/Qwen/Qwen3-4B-Instruct-2507 \
  --responses_api_api_key $HF_API_TOKEN \
  --responses_api_model_name Qwen/Qwen3-4B-Instruct-2507 \
  --responses_api_chat_size 8

```

---

## Step 2: Backend Selection in the Pipeline

In [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py), the `get_llm_handler()` function branches on `module_kwargs.llm_backend`:

```python

# src/speech_to_speech/s2s_pipeline.py (conceptual)

def get_llm_handler(module_kwargs, responses_api_kwargs, ...):
    if module_kwargs.llm_backend == "responses-api":
        return ResponsesApiModelHandler(**vars(responses_api_kwargs))
    elif module_kwargs.llm_backend == "chat-completions-api":
        return ChatCompletionsApiModelHandler(...)
    else:
        return LanguageModelHandler(...)  # local transformers

```

The `vars()` call unpacks all `ResponsesApiLanguageModelHandlerArguments` fields as keyword arguments to the handler constructor.

---

## Step 3: Handler Implementation

`ResponsesApiModelHandler` resides in [`src/speech_to_speech/LLM/responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/responses_api_language_model.py) and inherits from `BaseOpenAICompatibleHandler`. Its core responsibilities:

1. **Read** from `text_prompt_queue` (STT output)
2. **Build** OpenAI-compatible chat-completion requests
3. **POST** to the configured `base_url` with `api_key` header
4. **Write** response text to `lm_response_queue` (TTS input)

The handler respects the `chat_size` window, maintaining rolling conversation history across turns.

---

## Complete Integration Examples

### CLI: Realtime Mode with HF Inference

```bash
speech-to-speech \
  --mode realtime \
  --llm_backend responses-api \
  --responses_api_base_url https://api-inference.huggingface.co/models/mistralai/Mistral-7B-Instruct-v0.2 \
  --responses_api_api_key $HF_API_TOKEN \
  --responses_api_model_name mistralai/Mistral-7B-Instruct-v0.2 \
  --stt whisper \
  --tts kokoro \
  --vad_handler_kwargs.smart_turn true

```

### Python: Programmatic Configuration

```python
from speech_to_speech.s2s_pipeline import (
    parse_arguments,
    prepare_all_args,
    initialize_queues_and_events,
    build_pipeline,
)

# Parse base configuration

args = parse_arguments()

# Force HF Inference backend

args.module_kwargs.llm_backend = "responses-api"
args.responses_api_language_model_handler_kwargs.base_url = (
    "https://api-inference.huggingface.co/models/Qwen/Qwen3-4B-Instruct-2507"
)
args.responses_api_language_model_handler_kwargs.api_key = "hf_XXXXXXXXXXXXXXXXXXXX"
args.responses_api_language_model_handler_kwargs.model_name = "Qwen/Qwen3-4B-Instruct-2507"

# Initialize pipeline infrastructure

prepare_all_args(
    args.module_kwargs,
    args.whisper_stt_handler_kwargs,
    args.responses_api_language_model_handler_kwargs,
    # ... other handler kwargs

)
queues_and_events = initialize_queues_and_events()

# Build and start

manager = build_pipeline(
    args.module_kwargs,
    args.socket_receiver_kwargs,
    args.socket_sender_kwargs,
    # ... remaining kwargs

    queues_and_events,
)
manager.start()
manager.wait()

```

### Local Audio I/O with Remote LM

```bash
speech-to-speech \
  --mode local \
  --llm_backend responses-api \
  --responses_api_base_url https://api-inference.huggingface.co/models/meta-llama/Meta-Llama-3-8B-Instruct \
  --responses_api_api_key $HF_API_TOKEN

```

The `--mode local` flag keeps audio processing on-device while delegating all language model inference to the remote Hugging Face endpoint.

---

## Queue Flow and Realtime Configuration

The pipeline queues are wired in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) within `_build_pipeline_handlers()`:

- `text_prompt_queue` → LM handler input (STT output)
- `lm_response_queue` → LM handler output (TTS input)

When running in realtime mode, `RealtimeConfig` from [`src/speech_to_speech/api/openai_realtime/runtime_config.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/runtime_config.py) propagates chat parameters (`chat_size`, `init_chat_prompt`) to the handler through `setup_kwargs`, ensuring consistent behavior across backend types.

---

## Key Source Files

| File Path | Purpose |
|-----------|---------|
| [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) | Central orchestration; backend selection via `get_llm_handler()` |
| [`src/speech_to_speech/LLM/responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/responses_api_language_model.py) | `ResponsesApiModelHandler` implementation |
| [`src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py) | CLI argument definitions for HF Inference |
| [`src/speech_to_speech/api/openai_realtime/runtime_config.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/runtime_config.py) | Realtime chat configuration |
| [`src/speech_to_speech/pipeline/handler_types.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/handler_types.py) | Type aliases for queue message types |

---

## Summary

- **Select backend:** Use `--llm_backend responses-api` to enable Hugging Face Inference Provider integration
- **Configure endpoint:** Set `base_url` and `api_key` via dedicated CLI flags or programmatic kwargs
- **Preserve pipeline:** VAD, STT, and TTS components require no changes when switching LM backends
- **Leverage compatibility:** The `responses-api` backend uses standard OpenAI chat-completion protocol, supporting any compatible endpoint

---

## Frequently Asked Questions

### What model formats work with the responses-api backend?

Any model exposed through Hugging Face Inference API or compatible OpenAI chat-completion endpoints works. This includes instruction-tuned LLMs like Mistral, Llama, Qwen, and proprietary models served via Hugging Face's dedicated inference infrastructure.

### Do I need GPU resources locally when using HF Inference Providers?

No. The `responses-api` backend offloads all language model computation to the remote endpoint. Local resources are only consumed by VAD, STT, and TTS stages—which can themselves be configured to use remote services if desired.

### How does conversation history persist across turns?

The `ResponsesApiModelHandler` maintains a rolling window of the last `chat_size` messages, passed as the `messages` array in each API request. For stateless endpoints, history is reconstructed client-side; for stateful WebSocket endpoints, additional connection management may apply.

### Can I use this backend with third-party OpenAI-compatible APIs?

Yes. The `responses-api` backend is protocol-compatible with any service implementing OpenAI's chat completions schema. Replace `base_url` with your provider's endpoint (e.g., `https://api.openai.com/v1`, `https://api.anthropic.com/v1`, or self-hosted `vLLM` instances).