# Setting Up HF Inference Providers with OpenAI-Compatible Endpoints in Speech-to-Speech

> Easily set up HF Inference Providers with OpenAI-compatible endpoints for speech-to-speech. Configure LLM backend, API base URL, and API key for seamless integration.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-09

---

**To configure HF Inference Providers with the speech-to-speech library, set `--llm_backend responses-api`, point `--responses_api_base_url` to `https://router.huggingface.co/v1`, and provide your HF token via `--responses_api_api_key`.**

The **huggingface/speech-to-speech** repository implements a low-latency, fully modular voice-agent pipeline that mirrors the OpenAI Realtime API. By leveraging OpenAI-compatible endpoints, you can swap the default language model backend to use HF Inference Providers without modifying any other pipeline component.

## Architecture Overview

The speech-to-speech system organizes its realtime engine into three distinct layers. The **pipeline orchestration** layer creates a `S2SPipeline` in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) that runs each component in its own thread, connecting them via queues and exposing a WebSocket/WebRTC server compatible with the OpenAI Realtime protocol.

The **component handlers** layer defines abstract base classes in [`src/speech_to_speech/baseHandler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/baseHandler.py) (`BaseHandler`) and concrete implementations for VAD, STT, LLM, and TTS. Each handler follows a common interface with `init`, `process`, and `shutdown` methods.

The **OpenAI-compatible backends** layer provides two language-model adapters in the `src/speech_to_speech/LLM/` directory: `ResponsesApiModelHandler` ([`responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/responses_api_language_model.py)) and `ChatCompletionsApiModelHandler` ([`chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/chat_completions_language_model.py)). Both inherit from `BaseOpenAICompatibleHandler` in [`base_openai_compatible_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/base_openai_compatible_language_model.py) and reuse the same connection flags (`responses_api_*`), allowing them to point at any OpenAI-compatible service including OpenAI itself, Hugging Face Inference Providers, vLLM, or llama.cpp.

## How the OpenAI-Compatible LLM Backend Works

### Configuration Selection

CLI flags `--llm_backend responses-api` (default) or `--llm_backend chat-completions` select the appropriate adapter. Connection details are supplied via the `responses_api_*` arguments defined in [`src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py), including `--responses_api_base_url`, `--responses_api_api_key`, and `--responses_api_reasoning_effort`.

### Handler Initialization

The `BaseOpenAICompatibleHandler` creates an `openai.OpenAI` client using the provided `api_key` and `base_url` (see line 189 of [`base_openai_compatible_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/base_openai_compatible_language_model.py)). This client handles all request and response operations for the LLM component.

### Streaming and Tool Calls

For `responses-api`, the handler streams the `response.create` event and supports tool-call payloads directly from the OpenAI-compatible server. For `chat-completions`, the same streaming logic applies but follows the chat-completion schema. Both implementations support server-side streaming required for low-latency voice interactions.

### Optional LLM Proxy

When `--enable_llm_proxy` is set, the realtime server exposes the configured LLM as a plain OpenAI-compatible endpoint at `/v1/chat/completions` or `/v1/responses`. This allows side-tasks such as summarization without interrupting the active voice conversation, as documented in [`src/speech_to_speech/api/openai_realtime/README.md`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/README.md).

## CLI Configuration for HF Inference Providers

HF Inference Providers expose an OpenAI-compatible endpoint at `https://router.huggingface.co/v1`. Configure the pipeline to use this endpoint with the following command:

```bash

# Install the library

pip install speech-to-speech

# Run the pipeline with HF Inference Provider backend

speech-to-speech local \
  --stt parakeet-tdt \
  --llm_backend responses-api \
  --tts qwen3 \
  --model_name "Qwen/Qwen3.5-9B:together" \
  --responses_api_base_url "https://router.huggingface.co/v1" \
  --responses_api_api_key "$HF_TOKEN" \
  --responses_api_stream \
  --enable_live_transcription

```

**Key configuration flags:**

- `--llm_backend responses-api` – Selects the `/v1/responses` adapter (default behavior).
- `--responses_api_base_url` – Base URL of the OpenAI-compatible endpoint (HF router).
- `--responses_api_api_key` – Your Hugging Face token for authentication.
- `--model_name` – Model identifier in the format `<repo>:<revision>` understood by the HF router.
- `--responses_api_stream` – Enables server-side streaming, required for low-latency voice responses.
- `--enable_live_transcription` – Forwards partial STT results to the LLM while the user is still speaking.

## Python Client Integration

The library works with the official OpenAI Python SDK by pointing it at the local speech-to-speech server or directly at the HF router. The following example creates a realtime connection, sends a `session.update` payload, and processes incoming events:

```python
from openai import OpenAI

# Connect to the locally-run speech-to-speech server

client = OpenAI(
    base_url="http://localhost:8765/v1",
    websocket_base_url="ws://localhost:8765/v1",
    api_key="unused",  # Server ignores key; gateway can enforce authentication

)

with client.realtime.connect(model="local") as conn:
    # Initialize session with VAD-driven turn detection

    conn.send({
        "type": "session.update",
        "session": {
            "type": "realtime",
            "instructions": "You are a helpful assistant.",
            "audio": {
                "input": {
                    "turn_detection": {
                        "type": "server_vad",
                        "interrupt_response": True,
                    }
                }
            },
        },
    })

    # Iterate over realtime events (transcriptions, tool calls, audio deltas)

    for event in conn:
        print(event.type)

```

To bypass the local server and connect directly to the HF Inference Provider router, replace the URLs with `https://router.huggingface.co/v1` and `wss://router.huggingface.co/v1`, and provide your HF token as the `api_key`.

## Summary

- The **speech-to-speech** pipeline uses a modular architecture where VAD, STT, LLM, and TTS components communicate through queues managed by `S2SPipeline`.
- **OpenAI-compatible backends** are implemented via `BaseOpenAICompatibleHandler` with two concrete adapters: `ResponsesApiModelHandler` and `ChatCompletionsApiModelHandler`.
- **HF Inference Providers** work by setting `--responses_api_base_url` to `https://router.huggingface.co/v1` and authenticating with your HF token.
- The system supports **streaming responses** and **tool calls** through the OpenAI-compatible interface, maintaining low latency for realtime voice interactions.
- An optional **LLM Proxy** mode exposes the backend as a standalone OpenAI-compatible endpoint for auxiliary tasks.

## Frequently Asked Questions

### What file handles the OpenAI client initialization in the speech-to-speech library?

The `BaseOpenAICompatibleHandler` class in [`src/speech_to_speech/LLM/base_openai_compatible_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/base_openai_compatible_language_model.py) handles initialization. At line 189, it instantiates the `openai.OpenAI` client using the `api_key` and `base_url` parameters passed from the CLI arguments.

### Can I use chat completions instead of the responses API with HF Inference Providers?

Yes. Change `--llm_backend` from `responses-api` to `chat-completions`. The `ChatCompletionsApiModelHandler` in [`src/speech_to_speech/LLM/chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat_completions_language_model.py) uses the same connection flags (`--responses_api_base_url`, `--responses_api_api_key`) to connect to the HF router's `/v1/chat/completions` endpoint instead of `/v1/responses`.

### How does the model name format work with HF Inference Providers?

The `--model_name` flag accepts identifiers in the format `<repository>:<revision>` (for example, `"Qwen/Qwen3.5-9B:together"`). The HF router at `https://router.huggingface.co/v1` parses this to route your request to the appropriate inference provider and model version.

### Is it possible to enable the LLM proxy when using HF Inference Providers?

Yes. Add the `--enable_llm_proxy` flag when starting the server. This exposes the configured HF Inference Provider backend as a local OpenAI-compatible endpoint (either `/v1/responses` or `/v1/chat/completions`), allowing other clients to perform text-based queries without disrupting the active voice pipeline.