Setting Up HF Inference Providers with OpenAI-Compatible Endpoints in Speech-to-Speech

To configure HF Inference Providers with the speech-to-speech library, set --llm_backend responses-api, point --responses_api_base_url to https://router.huggingface.co/v1, and provide your HF token via --responses_api_api_key.

The huggingface/speech-to-speech repository implements a low-latency, fully modular voice-agent pipeline that mirrors the OpenAI Realtime API. By leveraging OpenAI-compatible endpoints, you can swap the default language model backend to use HF Inference Providers without modifying any other pipeline component.

Architecture Overview

The speech-to-speech system organizes its realtime engine into three distinct layers. The pipeline orchestration layer creates a S2SPipeline in src/speech_to_speech/s2s_pipeline.py that runs each component in its own thread, connecting them via queues and exposing a WebSocket/WebRTC server compatible with the OpenAI Realtime protocol.

The component handlers layer defines abstract base classes in src/speech_to_speech/baseHandler.py (BaseHandler) and concrete implementations for VAD, STT, LLM, and TTS. Each handler follows a common interface with init, process, and shutdown methods.

The OpenAI-compatible backends layer provides two language-model adapters in the src/speech_to_speech/LLM/ directory: ResponsesApiModelHandler (responses_api_language_model.py) and ChatCompletionsApiModelHandler (chat_completions_language_model.py). Both inherit from BaseOpenAICompatibleHandler in base_openai_compatible_language_model.py and reuse the same connection flags (responses_api_*), allowing them to point at any OpenAI-compatible service including OpenAI itself, Hugging Face Inference Providers, vLLM, or llama.cpp.

How the OpenAI-Compatible LLM Backend Works

Configuration Selection

CLI flags --llm_backend responses-api (default) or --llm_backend chat-completions select the appropriate adapter. Connection details are supplied via the responses_api_* arguments defined in src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py, including --responses_api_base_url, --responses_api_api_key, and --responses_api_reasoning_effort.

Handler Initialization

The BaseOpenAICompatibleHandler creates an openai.OpenAI client using the provided api_key and base_url (see line 189 of base_openai_compatible_language_model.py). This client handles all request and response operations for the LLM component.

Streaming and Tool Calls

For responses-api, the handler streams the response.create event and supports tool-call payloads directly from the OpenAI-compatible server. For chat-completions, the same streaming logic applies but follows the chat-completion schema. Both implementations support server-side streaming required for low-latency voice interactions.

Optional LLM Proxy

When --enable_llm_proxy is set, the realtime server exposes the configured LLM as a plain OpenAI-compatible endpoint at /v1/chat/completions or /v1/responses. This allows side-tasks such as summarization without interrupting the active voice conversation, as documented in src/speech_to_speech/api/openai_realtime/README.md.

CLI Configuration for HF Inference Providers

HF Inference Providers expose an OpenAI-compatible endpoint at https://router.huggingface.co/v1. Configure the pipeline to use this endpoint with the following command:

# Install the library

pip install speech-to-speech

# Run the pipeline with HF Inference Provider backend

speech-to-speech local \
  --stt parakeet-tdt \
  --llm_backend responses-api \
  --tts qwen3 \
  --model_name "Qwen/Qwen3.5-9B:together" \
  --responses_api_base_url "https://router.huggingface.co/v1" \
  --responses_api_api_key "$HF_TOKEN" \
  --responses_api_stream \
  --enable_live_transcription

Key configuration flags:

• --llm_backend responses-api – Selects the /v1/responses adapter (default behavior).
• --responses_api_base_url – Base URL of the OpenAI-compatible endpoint (HF router).
• --responses_api_api_key – Your Hugging Face token for authentication.
• --model_name – Model identifier in the format <repo>:<revision> understood by the HF router.
• --responses_api_stream – Enables server-side streaming, required for low-latency voice responses.
• --enable_live_transcription – Forwards partial STT results to the LLM while the user is still speaking.

Python Client Integration

The library works with the official OpenAI Python SDK by pointing it at the local speech-to-speech server or directly at the HF router. The following example creates a realtime connection, sends a session.update payload, and processes incoming events:

from openai import OpenAI

# Connect to the locally-run speech-to-speech server

client = OpenAI(
    base_url="http://localhost:8765/v1",
    websocket_base_url="ws://localhost:8765/v1",
    api_key="unused",  # Server ignores key; gateway can enforce authentication

)

with client.realtime.connect(model="local") as conn:
    # Initialize session with VAD-driven turn detection

    conn.send({
        "type": "session.update",
        "session": {
            "type": "realtime",
            "instructions": "You are a helpful assistant.",
            "audio": {
                "input": {
                    "turn_detection": {
                        "type": "server_vad",
                        "interrupt_response": True,
                    }
                }
            },
        },
    })

    # Iterate over realtime events (transcriptions, tool calls, audio deltas)

    for event in conn:
        print(event.type)

To bypass the local server and connect directly to the HF Inference Provider router, replace the URLs with https://router.huggingface.co/v1 and wss://router.huggingface.co/v1, and provide your HF token as the api_key.

Summary

• The speech-to-speech pipeline uses a modular architecture where VAD, STT, LLM, and TTS components communicate through queues managed by S2SPipeline.
• OpenAI-compatible backends are implemented via BaseOpenAICompatibleHandler with two concrete adapters: ResponsesApiModelHandler and ChatCompletionsApiModelHandler.
• HF Inference Providers work by setting --responses_api_base_url to https://router.huggingface.co/v1 and authenticating with your HF token.
• The system supports streaming responses and tool calls through the OpenAI-compatible interface, maintaining low latency for realtime voice interactions.
• An optional LLM Proxy mode exposes the backend as a standalone OpenAI-compatible endpoint for auxiliary tasks.

Frequently Asked Questions

What file handles the OpenAI client initialization in the speech-to-speech library?

The BaseOpenAICompatibleHandler class in src/speech_to_speech/LLM/base_openai_compatible_language_model.py handles initialization. At line 189, it instantiates the openai.OpenAI client using the api_key and base_url parameters passed from the CLI arguments.

Can I use chat completions instead of the responses API with HF Inference Providers?

Yes. Change --llm_backend from responses-api to chat-completions. The ChatCompletionsApiModelHandler in src/speech_to_speech/LLM/chat_completions_language_model.py uses the same connection flags (--responses_api_base_url, --responses_api_api_key) to connect to the HF router's /v1/chat/completions endpoint instead of /v1/responses.

How does the model name format work with HF Inference Providers?

The --model_name flag accepts identifiers in the format <repository>:<revision> (for example, "Qwen/Qwen3.5-9B:together"). The HF router at https://router.huggingface.co/v1 parses this to route your request to the appropriate inference provider and model version.

Is it possible to enable the LLM proxy when using HF Inference Providers?

Yes. Add the --enable_llm_proxy flag when starting the server. This exposes the configured HF Inference Provider backend as a local OpenAI-compatible endpoint (either /v1/responses or /v1/chat/completions), allowing other clients to perform text-based queries without disrupting the active voice pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}