# How to Use an OpenAI-Compatible API for the LLM Backend in speech-to-speech

> Integrate an OpenAI-compatible API with the huggingface speech-to-speech pipeline. Connect to Ollama, Azure OpenAI, or self-hosted LLMs using the responses_api backend.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-11

---

**Use the `responses_api` backend with `--llm-api-base` to point the huggingface/speech-to-speech pipeline at any OpenAI-compatible endpoint such as Ollama, Azure OpenAI, or a self-hosted server.**

The **speech-to-speech** library decouples the LLM component through a pluggable backend system. This design lets you swap the default language model for any server implementing the OpenAI HTTP API specification without modifying core pipeline code.

## Architecture Overview

The OpenAI-compatible integration spans five key components:

### Backend Registry

[`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) maintains a mapping from backend names to factory functions. When you specify `--llm-backend responses_api`, the registry instantiates the appropriate handler.

### Argument Configuration

[`src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py) declares the parameters needed to connect to an OpenAI-compatible server:

- `api_base` – base URL of the server (default: `https://api.openai.com/v1`)
- `api_key` – authentication token sent as `Authorization: Bearer …`
- `model` – model identifier recognized by the remote server
- `extra_body` – additional JSON payload fields
- `timeout` – request timeout in seconds

### Base Wrapper Implementation

[`src/speech_to_speech/LLM/base_openai_compatible_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/base_openai_compatible_language_model.py) provides `OpenAICompatibleLanguageModel`, a thin wrapper around the official `openai` Python client. It:

- Instantiates `OpenAI` with custom `api_key` and `base_url`
- Streams responses via `client.chat.completions.create`
- Normalizes chunks into internal `ToolCall` and `Usage` structures

### Runtime Propagation

[`src/speech_to_speech/api/openai_realtime/runtime_config.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/runtime_config.py) ensures tool calls, function definitions, and usage metrics flow correctly through the realtime service regardless of which LLM backend is active.

### Pipeline Integration

[`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) constructs the `SpeechToSpeechPipeline`. It parses arguments, queries the backend registry, and inserts the LLM handler into the message-flow graph.

## CLI Usage

Pass the OpenAI-compatible endpoint via command-line flags:

```bash
speech-to-speech \
  --llm-backend responses_api \
  --llm-api-base https://my-ollama-server.com/v1 \
  --llm-api-key dummy-token \
  --llm-model llama2:7b-chat \
  --tts-backend pocket_tts \
  --stt-backend faster_whisper

```

The CLI forwards `--llm-*` arguments to `ResponsesAPILanguageModelArguments`, which constructs the client internally.

## Programmatic Configuration

Create the pipeline directly in Python for finer control:

```python
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline
from speech_to_speech.arguments_classes.responses_api_language_model_arguments import (
    ResponsesAPILanguageModelArguments,
)

# Configure the OpenAI-compatible LLM backend

llm_args = ResponsesAPILanguageModelArguments(
    api_base="https://my-openai-compatible.com/v1",
    api_key="sk-my-secret-key",
    model="gpt-4o-mini",
)

pipeline = SpeechToSpeechPipeline(
    language_model_args=llm_args,
    tts_args=...,   # e.g., PocketTTSArguments(...)

    stt_args=...,   # e.g., FasterWhisperSTTArguments(...)

)

# Execute on microphone stream or async audio iterator

await pipeline.run()

```

`SpeechToSpeechPipeline` auto-registers the `responses_api` backend because `ResponsesAPILanguageModelArguments` sets `backend_name="responses_api"` and inherits from `LanguageModelBaseArguments`.

## Direct Wrapper Instantiation

For custom logic or testing, instantiate the wrapper directly:

```python
from speech_to_speech.LLM.base_openai_compatible_language_model import OpenAICompatibleLanguageModel

# Create client pointing at custom endpoint

lm = OpenAICompatibleLanguageModel(
    api_key="my-key",
    base_url="https://custom-openai.local/api",
    model="phi-3-mini-4k-instruct",
)

# Generate single response

response = await lm.chat([{"role": "user", "content": "Summarise this text"}])
print(response.text)

```

## Key Source Files

| File | Purpose |
|------|---------|
| [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) | Maps backend identifiers to factory functions |
| [`src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py) | CLI and programmatic arguments for OpenAI-compatible LLMs |
| [`src/speech_to_speech/LLM/base_openai_compatible_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/base_openai_compatible_language_model.py) | `openai` client wrapper with streaming normalization |
| [`src/speech_to_speech/api/openai_realtime/runtime_config.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/runtime_config.py) | Propagates LLM config into realtime service |
| [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) | Orchestrates STT, LLM, and TTS components |

## Supported OpenAI-Compatible Servers

Any server implementing the OpenAI HTTP API contract works without code changes:

- **OpenAI** – `https://api.openai.com/v1`
- **Azure OpenAI** – `https://{resource}.openai.azure.com/openai/deployments/{deployment}`
- **Ollama** – `http://localhost:11434/v1` (with `OLLAMA_HOST` configured)
- **vLLM** – self-hosted inference with OpenAI-compatible endpoint
- **text-generation-inference** – Hugging Face's serving stack

## Summary

- Use `--llm-backend responses_api` to enable OpenAI-compatible API mode
- Set `--llm-api-base` to your server's base URL (defaults to official OpenAI)
- Configure `--llm-api-key` and `--llm-model` according to your provider
- The wrapper in [`base_openai_compatible_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/base_openai_compatible_language_model.py) handles streaming and normalization
- Both CLI and programmatic APIs support full customization of the LLM backend

## Frequently Asked Questions

### Can I use a local model without internet access?

Yes. Run an OpenAI-compatible server locally with **Ollama**, **vLLM**, or **TGI**, then point `--llm-api-base` at `http://localhost:11434/v1` or your custom port. The speech-to-speech pipeline treats local and remote servers identically.

### Does tool calling work with non-OpenAI providers?

Tool calling depends on the remote server's implementation. The pipeline in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) submits tool definitions and parses `ToolCall` responses from the normalized stream. If your server supports the OpenAI tools format, functionality is transparent.

### How do I debug connection issues?

Set the `OPENAI_LOG` environment variable to `debug` or add `extra_body={"debug": True}` in `ResponsesAPILanguageModelArguments`. The [`base_openai_compatible_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/base_openai_compatible_language_model.py) wrapper surfaces HTTP errors and malformed JSON responses directly.