How to Use an OpenAI-Compatible API for the LLM Backend in speech-to-speech

Use the responses_api backend with --llm-api-base to point the huggingface/speech-to-speech pipeline at any OpenAI-compatible endpoint such as Ollama, Azure OpenAI, or a self-hosted server.

The speech-to-speech library decouples the LLM component through a pluggable backend system. This design lets you swap the default language model for any server implementing the OpenAI HTTP API specification without modifying core pipeline code.

Architecture Overview

The OpenAI-compatible integration spans five key components:

Backend Registry

src/speech_to_speech/backend_registry.py maintains a mapping from backend names to factory functions. When you specify --llm-backend responses_api, the registry instantiates the appropriate handler.

Argument Configuration

src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py declares the parameters needed to connect to an OpenAI-compatible server:

• api_base – base URL of the server (default: https://api.openai.com/v1)
• api_key – authentication token sent as Authorization: Bearer …
• model – model identifier recognized by the remote server
• extra_body – additional JSON payload fields
• timeout – request timeout in seconds

Base Wrapper Implementation

src/speech_to_speech/LLM/base_openai_compatible_language_model.py provides OpenAICompatibleLanguageModel, a thin wrapper around the official openai Python client. It:

• Instantiates OpenAI with custom api_key and base_url
• Streams responses via client.chat.completions.create
• Normalizes chunks into internal ToolCall and Usage structures

Runtime Propagation

src/speech_to_speech/api/openai_realtime/runtime_config.py ensures tool calls, function definitions, and usage metrics flow correctly through the realtime service regardless of which LLM backend is active.

Pipeline Integration

src/speech_to_speech/s2s_pipeline.py constructs the SpeechToSpeechPipeline. It parses arguments, queries the backend registry, and inserts the LLM handler into the message-flow graph.

CLI Usage

Pass the OpenAI-compatible endpoint via command-line flags:

speech-to-speech \
  --llm-backend responses_api \
  --llm-api-base https://my-ollama-server.com/v1 \
  --llm-api-key dummy-token \
  --llm-model llama2:7b-chat \
  --tts-backend pocket_tts \
  --stt-backend faster_whisper

The CLI forwards --llm-* arguments to ResponsesAPILanguageModelArguments, which constructs the client internally.

Programmatic Configuration

Create the pipeline directly in Python for finer control:

from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline
from speech_to_speech.arguments_classes.responses_api_language_model_arguments import (
    ResponsesAPILanguageModelArguments,
)

# Configure the OpenAI-compatible LLM backend

llm_args = ResponsesAPILanguageModelArguments(
    api_base="https://my-openai-compatible.com/v1",
    api_key="sk-my-secret-key",
    model="gpt-4o-mini",
)

pipeline = SpeechToSpeechPipeline(
    language_model_args=llm_args,
    tts_args=...,   # e.g., PocketTTSArguments(...)

    stt_args=...,   # e.g., FasterWhisperSTTArguments(...)

)

# Execute on microphone stream or async audio iterator

await pipeline.run()

SpeechToSpeechPipeline auto-registers the responses_api backend because ResponsesAPILanguageModelArguments sets backend_name="responses_api" and inherits from LanguageModelBaseArguments.

Direct Wrapper Instantiation

For custom logic or testing, instantiate the wrapper directly:

from speech_to_speech.LLM.base_openai_compatible_language_model import OpenAICompatibleLanguageModel

# Create client pointing at custom endpoint

lm = OpenAICompatibleLanguageModel(
    api_key="my-key",
    base_url="https://custom-openai.local/api",
    model="phi-3-mini-4k-instruct",
)

# Generate single response

response = await lm.chat([{"role": "user", "content": "Summarise this text"}])
print(response.text)

Key Source Files

File	Purpose
src/speech_to_speech/backend_registry.py	Maps backend identifiers to factory functions
src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py	CLI and programmatic arguments for OpenAI-compatible LLMs
src/speech_to_speech/LLM/base_openai_compatible_language_model.py	openai client wrapper with streaming normalization
src/speech_to_speech/api/openai_realtime/runtime_config.py	Propagates LLM config into realtime service
src/speech_to_speech/s2s_pipeline.py	Orchestrates STT, LLM, and TTS components

Supported OpenAI-Compatible Servers

Any server implementing the OpenAI HTTP API contract works without code changes:

• OpenAI – https://api.openai.com/v1
• Azure OpenAI – https://{resource}.openai.azure.com/openai/deployments/{deployment}
• Ollama – http://localhost:11434/v1 (with OLLAMA_HOST configured)
• vLLM – self-hosted inference with OpenAI-compatible endpoint
• text-generation-inference – Hugging Face's serving stack

Summary

• Use --llm-backend responses_api to enable OpenAI-compatible API mode
• Set --llm-api-base to your server's base URL (defaults to official OpenAI)
• Configure --llm-api-key and --llm-model according to your provider
• The wrapper in base_openai_compatible_language_model.py handles streaming and normalization
• Both CLI and programmatic APIs support full customization of the LLM backend

Frequently Asked Questions

Can I use a local model without internet access?

Yes. Run an OpenAI-compatible server locally with Ollama, vLLM, or TGI, then point --llm-api-base at http://localhost:11434/v1 or your custom port. The speech-to-speech pipeline treats local and remote servers identically.

Does tool calling work with non-OpenAI providers?

Tool calling depends on the remote server's implementation. The pipeline in s2s_pipeline.py submits tool definitions and parses ToolCall responses from the normalized stream. If your server supports the OpenAI tools format, functionality is transparent.

How do I debug connection issues?

Set the OPENAI_LOG environment variable to debug or add extra_body={"debug": True} in ResponsesAPILanguageModelArguments. The base_openai_compatible_language_model.py wrapper surfaces HTTP errors and malformed JSON responses directly.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}