How to Configure the Responses API for LLM Backend in huggingface/speech-to-speech

Configure the Responses API backend by setting ResponsesApiLanguageModelHandlerArguments with your model name, API key, and endpoint URL, then register it via backend_registry.select_backend() or use CLI flags prefixed with --responses_api_.

The Responses API is a built-in LLM backend in the huggingface/speech-to-speech repository that connects to OpenAI-compatible /v1/responses endpoints. It enables audio-aware conversations by routing text and audio through a unified API interface. This guide explains how to configure this backend through both command-line options and programmatic Python code.

Architecture Overview

The Responses API backend operates through four interconnected layers in the codebase:

Layer	Purpose	Key File
Argument definition	Defines CLI and programmatic configuration options	src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py
Backend registry	Maps the logical name responses-api to the handler factory	src/speech_to_speech/backend_registry.py
Handler implementation	Executes HTTP calls to the Responses API endpoint	src/speech_to_speech/LLM/responses_api_language_model.py
Chat serialization	Converts conversation history to Responses API format	src/speech_to_speech/LLM/chat.py

Configuration Parameters

All Responses API settings are defined in the ResponsesApiLanguageModelHandlerArguments dataclass. These parameters use the config_prefix="responses_api" namespace to avoid collisions with other backends.

Core Parameters

from speech_to_speech.arguments_classes.responses_api_language_model_arguments import (
    ResponsesApiLanguageModelHandlerArguments,
)

args = ResponsesApiLanguageModelHandlerArguments(
    model_name="gpt-4o-mini",                    # Model identifier

    responses_api_api_key=None,                   # API authentication key

    responses_api_base_url=None,                  # Custom endpoint URL

    responses_api_stream=True,                    # Enable streaming responses

)

Audio-Specific Parameters

For audio-enabled conversations, configure these additional settings:

args = ResponsesApiLanguageModelHandlerArguments(
    responses_api_audio_max_tokens=256,           # Max tokens for audio responses

    responses_api_audio_temperature=0.0,          # Sampling temperature

    responses_api_audio_content_type="input_audio",  # "input_audio" or "audio_url"

    responses_api_audio_history_turns=1,          # Audio turns to retain in context

)

The responses_api_audio_content_type parameter controls how audio is transmitted:

• ** "input_audio" ** – Audio data embedded directly in the request
• ** "audio_url" ** – Audio referenced by URL

The responses_api_audio_history_turns parameter limits how many previous audio exchanges remain in the conversation buffer. Set to 0 to replace all completed audio turns with placeholder text.

CLI Configuration

Use the --llm responses-api flag to select this backend, then prefix all related options with --responses_api_:

speech-to-speech \
    --llm responses-api \
    --responses_api_model_name "gpt-4o-mini" \
    --responses_api_api_key "$HF_API_KEY" \
    --responses_api_base_url "https://api.mycompany.com/v1/responses" \
    --responses_api_stream true \
    --responses_api_audio_max_tokens 512 \
    --responses_api_audio_temperature 0.2 \
    --responses_api_audio_content_type audio_url \
    --responses_api_audio_history_turns 2

The CLI parser automatically maps these flags to the ResponsesApiLanguageModelHandlerArguments dataclass due to the config_prefix="responses_api" registration in backend_registry.py.

Programmatic Configuration

For Python applications, instantiate the handler directly through the backend registry:

from speech_to_speech.arguments_classes.responses_api_language_model_arguments import (
    ResponsesApiLanguageModelHandlerArguments,
)
from speech_to_speech.backend_registry import select_backend, LLM_BACKENDS, HandlerContext
from speech_to_speech.utils.cancel_scope import CancelScope
from speech_to_speech.utils.speculative_turn_tracker import SpeculativeTurnTracker
import threading
import queue

# Step 1: Configure arguments

args = ResponsesApiLanguageModelHandlerArguments(
    model_name="gpt-4o-mini",
    responses_api_api_key="hf_secret",
    responses_api_base_url="https://api.mycompany.com/v1/responses",
    responses_api_stream=True,
    responses_api_audio_max_tokens=512,
    responses_api_audio_temperature=0.2,
    responses_api_audio_content_type="audio_url",
    responses_api_audio_history_turns=2,
)

# Step 2: Normalize and select backend

selection = select_backend(LLM_BACKENDS, "responses-api", args)

# Step 3: Create handler context with required runtime objects

context = HandlerContext(
    stop_event=threading.Event(),
    queue_in=queue.Queue(),
    queue_out=queue.Queue(),
    text_output_queue=queue.Queue(),
    should_listen=threading.Event(),
    cancel_scope=CancelScope(),
    speculative_turns=SpeculativeTurnTracker(),
    pipeline_index=0,
    sample_rate=16000,
    enable_live_transcription=False,
    live_transcription_update_interval=0.5,
)

# Step 4: Instantiate handler

handler = selection.spec.create_handler(context, selection.config)

# Step 5: Verify connectivity

handler.warmup()

The selection.spec.create_handler() factory in backend_registry.py imports ResponsesApiModelHandler from responses_api_language_model.py and injects both the parsed configuration and runtime context.

Making Requests with the Configured Backend

Once configured, the handler provides a standard interface for conversation requests. The internal flow converts your Chat object to Responses API format via Chat.to_responses_api_chat():

from speech_to_speech.LLM.chat import Chat

# Initialize conversation with system prompt

chat = Chat(size=30)
chat.init_chat("You are a helpful assistant.")

# Add user message with audio

audio_b64 = "UklGRiQAAABXQVZFZm10IBAAAAABAAEAQB8AAIA+AAACABAAZGF0YQAAAAA="
chat.add_item({
    "type": "message",
    "role": "user",
    "content": [{"type": "input_audio", "audio": audio_b64}]
})

# Serialize and send request

api_input = handler._serialize(chat)  # Calls Chat.to_responses_api_chat()

optional_kwargs = handler._build_optional_kwargs(req_tools=None, req_tool_choice=None)
api_response = handler._request(api_input, optional_kwargs)

# Process streaming events

for event in handler._iter_stream_events(api_response):
    if event.type == "TextDelta":
        print(event.delta, end="")
    elif event.type == "AssistantMessage":
        print(f"\n[Complete response: {event.content}]")

The ResponsesApiModelHandler._request() method in responses_api_language_model.py constructs the actual HTTP call to client.responses.create(), while _iter_stream_events() translates Responses API events (ResponseTextDeltaEvent, ResponseOutputMessage, etc.) into the internal provider event system.

How Chat Serialization Works

The Chat.to_responses_api_chat() method in src/speech_to_speech/LLM/chat.py performs the critical conversion between internal conversation state and Responses API payload format:

• System messages → ResponseMessage with role "system"
• User messages → ResponseInputTextParam objects (audio becomes "[User audio input]" placeholder or URL reference)
• Assistant messages → ResponseOutputMessageParam containing ResponseOutputTextParam
• Function calls → ResponseFunctionToolCallParam and FunctionCallOutput

This serialization enables seamless audio-aware interactions through the standard Responses API endpoint.

Summary

• Use ResponsesApiLanguageModelHandlerArguments to define all configuration parameters for the Responses API backend
• Prefix CLI flags with --responses_api_ when using command-line invocation
• Call select_backend(LLM_BACKENDS, "responses-api", args) for programmatic handler creation
• Invoke handler.warmup() to verify endpoint connectivity before production use
• Set responses_api_audio_history_turns=0 to disable audio context retention for endpoints that don't support multiple audio chunks
• Reference responses_api_language_model.py for the complete implementation of HTTP calls and streaming event handling

Frequently Asked Questions

What is the difference between the Responses API and Chat Completions backends?

Both backends implement the same LLMHandler interface and can be swapped without code changes. The Responses API backend (responses-api) targets the /v1/responses endpoint with native audio support, while the Chat Completions backend (chat-completions) uses the traditional /v1/chat/completions endpoint. Configuration parameters differ only by prefix—responses_api_* versus chat_completions_*—as defined in their respective argument classes.

How do I configure a custom endpoint URL for the Responses API?

Set the responses_api_base_url parameter in ResponsesApiLanguageModelHandlerArguments or use --responses_api_base_url on the CLI. The handler passes this directly to the OpenAI client initialization in responses_api_language_model.py. Ensure your endpoint is fully compatible with the /v1/responses specification including streaming response formats.

Why would I set responses_api_audio_history_turns to zero?

Setting responses_api_audio_history_turns=0 replaces all previous audio turns with placeholder text "[User audio input]" in the serialized chat. This is necessary when your remote model provider doesn't support multiple audio chunks in context or when you want to minimize token usage. The current turn's audio data is always transmitted regardless of this setting.

Can I use the same configuration for multiple pipeline instances?

Yes. Create one ResponsesApiLanguageModelHandlerArguments instance and pass it to select_backend() for each pipeline. Each call to selection.spec.create_handler() produces an independent handler instance with its own HandlerContext, while sharing the same underlying configuration. This pattern supports multi-tenant deployments without duplicating configuration code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}