# How to Configure the Responses API for LLM Backend in huggingface/speech-to-speech

> Easily configure the Responses API for your LLM backend in huggingface/speech-to-speech. Learn to set API keys, model names, and endpoint URLs for seamless integration.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-11

---

**Configure the Responses API backend by setting `ResponsesApiLanguageModelHandlerArguments` with your model name, API key, and endpoint URL, then register it via `backend_registry.select_backend()` or use CLI flags prefixed with `--responses_api_`.**

The **Responses API** is a built-in LLM backend in the `huggingface/speech-to-speech` repository that connects to OpenAI-compatible `/v1/responses` endpoints. It enables audio-aware conversations by routing text and audio through a unified API interface. This guide explains how to configure this backend through both command-line options and programmatic Python code.

## Architecture Overview

The Responses API backend operates through four interconnected layers in the codebase:

| Layer | Purpose | Key File |
|-------|---------|----------|
| **Argument definition** | Defines CLI and programmatic configuration options | [`src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py) |
| **Backend registry** | Maps the logical name `responses-api` to the handler factory | [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) |
| **Handler implementation** | Executes HTTP calls to the Responses API endpoint | [`src/speech_to_speech/LLM/responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/responses_api_language_model.py) |
| **Chat serialization** | Converts conversation history to Responses API format | [`src/speech_to_speech/LLM/chat.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat.py) |

## Configuration Parameters

All Responses API settings are defined in the `ResponsesApiLanguageModelHandlerArguments` dataclass. These parameters use the `config_prefix="responses_api"` namespace to avoid collisions with other backends.

### Core Parameters

```python
from speech_to_speech.arguments_classes.responses_api_language_model_arguments import (
    ResponsesApiLanguageModelHandlerArguments,
)

args = ResponsesApiLanguageModelHandlerArguments(
    model_name="gpt-4o-mini",                    # Model identifier

    responses_api_api_key=None,                   # API authentication key

    responses_api_base_url=None,                  # Custom endpoint URL

    responses_api_stream=True,                    # Enable streaming responses

)

```

### Audio-Specific Parameters

For audio-enabled conversations, configure these additional settings:

```python
args = ResponsesApiLanguageModelHandlerArguments(
    responses_api_audio_max_tokens=256,           # Max tokens for audio responses

    responses_api_audio_temperature=0.0,          # Sampling temperature

    responses_api_audio_content_type="input_audio",  # "input_audio" or "audio_url"

    responses_api_audio_history_turns=1,          # Audio turns to retain in context

)

```

The `responses_api_audio_content_type` parameter controls how audio is transmitted:
- ** `"input_audio"` ** – Audio data embedded directly in the request
- ** `"audio_url"` ** – Audio referenced by URL

The `responses_api_audio_history_turns` parameter limits how many previous audio exchanges remain in the conversation buffer. Set to `0` to replace all completed audio turns with placeholder text.

## CLI Configuration

Use the `--llm responses-api` flag to select this backend, then prefix all related options with `--responses_api_`:

```bash
speech-to-speech \
    --llm responses-api \
    --responses_api_model_name "gpt-4o-mini" \
    --responses_api_api_key "$HF_API_KEY" \
    --responses_api_base_url "https://api.mycompany.com/v1/responses" \
    --responses_api_stream true \
    --responses_api_audio_max_tokens 512 \
    --responses_api_audio_temperature 0.2 \
    --responses_api_audio_content_type audio_url \
    --responses_api_audio_history_turns 2

```

The CLI parser automatically maps these flags to the `ResponsesApiLanguageModelHandlerArguments` dataclass due to the `config_prefix="responses_api"` registration in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py).

## Programmatic Configuration

For Python applications, instantiate the handler directly through the backend registry:

```python
from speech_to_speech.arguments_classes.responses_api_language_model_arguments import (
    ResponsesApiLanguageModelHandlerArguments,
)
from speech_to_speech.backend_registry import select_backend, LLM_BACKENDS, HandlerContext
from speech_to_speech.utils.cancel_scope import CancelScope
from speech_to_speech.utils.speculative_turn_tracker import SpeculativeTurnTracker
import threading
import queue

# Step 1: Configure arguments

args = ResponsesApiLanguageModelHandlerArguments(
    model_name="gpt-4o-mini",
    responses_api_api_key="hf_secret",
    responses_api_base_url="https://api.mycompany.com/v1/responses",
    responses_api_stream=True,
    responses_api_audio_max_tokens=512,
    responses_api_audio_temperature=0.2,
    responses_api_audio_content_type="audio_url",
    responses_api_audio_history_turns=2,
)

# Step 2: Normalize and select backend

selection = select_backend(LLM_BACKENDS, "responses-api", args)

# Step 3: Create handler context with required runtime objects

context = HandlerContext(
    stop_event=threading.Event(),
    queue_in=queue.Queue(),
    queue_out=queue.Queue(),
    text_output_queue=queue.Queue(),
    should_listen=threading.Event(),
    cancel_scope=CancelScope(),
    speculative_turns=SpeculativeTurnTracker(),
    pipeline_index=0,
    sample_rate=16000,
    enable_live_transcription=False,
    live_transcription_update_interval=0.5,
)

# Step 4: Instantiate handler

handler = selection.spec.create_handler(context, selection.config)

# Step 5: Verify connectivity

handler.warmup()

```

The `selection.spec.create_handler()` factory in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py) imports `ResponsesApiModelHandler` from [`responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/responses_api_language_model.py) and injects both the parsed configuration and runtime context.

## Making Requests with the Configured Backend

Once configured, the handler provides a standard interface for conversation requests. The internal flow converts your `Chat` object to Responses API format via `Chat.to_responses_api_chat()`:

```python
from speech_to_speech.LLM.chat import Chat

# Initialize conversation with system prompt

chat = Chat(size=30)
chat.init_chat("You are a helpful assistant.")

# Add user message with audio

audio_b64 = "UklGRiQAAABXQVZFZm10IBAAAAABAAEAQB8AAIA+AAACABAAZGF0YQAAAAA="
chat.add_item({
    "type": "message",
    "role": "user",
    "content": [{"type": "input_audio", "audio": audio_b64}]
})

# Serialize and send request

api_input = handler._serialize(chat)  # Calls Chat.to_responses_api_chat()

optional_kwargs = handler._build_optional_kwargs(req_tools=None, req_tool_choice=None)
api_response = handler._request(api_input, optional_kwargs)

# Process streaming events

for event in handler._iter_stream_events(api_response):
    if event.type == "TextDelta":
        print(event.delta, end="")
    elif event.type == "AssistantMessage":
        print(f"\n[Complete response: {event.content}]")

```

The `ResponsesApiModelHandler._request()` method in [`responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/responses_api_language_model.py) constructs the actual HTTP call to `client.responses.create()`, while `_iter_stream_events()` translates Responses API events (`ResponseTextDeltaEvent`, `ResponseOutputMessage`, etc.) into the internal provider event system.

## How Chat Serialization Works

The `Chat.to_responses_api_chat()` method in [`src/speech_to_speech/LLM/chat.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat.py) performs the critical conversion between internal conversation state and Responses API payload format:

- **System messages** → `ResponseMessage` with role `"system"`
- **User messages** → `ResponseInputTextParam` objects (audio becomes `"[User audio input]"` placeholder or URL reference)
- **Assistant messages** → `ResponseOutputMessageParam` containing `ResponseOutputTextParam`
- **Function calls** → `ResponseFunctionToolCallParam` and `FunctionCallOutput`

This serialization enables seamless audio-aware interactions through the standard Responses API endpoint.

## Summary

- **Use `ResponsesApiLanguageModelHandlerArguments`** to define all configuration parameters for the Responses API backend
- **Prefix CLI flags with `--responses_api_`** when using command-line invocation
- **Call `select_backend(LLM_BACKENDS, "responses-api", args)`** for programmatic handler creation
- **Invoke `handler.warmup()`** to verify endpoint connectivity before production use
- **Set `responses_api_audio_history_turns=0`** to disable audio context retention for endpoints that don't support multiple audio chunks
- **Reference [`responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/responses_api_language_model.py)** for the complete implementation of HTTP calls and streaming event handling

## Frequently Asked Questions

### What is the difference between the Responses API and Chat Completions backends?

Both backends implement the same `LLMHandler` interface and can be swapped without code changes. The **Responses API** backend (`responses-api`) targets the `/v1/responses` endpoint with native audio support, while the **Chat Completions** backend (`chat-completions`) uses the traditional `/v1/chat/completions` endpoint. Configuration parameters differ only by prefix—`responses_api_*` versus `chat_completions_*`—as defined in their respective argument classes.

### How do I configure a custom endpoint URL for the Responses API?

Set the `responses_api_base_url` parameter in `ResponsesApiLanguageModelHandlerArguments` or use `--responses_api_base_url` on the CLI. The handler passes this directly to the OpenAI client initialization in [`responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/responses_api_language_model.py). Ensure your endpoint is fully compatible with the `/v1/responses` specification including streaming response formats.

### Why would I set `responses_api_audio_history_turns` to zero?

Setting `responses_api_audio_history_turns=0` replaces all previous audio turns with placeholder text `"[User audio input]"` in the serialized chat. This is necessary when your remote model provider doesn't support multiple audio chunks in context or when you want to minimize token usage. The current turn's audio data is always transmitted regardless of this setting.

### Can I use the same configuration for multiple pipeline instances?

Yes. Create one `ResponsesApiLanguageModelHandlerArguments` instance and pass it to `select_backend()` for each pipeline. Each call to `selection.spec.create_handler()` produces an independent handler instance with its own `HandlerContext`, while sharing the same underlying configuration. This pattern supports multi-tenant deployments without duplicating configuration code.