# Setting Up LLM Proxy for Concurrent Background Tasks in Voice Conversations: A Complete Guide

> Set up an LLM proxy for concurrent background tasks in voice conversations. This guide explains how to use the OpenAI compatible passthrough for parallel audio processing and external LLM requests.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-09

---

**The LLM Proxy is an OpenAI-compatible passthrough that mounts `/v1/chat/completions` and `/v1/responses` endpoints outside the main speech pipeline, allowing the server to forward requests to external LLM providers while continuing to process audio input in parallel.**

The huggingface/speech-to-speech repository implements a real-time voice conversation system where blocking operations can disrupt conversational flow. Setting up an LLM proxy for concurrent background tasks in voice conversations enables the server to offload language model inference to external APIs while maintaining continuous speech processing through its VAD → STT → TTS pipeline.

## Why the Voice Pipeline Requires an LLM Proxy

Voice interactions follow a strict pipeline: **Voice Activity Detection (VAD)** → **Speech-to-Text (STT)** → **Language Model (LLM)** → **Text-to-Speech (TTS)**. When the pipeline invokes the LLM directly, it blocks on the completion result, preventing the server from listening to new microphone input or processing subsequent conversation turns.

By mounting a proxy endpoint that forwards requests **outside** the pipeline’s unit queues, the system achieves true concurrency. The speech engine continues accepting audio chunks and performing VAD detection while the proxy handles upstream LLM communication independently.

## Architecture of the LLM Proxy System

The proxy architecture separates external API communication from the core audio processing loop through three key components:

### Mounting and Configuration

The `mount_llm_proxy` function in [`src/speech_to_speech/api/openai_realtime/llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/llm_proxy.py) (lines 38-77) inspects the configuration and registers either:

- A **passthrough handler** for supported backends (`chat-completions` or `responses-api`)
- A **stub handler** that returns HTTP 501 when the proxy is disabled or misconfigured

### Request Forwarding Flow

The proxy receives JSON requests at `/v1/chat/completions` or `/v1/responses`, injects the configured model name, and forces `include_usage=True` for streaming responses. It forwards requests to the upstream provider using the API key specified in `LLMProxyConfig.upstream_api_key`, setting the `Authorization: Bearer` header automatically.

### Response Handling Strategies

For **non-streaming** responses, the proxy returns the upstream JSON verbatim after extracting `usage` fields to update internal counters. For **streaming** responses, it relays bytes via `StreamingResponse` while parsing Server-Sent Events (SSE) in the background to capture token usage without altering the client’s view of the stream.

### Usage Accounting

The `LLMProxyUsage` class tracks requests, response codes, and token counts. These counters integrate into the server’s `/v1/usage` endpoint under the `llm_proxy` key, as implemented in [`src/speech_to_speech/api/openai_realtime/websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py) (lines 537-539).

## Enabling the LLM Proxy Configuration

Activating the proxy requires specific CLI arguments and backend selection validated against the registry:

### Step 1: Enable the Proxy Flag

Add `--enable_llm_proxy` to your launch command. This sets `ModuleArguments.enable_llm_proxy = True` as defined in [`src/speech_to_speech/arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/module_arguments.py) (lines 69-78).

### Step 2: Select a Compatible Backend

Choose `--llm_backend chat-completions` or `--llm_backend responses-api`. The `BackendCapabilities.supports_llm_proxy` property in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) (lines 393-405) validates compatibility. Attempting to enable the proxy with an incompatible backend raises `ValueError` during pipeline initialization in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) (lines 113-118).

### Step 3: Configure Upstream Credentials

Provide `--llm_proxy_upstream_api_key` or rely on automatic extraction from the backend specification. The `build_llm_proxy_config` function in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) (lines 106-127) constructs the final `LLMProxyConfig` object containing `upstream_base_url`, `model_name`, and connection timeouts.

### Step 4: Initialize the Server

The `RealtimeServer` class in [`src/speech_to_speech/api/openai_realtime/server.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/server.py) (lines 29-41) calls `mount_llm_proxy` during FastAPI application setup, exposing the proxy endpoints alongside the WebSocket voice interface.

## Concurrent Processing and Pipeline Isolation

The LLM proxy operates **outside** the pipeline’s queue network, meaning it does not consume `AudioInItem`, `AudioOutItem`, or `LMOutItem` queue slots. This isolation ensures that:

- New audio chunks continue processing through VAD while LLM requests are in flight
- Multiple conversation turns can initiate before previous LLM responses complete
- Token usage from upstream providers is accounted for via `LLMProxyUsage` without introducing latency into the speech synthesis path

## Error Handling and Resilience

The proxy implements specific HTTP status codes for failure modes:

- **501 Not Implemented**: Returned by `_error_response` when the proxy is disabled or the backend lacks `supports_llm_proxy=True`
- **502 Bad Gateway**: Returned by `_upstream_unreachable` when the external LLM provider connection fails
- **Content preservation**: Successful responses pass through unmodified, ensuring full compatibility with OpenAI-compatible clients regardless of the upstream provider

## Practical Implementation Examples

### Launching with CLI Arguments

```bash
python -m speech_to_speech serve \
    --enable_llm_proxy \
    --llm_backend chat-completions \
    --llm_proxy_connect_timeout_s 15.0

```

### Building Configuration Programmatically

```python
from speech_to_speech.s2s_pipeline import build_llm_proxy_config
from speech_to_speech.arguments_classes.module_arguments import ModuleArguments
from speech_to_speech.backend_registry import BackendSelection

module_args = ModuleArguments(enable_llm_proxy=True)

# Assume llm_backend is a BackendSelection instance

proxy_cfg = build_llm_proxy_config(module_args, llm_backend)
print(proxy_cfg.dict())

# Output includes: enabled, upstream_base_url, model_name, connect_timeout_s

```

### Mounting in a Custom FastAPI Application

```python
from fastapi import FastAPI
from speech_to_speech.api.openai_realtime.llm_proxy import LLMProxyConfig, mount_llm_proxy

app = FastAPI()
cfg = LLMProxyConfig(
    enabled=True,
    llm_backend="chat-completions",
    upstream_api_key="sk-my-key",
    model_name="gpt-4o",
)
proxy_usage = mount_llm_proxy(app, cfg)

# Inspect accumulated counters

print(f"Requests: {proxy_usage.requests}, Tokens: {proxy_usage.output_tokens}")

```

### Querying Usage Statistics

```bash
curl http://localhost:8000/v1/usage

# Response includes:

# "llm_proxy": {

#   "requests": 3,

#   "responses_2xx": 3,

#   "input_tokens": 124,

#   "output_tokens": 342

# }

```

## Summary

- The **LLM Proxy** prevents blocking in the voice pipeline by mounting OpenAI-compatible endpoints outside the main processing queues in [`llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/llm_proxy.py)
- **Concurrent background tasks** are achieved because the proxy does not consume `AudioInItem` or `LMOutItem` queue slots, allowing continuous audio processing
- **Configuration** requires `--enable_llm_proxy` with a compatible backend (`chat-completions` or `responses-api`) validated through [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py)
- **Token accounting** occurs via `LLMProxyUsage` counters integrated into the `/v1/usage` endpoint without impacting streaming latency
- **Error handling** distinguishes between configuration errors (501), upstream failures (502), and successful passthrough responses

## Frequently Asked Questions

### Which LLM backends support proxying in the speech-to-speech server?

Only backends with `supports_llm_proxy=True` in their `BackendCapabilities` declaration can operate with the proxy enabled. According to [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) (lines 393-405), the `chat-completions` and `responses-api` backends support proxying, while others will raise a `ValueError` during pipeline initialization if the proxy is enabled.

### Does the LLM proxy modify the content of streaming responses?

No. The proxy relays streaming responses byte-by-byte via `StreamingResponse` without altering the content. While forwarding, it asynchronously parses SSE events to extract usage statistics, but the client receives the exact bytes sent by the upstream provider, ensuring compatibility with any OpenAI-compatible client implementation.

### How does the server track token usage when using an external LLM provider?

The `LLMProxyUsage` class accumulates counters for requests, response status codes, input tokens, and output tokens. For streaming responses, the proxy watches for SSE chunks containing usage data. These counters are exposed through the `/v1/usage` endpoint under the `llm_proxy` key, as implemented in [`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_router.py) (lines 537-539), enabling accurate billing and monitoring without proxying overhead.

### Can the LLM proxy operate independently of the speech pipeline?

Yes. While designed for the speech-to-speech architecture, the `mount_llm_proxy` function can attach to any FastAPI application. The proxy logic in [`llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/llm_proxy.py) is self-contained and only requires an `LLMProxyConfig` instance to function, making it suitable for standalone OpenAI-compatible proxy deployments or integration into other real-time systems.