Setting Up LLM Proxy for Concurrent Background Tasks in Voice Conversations: A Complete Guide

The LLM Proxy is an OpenAI-compatible passthrough that mounts /v1/chat/completions and /v1/responses endpoints outside the main speech pipeline, allowing the server to forward requests to external LLM providers while continuing to process audio input in parallel.

The huggingface/speech-to-speech repository implements a real-time voice conversation system where blocking operations can disrupt conversational flow. Setting up an LLM proxy for concurrent background tasks in voice conversations enables the server to offload language model inference to external APIs while maintaining continuous speech processing through its VAD → STT → TTS pipeline.

Why the Voice Pipeline Requires an LLM Proxy

Voice interactions follow a strict pipeline: Voice Activity Detection (VAD) → Speech-to-Text (STT) → Language Model (LLM) → Text-to-Speech (TTS). When the pipeline invokes the LLM directly, it blocks on the completion result, preventing the server from listening to new microphone input or processing subsequent conversation turns.

By mounting a proxy endpoint that forwards requests outside the pipeline’s unit queues, the system achieves true concurrency. The speech engine continues accepting audio chunks and performing VAD detection while the proxy handles upstream LLM communication independently.

Architecture of the LLM Proxy System

The proxy architecture separates external API communication from the core audio processing loop through three key components:

Mounting and Configuration

The mount_llm_proxy function in src/speech_to_speech/api/openai_realtime/llm_proxy.py (lines 38-77) inspects the configuration and registers either:

• A passthrough handler for supported backends (chat-completions or responses-api)
• A stub handler that returns HTTP 501 when the proxy is disabled or misconfigured

Request Forwarding Flow

The proxy receives JSON requests at /v1/chat/completions or /v1/responses, injects the configured model name, and forces include_usage=True for streaming responses. It forwards requests to the upstream provider using the API key specified in LLMProxyConfig.upstream_api_key, setting the Authorization: Bearer header automatically.

Response Handling Strategies

For non-streaming responses, the proxy returns the upstream JSON verbatim after extracting usage fields to update internal counters. For streaming responses, it relays bytes via StreamingResponse while parsing Server-Sent Events (SSE) in the background to capture token usage without altering the client’s view of the stream.

Usage Accounting

The LLMProxyUsage class tracks requests, response codes, and token counts. These counters integrate into the server’s /v1/usage endpoint under the llm_proxy key, as implemented in src/speech_to_speech/api/openai_realtime/websocket_router.py (lines 537-539).

Enabling the LLM Proxy Configuration

Activating the proxy requires specific CLI arguments and backend selection validated against the registry:

Step 1: Enable the Proxy Flag

Add --enable_llm_proxy to your launch command. This sets ModuleArguments.enable_llm_proxy = True as defined in src/speech_to_speech/arguments_classes/module_arguments.py (lines 69-78).

Step 2: Select a Compatible Backend

Choose --llm_backend chat-completions or --llm_backend responses-api. The BackendCapabilities.supports_llm_proxy property in src/speech_to_speech/backend_registry.py (lines 393-405) validates compatibility. Attempting to enable the proxy with an incompatible backend raises ValueError during pipeline initialization in src/speech_to_speech/s2s_pipeline.py (lines 113-118).

Step 3: Configure Upstream Credentials

Provide --llm_proxy_upstream_api_key or rely on automatic extraction from the backend specification. The build_llm_proxy_config function in src/speech_to_speech/s2s_pipeline.py (lines 106-127) constructs the final LLMProxyConfig object containing upstream_base_url, model_name, and connection timeouts.

Step 4: Initialize the Server

The RealtimeServer class in src/speech_to_speech/api/openai_realtime/server.py (lines 29-41) calls mount_llm_proxy during FastAPI application setup, exposing the proxy endpoints alongside the WebSocket voice interface.

Concurrent Processing and Pipeline Isolation

The LLM proxy operates outside the pipeline’s queue network, meaning it does not consume AudioInItem, AudioOutItem, or LMOutItem queue slots. This isolation ensures that:

• New audio chunks continue processing through VAD while LLM requests are in flight
• Multiple conversation turns can initiate before previous LLM responses complete
• Token usage from upstream providers is accounted for via LLMProxyUsage without introducing latency into the speech synthesis path

Error Handling and Resilience

The proxy implements specific HTTP status codes for failure modes:

• 501 Not Implemented: Returned by _error_response when the proxy is disabled or the backend lacks supports_llm_proxy=True
• 502 Bad Gateway: Returned by _upstream_unreachable when the external LLM provider connection fails
• Content preservation: Successful responses pass through unmodified, ensuring full compatibility with OpenAI-compatible clients regardless of the upstream provider

Practical Implementation Examples

Launching with CLI Arguments

python -m speech_to_speech serve \
    --enable_llm_proxy \
    --llm_backend chat-completions \
    --llm_proxy_connect_timeout_s 15.0

Building Configuration Programmatically

from speech_to_speech.s2s_pipeline import build_llm_proxy_config
from speech_to_speech.arguments_classes.module_arguments import ModuleArguments
from speech_to_speech.backend_registry import BackendSelection

module_args = ModuleArguments(enable_llm_proxy=True)

# Assume llm_backend is a BackendSelection instance

proxy_cfg = build_llm_proxy_config(module_args, llm_backend)
print(proxy_cfg.dict())

# Output includes: enabled, upstream_base_url, model_name, connect_timeout_s

Mounting in a Custom FastAPI Application

from fastapi import FastAPI
from speech_to_speech.api.openai_realtime.llm_proxy import LLMProxyConfig, mount_llm_proxy

app = FastAPI()
cfg = LLMProxyConfig(
    enabled=True,
    llm_backend="chat-completions",
    upstream_api_key="sk-my-key",
    model_name="gpt-4o",
)
proxy_usage = mount_llm_proxy(app, cfg)

# Inspect accumulated counters

print(f"Requests: {proxy_usage.requests}, Tokens: {proxy_usage.output_tokens}")

Querying Usage Statistics

curl http://localhost:8000/v1/usage

# Response includes:

# "llm_proxy": {

#   "requests": 3,

#   "responses_2xx": 3,

#   "input_tokens": 124,

#   "output_tokens": 342

# }

Summary

• The LLM Proxy prevents blocking in the voice pipeline by mounting OpenAI-compatible endpoints outside the main processing queues in llm_proxy.py
• Concurrent background tasks are achieved because the proxy does not consume AudioInItem or LMOutItem queue slots, allowing continuous audio processing
• Configuration requires --enable_llm_proxy with a compatible backend (chat-completions or responses-api) validated through backend_registry.py
• Token accounting occurs via LLMProxyUsage counters integrated into the /v1/usage endpoint without impacting streaming latency
• Error handling distinguishes between configuration errors (501), upstream failures (502), and successful passthrough responses

Frequently Asked Questions

Which LLM backends support proxying in the speech-to-speech server?

Only backends with supports_llm_proxy=True in their BackendCapabilities declaration can operate with the proxy enabled. According to src/speech_to_speech/backend_registry.py (lines 393-405), the chat-completions and responses-api backends support proxying, while others will raise a ValueError during pipeline initialization if the proxy is enabled.

Does the LLM proxy modify the content of streaming responses?

No. The proxy relays streaming responses byte-by-byte via StreamingResponse without altering the content. While forwarding, it asynchronously parses SSE events to extract usage statistics, but the client receives the exact bytes sent by the upstream provider, ensuring compatibility with any OpenAI-compatible client implementation.

How does the server track token usage when using an external LLM provider?

The LLMProxyUsage class accumulates counters for requests, response status codes, input tokens, and output tokens. For streaming responses, the proxy watches for SSE chunks containing usage data. These counters are exposed through the /v1/usage endpoint under the llm_proxy key, as implemented in websocket_router.py (lines 537-539), enabling accurate billing and monitoring without proxying overhead.

Can the LLM proxy operate independently of the speech pipeline?

Yes. While designed for the speech-to-speech architecture, the mount_llm_proxy function can attach to any FastAPI application. The proxy logic in llm_proxy.py is self-contained and only requires an LLMProxyConfig instance to function, making it suitable for standalone OpenAI-compatible proxy deployments or integration into other real-time systems.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}