How to Use the LLM Proxy for Concurrent Background Tasks in Speech-to-Speech

The LLM proxy in Huggingface's Speech-to-Speech server forwards language model requests to remote providers without blocking the audio pipeline, enabling true concurrent execution.

The huggingface/speech-to-speech repository provides an LLM proxy feature that lets you offload large language model calls to external APIs while your local speech pipeline keeps processing audio. This design is essential for real-time applications where blocking on model inference would break conversational flow. This guide explains how to configure, run, and monitor the proxy for concurrent background tasks.

Enabling the LLM Proxy

The proxy is controlled by a single configuration flag and reuses the same credentials as the built-in language model handler.

Command-Line Activation

speech-to-speech serve \
  --enable_llm_proxy \
  --llm_backend chat-completions \
  --responses_api_base_url https://router.huggingface.co/v1 \
  --responses_api_api_key hf_secret \
  --responses_api_model_name google/gemma-test

Programmatic Configuration

For Python-based setups, construct the configuration using build_llm_proxy_config in src/speech_to_speech/s2s_pipeline.py (lines 148-165):

from speech_to_speech.arguments_classes.module_arguments import ModuleArguments
from speech_to_speech.arguments_classes.responses_api_language_model_arguments import (
    ResponsesApiLanguageModelHandlerArguments,
)
from speech_to_speech.s2s_pipeline import build_llm_proxy_config

module_cfg = ModuleArguments(enable_llm_proxy=True, llm_backend="chat-completions")
lm_cfg = ResponsesApiLanguageModelHandlerArguments(
    model_name="google/gemma-test",
    responses_api_base_url="https://router.huggingface.co/v1",
    responses_api_api_key="hf_secret",
)

proxy_cfg = build_llm_proxy_config(module_cfg, lm_cfg)
print(proxy_cfg.enabled)           # True

print(proxy_cfg.upstream_base_url) # https://router.huggingface.co/v1

The LLMProxyConfig produced here feeds directly into the FastAPI application setup.

Proxy Architecture and Endpoint Mounting

The proxy mounts two possible paths—/v1/chat/completions and /v1/responses—depending on which backend you've configured. In src/speech_to_speech/api/openai_realtime/llm_proxy.py (lines 38-77), the mount_llm_proxy function creates endpoint handlers where:

• The matching backend path becomes a streaming passthrough
• The non-matching path returns HTTP 501 Not Implemented

This is invoked from websocket_router.create_app in src/speech_to_speech/api/openai_realtime/websocket_router.py (lines 419-445).

Request Handling and Streaming Passthrough

Non-Streaming Requests

For standard requests, the proxy:

1. Parses the incoming JSON body
2. Overwrites the model field with the configured model_name
3. Forwards via httpx.AsyncClient
4. Returns the upstream response verbatim
5. Records token usage from the response payload

See src/speech_to_speech/api/openai_realtime/llm_proxy.py lines 132-144.

Streaming with Background Cleanup

For streaming concurrent background tasks, the proxy implements a more sophisticated pattern (lines 246-289):

# Conceptual flow from llm_proxy.py implementation

async def streaming_handler(request):
    # 1. Inject include_usage: true for token accounting

    body = await request.json()
    body["include_usage"] = True
    
    # 2. Open streaming connection to upstream

    async with httpx.AsyncClient() as client:
        upstream = await client.stream("POST", upstream_url, json=body)
        
        # 3. Yield chunks immediately as they arrive

        async for chunk in upstream.aiter_raw():
            yield chunk
            
        # 4. Schedule background cleanup (runs even if client disconnects)

        background_tasks.add_task(close_resources, upstream, client)

The Starlette background task guarantees that upstream.aclose() and client.aclose() execute even when the client disconnects prematurely. This prevents connection leaks during high-concurrency scenarios.

Achieving True Concurrency with the Speech Pipeline

The critical design advantage: the proxy runs in its own ASGI handler scope, completely separate from the speech processing queues. This means:

• VAD (voice activity detection) continues analyzing incoming audio
• STT (speech-to-text) keeps transcribing
• TTS (text-to-speech) maintains output generation
• Compaction workers process context windows

None of these components block, pause, or cancel when an LLM proxy request is active. The proxy's background cleanup task further ensures resource isolation—you can fire multiple concurrent LLM requests without impacting audio latency.

Monitoring Proxy Usage

Every proxied request updates replica-local counters exposed at /v1/usage. The LLMProxyUsage model in src/speech_to_speech/api/openai_realtime/llm_proxy.py (lines 43-58) tracks:

• Total request count
• Response category distribution
• Input and output token totals

Fetching Statistics

import requests

usage = requests.get("http://localhost:8000/v1/usage").json()["llm_proxy"]
print(f"Requests: {usage['requests']}")
print(f"Input tokens: {usage.get('input_tokens', 0)}")
print(f"Output tokens: {usage.get('output_tokens', 0)}")

This endpoint is invaluable for capacity planning and debugging background task load.

Client Integration Example

Here's a complete pattern for making non-blocking LLM calls from your application code:

import httpx
import asyncio

async def background_llm_call():
    url = "http://localhost:8000/v1/chat/completions"
    payload = {
        "model": "any-value-overwritten",  # replaced by proxy config

        "messages": [{"role": "user", "content": "Summarize this conversation"}],
        "stream": True,
    }
    
    async with httpx.AsyncClient() as client:
        async with client.stream("POST", url, json=payload) as resp:
            # Process streaming response without blocking other work

            async for chunk in resp.aiter_text():
                # Handle Server-Sent Events format

                if chunk.startswith("data:"):
                    print(chunk)

# Run alongside your audio processing

async def main():
    await asyncio.gather(
        background_llm_call(),
        # your audio pipeline coroutine here

    )

Summary

• Enable with --enable_llm_proxy or ModuleArguments(enable_llm_proxy=True) to activate the feature
• Reuse existing credentials from ResponsesApiLanguageModelHandlerArguments—no separate configuration needed
• Streaming passthrough with background cleanup prevents resource leaks on early disconnects
• True concurrency is achieved because the proxy operates outside the speech pipeline's queue system
• Monitor via /v1/usage to track background task load and token consumption

Frequently Asked Questions

Does the LLM proxy support both OpenAI Chat Completions and Responses APIs?

Yes. The mount_llm_proxy function in llm_proxy.py mounts either /v1/chat/completions or /v1/responses as the active passthrough endpoint based on your --llm_backend setting. The non-selected path returns 501 Not Implemented. This allows the same codebase to support both API styles without configuration confusion.

What happens if the client disconnects during a streaming LLM response?

The Starlette background task scheduled in streaming_passthrough (lines 246-289) ensures cleanup runs regardless of client state. Even if the WebSocket or HTTP client drops, upstream_response.aclose() and http_client.aclose() execute via background_tasks.add_task(), preventing file descriptor and connection pool exhaustion.

Can I run multiple concurrent LLM proxy requests simultaneously?

Absolutely. Each request gets its own httpx.AsyncClient instance and ASGI handler scope. There's no global lock or shared state that serializes proxied requests. The speech pipeline queues (VAD, STT, TTS) remain fully operational during any number of concurrent LLM calls. Monitor replica load through the /v1/usage endpoint to tune capacity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}