# How to Use the LLM Proxy for Concurrent Background Tasks in Speech-to-Speech

> Learn how to use the LLM proxy for concurrent background tasks in Huggingface Speech-to-Speech. Enable non-blocking execution and streamline your audio pipeline.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-07

---

**The LLM proxy in Huggingface's Speech-to-Speech server forwards language model requests to remote providers without blocking the audio pipeline, enabling true concurrent execution.**

The `huggingface/speech-to-speech` repository provides an **LLM proxy feature** that lets you offload large language model calls to external APIs while your local speech pipeline keeps processing audio. This design is essential for real-time applications where blocking on model inference would break conversational flow. This guide explains how to configure, run, and monitor the proxy for concurrent background tasks.

## Enabling the LLM Proxy

The proxy is controlled by a single configuration flag and reuses the same credentials as the built-in language model handler.

### Command-Line Activation

```bash
speech-to-speech serve \
  --enable_llm_proxy \
  --llm_backend chat-completions \
  --responses_api_base_url https://router.huggingface.co/v1 \
  --responses_api_api_key hf_secret \
  --responses_api_model_name google/gemma-test

```

### Programmatic Configuration

For Python-based setups, construct the configuration using `build_llm_proxy_config` in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) (lines 148-165):

```python
from speech_to_speech.arguments_classes.module_arguments import ModuleArguments
from speech_to_speech.arguments_classes.responses_api_language_model_arguments import (
    ResponsesApiLanguageModelHandlerArguments,
)
from speech_to_speech.s2s_pipeline import build_llm_proxy_config

module_cfg = ModuleArguments(enable_llm_proxy=True, llm_backend="chat-completions")
lm_cfg = ResponsesApiLanguageModelHandlerArguments(
    model_name="google/gemma-test",
    responses_api_base_url="https://router.huggingface.co/v1",
    responses_api_api_key="hf_secret",
)

proxy_cfg = build_llm_proxy_config(module_cfg, lm_cfg)
print(proxy_cfg.enabled)           # True

print(proxy_cfg.upstream_base_url) # https://router.huggingface.co/v1

```

The `LLMProxyConfig` produced here feeds directly into the FastAPI application setup.

## Proxy Architecture and Endpoint Mounting

The proxy mounts two possible paths—`/v1/chat/completions` and `/v1/responses`—depending on which backend you've configured. In [`src/speech_to_speech/api/openai_realtime/llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/llm_proxy.py) (lines 38-77), the `mount_llm_proxy` function creates endpoint handlers where:

- The **matching backend path** becomes a streaming passthrough
- The **non-matching path** returns HTTP 501 Not Implemented

This is invoked from `websocket_router.create_app` in [`src/speech_to_speech/api/openai_realtime/websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py) (lines 419-445).

## Request Handling and Streaming Passthrough

### Non-Streaming Requests

For standard requests, the proxy:
1. Parses the incoming JSON body
2. Overwrites the `model` field with the configured `model_name`
3. Forwards via `httpx.AsyncClient`
4. Returns the upstream response verbatim
5. Records token usage from the response payload

See [`src/speech_to_speech/api/openai_realtime/llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/llm_proxy.py) lines 132-144.

### Streaming with Background Cleanup

For **streaming concurrent background tasks**, the proxy implements a more sophisticated pattern (lines 246-289):

```python

# Conceptual flow from llm_proxy.py implementation

async def streaming_handler(request):
    # 1. Inject include_usage: true for token accounting

    body = await request.json()
    body["include_usage"] = True
    
    # 2. Open streaming connection to upstream

    async with httpx.AsyncClient() as client:
        upstream = await client.stream("POST", upstream_url, json=body)
        
        # 3. Yield chunks immediately as they arrive

        async for chunk in upstream.aiter_raw():
            yield chunk
            
        # 4. Schedule background cleanup (runs even if client disconnects)

        background_tasks.add_task(close_resources, upstream, client)

```

The **Starlette background task** guarantees that `upstream.aclose()` and `client.aclose()` execute even when the client disconnects prematurely. This prevents connection leaks during high-concurrency scenarios.

## Achieving True Concurrency with the Speech Pipeline

The critical design advantage: **the proxy runs in its own ASGI handler scope**, completely separate from the speech processing queues. This means:

- **VAD** (voice activity detection) continues analyzing incoming audio
- **STT** (speech-to-text) keeps transcribing
- **TTS** (text-to-speech) maintains output generation
- **Compaction workers** process context windows

None of these components block, pause, or cancel when an LLM proxy request is active. The proxy's background cleanup task further ensures resource isolation—you can fire multiple concurrent LLM requests without impacting audio latency.

## Monitoring Proxy Usage

Every proxied request updates replica-local counters exposed at `/v1/usage`. The `LLMProxyUsage` model in [`src/speech_to_speech/api/openai_realtime/llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/llm_proxy.py) (lines 43-58) tracks:

- Total request count
- Response category distribution
- Input and output token totals

### Fetching Statistics

```python
import requests

usage = requests.get("http://localhost:8000/v1/usage").json()["llm_proxy"]
print(f"Requests: {usage['requests']}")
print(f"Input tokens: {usage.get('input_tokens', 0)}")
print(f"Output tokens: {usage.get('output_tokens', 0)}")

```

This endpoint is invaluable for capacity planning and debugging background task load.

## Client Integration Example

Here's a complete pattern for making non-blocking LLM calls from your application code:

```python
import httpx
import asyncio

async def background_llm_call():
    url = "http://localhost:8000/v1/chat/completions"
    payload = {
        "model": "any-value-overwritten",  # replaced by proxy config

        "messages": [{"role": "user", "content": "Summarize this conversation"}],
        "stream": True,
    }
    
    async with httpx.AsyncClient() as client:
        async with client.stream("POST", url, json=payload) as resp:
            # Process streaming response without blocking other work

            async for chunk in resp.aiter_text():
                # Handle Server-Sent Events format

                if chunk.startswith("data:"):
                    print(chunk)

# Run alongside your audio processing

async def main():
    await asyncio.gather(
        background_llm_call(),
        # your audio pipeline coroutine here

    )

```

## Summary

- **Enable with `--enable_llm_proxy`** or `ModuleArguments(enable_llm_proxy=True)` to activate the feature
- **Reuse existing credentials** from `ResponsesApiLanguageModelHandlerArguments`—no separate configuration needed
- **Streaming passthrough** with background cleanup prevents resource leaks on early disconnects
- **True concurrency** is achieved because the proxy operates outside the speech pipeline's queue system
- **Monitor via `/v1/usage`** to track background task load and token consumption

## Frequently Asked Questions

### Does the LLM proxy support both OpenAI Chat Completions and Responses APIs?

Yes. The `mount_llm_proxy` function in [`llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/llm_proxy.py) mounts either `/v1/chat/completions` or `/v1/responses` as the active passthrough endpoint based on your `--llm_backend` setting. The non-selected path returns 501 Not Implemented. This allows the same codebase to support both API styles without configuration confusion.

### What happens if the client disconnects during a streaming LLM response?

The Starlette background task scheduled in `streaming_passthrough` (lines 246-289) ensures cleanup runs regardless of client state. Even if the WebSocket or HTTP client drops, `upstream_response.aclose()` and `http_client.aclose()` execute via `background_tasks.add_task()`, preventing file descriptor and connection pool exhaustion.

### Can I run multiple concurrent LLM proxy requests simultaneously?

Absolutely. Each request gets its own `httpx.AsyncClient` instance and ASGI handler scope. There's no global lock or shared state that serializes proxied requests. The speech pipeline queues (VAD, STT, TTS) remain fully operational during any number of concurrent LLM calls. Monitor replica load through the `/v1/usage` endpoint to tune capacity.