# How the LLM Proxy Enables Concurrent Text Requests in Speech-to-Speech

> Discover how the LLM proxy in huggingface/speech-to-speech enables concurrent text requests. This FastAPI passthrough forwards requests asynchronously, allowing parallel text and audio processing without resource contention.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: internals
- Published: 2026-08-04

---

**The LLM proxy is a FastAPI passthrough that forwards OpenAI-compatible text-generation requests to a remote backend asynchronously, allowing text and audio processing to run in parallel without resource contention.**

The huggingface/speech-to-speech repository implements a lightweight proxy mechanism that isolates text-only LLM traffic from the real-time audio pipeline. This architectural separation ensures that demanding language model calls do not block speech transcription and generation queues.

## Architectural Isolation from the Speech Pipeline

The proxy achieves concurrency by registering endpoints that are completely independent of the speech-processing units.

### Separate Mount Points

In [`src/speech_to_speech/api/openai_realtime/llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/llm_proxy.py), the `mount_llm_proxy` function registers HTTP endpoints at `/v1/chat/completions` and `/v1/responses` that do not interact with the speech-pipeline queues or cancel scopes. As implemented in lines 9-10, this isolation guarantees that proxied text requests never touch speech-generation units, enabling true parallel execution.

### Selective Endpoint Activation

When enabled via configuration flags defined in [`src/speech_to_speech/api/openai_realtime/runtime_config.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/runtime_config.py), the proxy attaches a passthrough handler (`_mount_passthrough`) only for the configured backend—either `chat-completions` or `responses-api`—while registering an unavailable handler for the inactive path (lines 38-44 and 66-73). This selective mounting prevents route collisions and ensures that only the intended LLM backend is exposed, without blocking other server routes.

## Non-Blocking Request Forwarding

Concurrency relies on asynchronous HTTP handling rather than synchronous blocking calls.

### Streaming with httpx.AsyncClient

For streamed requests, the proxy utilizes `httpx.AsyncClient` to open a connection to the upstream LLM backend. The `_forward_and_account` coroutine yields response chunks immediately to the client while forwarding bytes in the background (lines 45-51, 70-82, and 84-89). Because the upstream request is performed asynchronously, the FastAPI event loop remains free to accept additional text requests concurrently, even while waiting for large language model generations.

### Connection-Scoped Timeouts

To prevent long-running generations from exhausting connection pools, the proxy applies `httpx.Timeout(None, connect=…)` with a configurable connect timeout while leaving read operations unlimited (lines 28-31 and 32-33). This design ensures that slow LLM responses do not block subsequent incoming requests, further supporting high-concurrency scenarios.

## Independent Resource Accounting

Each proxied request maintains its own metrics without interfering with speech pipeline telemetry.

### Replica-Local Usage Counters

The `LLMProxyUsage` class tracks request status, token payloads, and SSE events through methods like `usage.record_status`, `usage.record_token_payload`, and `usage.record_sse_event` (lines 51-69 and 70-78). These counters are replica-local and isolated from global speech-pipeline usage metrics managed in [`src/speech_to_speech/api/openai_realtime/pipeline_unit.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/pipeline_unit.py), eliminating contention between concurrent text calls and audio processing statistics.

## Implementation Example

Configure and mount the proxy in your server startup, typically within [`src/speech_to_speech/api/openai_realtime/server.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/server.py):

```python
from fastapi import FastAPI
from speech_to_speech.api.openai_realtime.llm_proxy import (
    LLMProxyConfig,
    mount_llm_proxy,
)

app = FastAPI()

proxy_cfg = LLMProxyConfig(
    enabled=True,
    llm_backend="chat-completions",
    upstream_base_url="https://api.openai.com/v1",
    upstream_api_key="sk-<YOUR-KEY>",
    model_name="gpt-4o-mini",
    connect_timeout_s=10.0,
)

llm_usage = mount_llm_proxy(app, proxy_cfg)

```

Send concurrent requests using standard OpenAI-compatible clients:

```bash

# Non-streaming request

curl -X POST https://my-server/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"What is the weather?"}],"stream":false}'

# Streaming request (runs concurrently)

curl -X POST https://my-server/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Tell me a story"}],"stream":true}'

```

Both calls execute independently while the audio pipeline continues processing speech.

## Summary

- **Isolated Endpoints**: The LLM proxy registers `/v1/chat/completions` and `/v1/responses` at separate mount points in [`llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/llm_proxy.py) (lines 9-10), ensuring text requests never interfere with speech pipeline queues.
- **Asynchronous Streaming**: Using `httpx.AsyncClient` and `_forward_and_account`, the proxy handles OpenAI-compatible streaming without blocking the FastAPI event loop (lines 45-51, 70-82).
- **Connection-Only Timeouts**: Timeouts are restricted to the connection phase (`httpx.Timeout(None, connect=…)`), allowing long LLM generations to complete without holding up concurrent requests (lines 28-33).
- **Independent Metrics**: `LLMProxyUsage` tracks text-generation metrics locally, preventing race conditions with speech-processing usage counters (lines 51-78).

## Frequently Asked Questions

### Can the LLM proxy handle multiple simultaneous streaming requests?

Yes. The proxy uses `httpx.AsyncClient` to manage HTTP connections asynchronously, allowing multiple streaming requests to `/v1/chat/completions` to run concurrently. Each stream yields chunks immediately while the FastAPI event loop processes other requests, as implemented in lines 45-51 of [`llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/llm_proxy.py).

### Does enabling the LLM proxy affect speech-to-speech latency?

No. Because the proxy endpoints are mounted independently of the speech pipeline (lines 9-10), text requests do not compete for the same queues or cancel scopes used by audio processing. This architectural separation ensures that LLM latency does not block real-time speech operations.

### Which LLM backends are supported by the proxy?

The proxy supports OpenAI-compatible `chat-completions` and `responses-api` backends. You select the active backend via the `llm_backend` parameter in `LLMProxyConfig`, and `mount_llm_proxy` attaches the appropriate passthrough handler while disabling the unused route (lines 38-44 and 66-73).

### How does the proxy prevent usage metric collisions?

The proxy instantiates a local `LLMProxyUsage` object that tracks text-generation metrics separately from the global speech pipeline counters. Methods like `record_status` and `record_token_payload` update replica-local state only (lines 51-69), ensuring concurrent text requests do not corrupt speech-related usage statistics.