How the LLM Proxy Enables Concurrent Text Requests in Speech-to-Speech

The LLM proxy is a FastAPI passthrough that forwards OpenAI-compatible text-generation requests to a remote backend asynchronously, allowing text and audio processing to run in parallel without resource contention.

The huggingface/speech-to-speech repository implements a lightweight proxy mechanism that isolates text-only LLM traffic from the real-time audio pipeline. This architectural separation ensures that demanding language model calls do not block speech transcription and generation queues.

Architectural Isolation from the Speech Pipeline

The proxy achieves concurrency by registering endpoints that are completely independent of the speech-processing units.

Separate Mount Points

In src/speech_to_speech/api/openai_realtime/llm_proxy.py, the mount_llm_proxy function registers HTTP endpoints at /v1/chat/completions and /v1/responses that do not interact with the speech-pipeline queues or cancel scopes. As implemented in lines 9-10, this isolation guarantees that proxied text requests never touch speech-generation units, enabling true parallel execution.

Selective Endpoint Activation

When enabled via configuration flags defined in src/speech_to_speech/api/openai_realtime/runtime_config.py, the proxy attaches a passthrough handler (_mount_passthrough) only for the configured backend—either chat-completions or responses-api—while registering an unavailable handler for the inactive path (lines 38-44 and 66-73). This selective mounting prevents route collisions and ensures that only the intended LLM backend is exposed, without blocking other server routes.

Non-Blocking Request Forwarding

Concurrency relies on asynchronous HTTP handling rather than synchronous blocking calls.

Streaming with httpx.AsyncClient

For streamed requests, the proxy utilizes httpx.AsyncClient to open a connection to the upstream LLM backend. The _forward_and_account coroutine yields response chunks immediately to the client while forwarding bytes in the background (lines 45-51, 70-82, and 84-89). Because the upstream request is performed asynchronously, the FastAPI event loop remains free to accept additional text requests concurrently, even while waiting for large language model generations.

Connection-Scoped Timeouts

To prevent long-running generations from exhausting connection pools, the proxy applies httpx.Timeout(None, connect=…) with a configurable connect timeout while leaving read operations unlimited (lines 28-31 and 32-33). This design ensures that slow LLM responses do not block subsequent incoming requests, further supporting high-concurrency scenarios.

Independent Resource Accounting

Each proxied request maintains its own metrics without interfering with speech pipeline telemetry.

Replica-Local Usage Counters

The LLMProxyUsage class tracks request status, token payloads, and SSE events through methods like usage.record_status, usage.record_token_payload, and usage.record_sse_event (lines 51-69 and 70-78). These counters are replica-local and isolated from global speech-pipeline usage metrics managed in src/speech_to_speech/api/openai_realtime/pipeline_unit.py, eliminating contention between concurrent text calls and audio processing statistics.

Implementation Example

Configure and mount the proxy in your server startup, typically within src/speech_to_speech/api/openai_realtime/server.py:

from fastapi import FastAPI
from speech_to_speech.api.openai_realtime.llm_proxy import (
    LLMProxyConfig,
    mount_llm_proxy,
)

app = FastAPI()

proxy_cfg = LLMProxyConfig(
    enabled=True,
    llm_backend="chat-completions",
    upstream_base_url="https://api.openai.com/v1",
    upstream_api_key="sk-<YOUR-KEY>",
    model_name="gpt-4o-mini",
    connect_timeout_s=10.0,
)

llm_usage = mount_llm_proxy(app, proxy_cfg)

Send concurrent requests using standard OpenAI-compatible clients:

# Non-streaming request

curl -X POST https://my-server/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"What is the weather?"}],"stream":false}'

# Streaming request (runs concurrently)

curl -X POST https://my-server/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Tell me a story"}],"stream":true}'

Both calls execute independently while the audio pipeline continues processing speech.

Summary

• Isolated Endpoints: The LLM proxy registers /v1/chat/completions and /v1/responses at separate mount points in llm_proxy.py (lines 9-10), ensuring text requests never interfere with speech pipeline queues.
• Asynchronous Streaming: Using httpx.AsyncClient and _forward_and_account, the proxy handles OpenAI-compatible streaming without blocking the FastAPI event loop (lines 45-51, 70-82).
• Connection-Only Timeouts: Timeouts are restricted to the connection phase (httpx.Timeout(None, connect=…)), allowing long LLM generations to complete without holding up concurrent requests (lines 28-33).
• Independent Metrics: LLMProxyUsage tracks text-generation metrics locally, preventing race conditions with speech-processing usage counters (lines 51-78).

Frequently Asked Questions

Can the LLM proxy handle multiple simultaneous streaming requests?

Yes. The proxy uses httpx.AsyncClient to manage HTTP connections asynchronously, allowing multiple streaming requests to /v1/chat/completions to run concurrently. Each stream yields chunks immediately while the FastAPI event loop processes other requests, as implemented in lines 45-51 of llm_proxy.py.

Does enabling the LLM proxy affect speech-to-speech latency?

No. Because the proxy endpoints are mounted independently of the speech pipeline (lines 9-10), text requests do not compete for the same queues or cancel scopes used by audio processing. This architectural separation ensures that LLM latency does not block real-time speech operations.

Which LLM backends are supported by the proxy?

The proxy supports OpenAI-compatible chat-completions and responses-api backends. You select the active backend via the llm_backend parameter in LLMProxyConfig, and mount_llm_proxy attaches the appropriate passthrough handler while disabling the unused route (lines 38-44 and 66-73).

How does the proxy prevent usage metric collisions?

The proxy instantiates a local LLMProxyUsage object that tracks text-generation metrics separately from the global speech pipeline counters. Methods like record_status and record_token_payload update replica-local state only (lines 51-69), ensuring concurrent text requests do not corrupt speech-related usage statistics.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}