LLM Proxy: Exposing the Configured LLM as an OpenAI-Compatible Endpoint in Hugging Face speech-to-speech

The LLM Proxy in the Hugging Face speech-to-speech repository enables the server to present any remote LLM backend as an OpenAI-compatible API by running a FastAPI proxy that forwards chat-completion requests to a configured upstream service.

The LLM Proxy is an optional component that bridges local speech-to-speech pipelines with external LLM providers. When enabled, it eliminates the need to host a local LLM, allowing you to route all language model requests to services like OpenAI, Anthropic, or any OpenAI-compatible endpoint. This article explains how the proxy works, how to configure it, and how to interact with it using standard OpenAI client libraries.

How the LLM Proxy Works

The proxy is implemented as a lightweight FastAPI sub-application that intercepts standard OpenAI API calls and forwards them to a remote backend. According to the huggingface/speech-to-speech source code, the core functionality lives in src/speech_to_speech/api/openai_realtime/llm_proxy.py.

Key Configuration Options

The LLMProxyConfig dataclass defines runtime behavior through these fields:

• enabled — toggles the proxy on or off
• llm_backend — selects the backend type (responses-api or chat-completions)
• connect_timeout_s — sets the TCP connection timeout for upstream requests
• upstream_url — specifies the remote LLM service URL (defaults to OpenAI's endpoint)

When disabled, the server falls back to its locally-hosted LLM using frameworks like transformers or mlx-lm.

Mounted Endpoints

The proxy registers two OpenAI-compatible endpoints via mount_llm_proxy(app, llm_proxy_config) in src/speech_to_speech/api/openai_realtime/websocket_router.py:

Endpoint	Purpose
/v1/chat/completions	Forwards client requests to the upstream LLM, streams responses, and records token usage
/v1/usage	Exposes server metrics including a dedicated llm_proxy section with request counters and error statistics

The proxy handles response compression, decodes upstream payloads, and updates internal usage counters to keep metrics accurate.

Enabling and Configuring the LLM Proxy

Command-Line Activation

The proxy is controlled through CLI flags defined in src/speech_to_speech/arguments_classes/module_arguments.py. To activate it:

python -m speech_to_speech.main \
    --enable_llm_proxy \
    --llm_backend responses-api \
    --llm_proxy_connect_timeout_s 5

These arguments are processed in src/speech_to_speech/s2s_pipeline.py, which constructs the LLMProxyConfig and passes it to the server initialization.

Backend Selection

Two backend modes are supported:

• responses-api — uses the newer OpenAI responses format
• chat-completions — targets the traditional chat completions endpoint

Choose based on your upstream provider's compatibility.

Using the LLM Proxy with OpenAI Clients

Once enabled, the proxy presents a fully OpenAI-compatible interface. You can use any standard client library without modification.

Basic Chat Completion

import openai

client = openai.OpenAI(
    api_key="dummy",  # not validated by the proxy

    base_url="http://localhost:8000/v1"
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Say hello!"}],
    stream=False
)

print(response.choices[0].message.content)

The proxy forwards this request to your configured upstream URL and returns the remote LLM's response transparently.

Streaming Responses

The proxy supports Server-Sent Events (SSE) streaming for real-time applications:

import openai

client = openai.OpenAI(
    api_key="dummy",
    base_url="http://localhost:8000/v1"
)

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Tell me a story"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Querying Proxy Usage Metrics

The augmented /v1/usage endpoint exposes proxy-specific telemetry:

import requests

usage = requests.get("http://localhost:8000/v1/usage")
proxy_stats = usage.json()["llm_proxy"]

print(f"Proxied requests: {proxy_stats['requests']}")
print(f"Bytes sent: {proxy_stats['bytes_sent']}")
print(f"Bytes received: {proxy_stats['bytes_received']}")
print(f"Errors: {proxy_stats['errors']}")

This integration allows monitoring of bandwidth consumption and error rates for the LLM proxy component specifically.

Architecture and Implementation Details

Request Flow

1. Client sends OpenAI-formatted request to local server (/v1/chat/completions)
2. LLMProxy receives request in src/speech_to_speech/api/openai_realtime/llm_proxy.py
3. Proxy reformats headers and body for the upstream provider
4. Async HTTP client forwards request with configured timeout
5. Response streams back through the proxy to the original client
6. Token counters and byte statistics are updated atomically

Error Handling

The proxy implements retry logic for transient failures and maintains an error counter exposed through the usage endpoint. Connection timeouts are configurable via --llm_proxy_connect_timeout_s to accommodate slow or distant upstream services.

Compression Support

Upstream responses using gzip or deflate compression are automatically detected and decompressed by the proxy before streaming to clients, ensuring compatibility with clients that don't support compressed SSE streams.

When to Use the LLM Proxy

Enable the LLM proxy when you want to:

• Offload LLM inference to managed services (OpenAI, Azure, Groq, etc.)
• Reduce local GPU memory requirements
• Access specialized models not available in open weights
• Maintain OpenAI API compatibility across deployments

Disable the proxy (default behavior) when you:

• Require offline operation or data privacy
• Want to minimize latency through local inference
• Need to run on hardware without external network access

Summary

• The LLM Proxy exposes any remote OpenAI-compatible service through a local FastAPI interface in the huggingface/speech-to-speech repository
• Core implementation resides in src/speech_to_speech/api/openai_realtime/llm_proxy.py with configuration through LLMProxyConfig
• Activate via --enable_llm_proxy CLI flag with optional timeout and backend selection
• Provides standard /v1/chat/completions and augmented /v1/usage endpoints
• Supports streaming, compression, and usage accounting for production monitoring

Frequently Asked Questions

How do I switch between local and remote LLM in the same deployment?

Toggle the --enable_llm_proxy flag when starting the server. Without this flag, src/speech_to_speech/s2s_pipeline.py initializes a local LLM using transformers or mlx-lm. With the flag enabled, all LLM requests route to your configured upstream_url. No client code changes are required between modes since both present the same OpenAI-compatible interface.

Can I use the LLM proxy with non-OpenAI providers?

Yes. Set --upstream_url to any OpenAI-compatible endpoint. The proxy expects standard chat completion request/response formats. Providers like Groq, Together AI, and self-hosted vLLM servers work transparently. Verify your provider's compatibility with the responses-api versus chat-completions backend modes.

Does the LLM proxy add significant latency?

The proxy introduces minimal overhead—typically single-digit milliseconds for request forwarding plus network round-trip time to the upstream service. The async implementation in llm_proxy.py uses httpx with connection pooling. Compression handling and streaming pass-through are optimized to avoid buffering entire responses.

What metrics are available for monitoring the LLM proxy?

The /v1/usage endpoint includes an llm_proxy object with: requests (total proxied calls), bytes_sent and bytes_received (network I/O), and errors (failed upstream connections). These counters persist for the server process lifetime and reset on restart. No external metrics service integration is required.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}