# LLM Proxy: Exposing the Configured LLM as an OpenAI-Compatible Endpoint in Hugging Face speech-to-speech

> Use the LLM Proxy in Hugging Face speech-to-speech to expose your configured LLM as an OpenAI-compatible API. This FastAPI proxy seamlessly forwards requests to your upstream service.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-03

---

**The LLM Proxy in the Hugging Face speech-to-speech repository enables the server to present any remote LLM backend as an OpenAI-compatible API by running a FastAPI proxy that forwards chat-completion requests to a configured upstream service.**

The **LLM Proxy** is an optional component that bridges local speech-to-speech pipelines with external LLM providers. When enabled, it eliminates the need to host a local LLM, allowing you to route all language model requests to services like OpenAI, Anthropic, or any OpenAI-compatible endpoint. This article explains how the proxy works, how to configure it, and how to interact with it using standard OpenAI client libraries.

## How the LLM Proxy Works

The proxy is implemented as a lightweight FastAPI sub-application that intercepts standard OpenAI API calls and forwards them to a remote backend. According to the `huggingface/speech-to-speech` source code, the core functionality lives in [`src/speech_to_speech/api/openai_realtime/llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/llm_proxy.py).

### Key Configuration Options

The `LLMProxyConfig` dataclass defines runtime behavior through these fields:

- **`enabled`** — toggles the proxy on or off
- **`llm_backend`** — selects the backend type (`responses-api` or `chat-completions`)
- **`connect_timeout_s`** — sets the TCP connection timeout for upstream requests
- **`upstream_url`** — specifies the remote LLM service URL (defaults to OpenAI's endpoint)

When disabled, the server falls back to its locally-hosted LLM using frameworks like `transformers` or `mlx-lm`.

### Mounted Endpoints

The proxy registers two OpenAI-compatible endpoints via `mount_llm_proxy(app, llm_proxy_config)` in [`src/speech_to_speech/api/openai_realtime/websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py):

| Endpoint | Purpose |
|----------|---------|
| `/v1/chat/completions` | Forwards client requests to the upstream LLM, streams responses, and records token usage |
| `/v1/usage` | Exposes server metrics including a dedicated `llm_proxy` section with request counters and error statistics |

The proxy handles response compression, decodes upstream payloads, and updates internal usage counters to keep metrics accurate.

## Enabling and Configuring the LLM Proxy

### Command-Line Activation

The proxy is controlled through CLI flags defined in [`src/speech_to_speech/arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/module_arguments.py). To activate it:

```bash
python -m speech_to_speech.main \
    --enable_llm_proxy \
    --llm_backend responses-api \
    --llm_proxy_connect_timeout_s 5

```

These arguments are processed in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py), which constructs the `LLMProxyConfig` and passes it to the server initialization.

### Backend Selection

Two backend modes are supported:

- **`responses-api`** — uses the newer OpenAI responses format
- **`chat-completions`** — targets the traditional chat completions endpoint

Choose based on your upstream provider's compatibility.

## Using the LLM Proxy with OpenAI Clients

Once enabled, the proxy presents a fully OpenAI-compatible interface. You can use any standard client library without modification.

### Basic Chat Completion

```python
import openai

client = openai.OpenAI(
    api_key="dummy",  # not validated by the proxy

    base_url="http://localhost:8000/v1"
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Say hello!"}],
    stream=False
)

print(response.choices[0].message.content)

```

The proxy forwards this request to your configured upstream URL and returns the remote LLM's response transparently.

### Streaming Responses

The proxy supports Server-Sent Events (SSE) streaming for real-time applications:

```python
import openai

client = openai.OpenAI(
    api_key="dummy",
    base_url="http://localhost:8000/v1"
)

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Tell me a story"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

```

### Querying Proxy Usage Metrics

The augmented `/v1/usage` endpoint exposes proxy-specific telemetry:

```python
import requests

usage = requests.get("http://localhost:8000/v1/usage")
proxy_stats = usage.json()["llm_proxy"]

print(f"Proxied requests: {proxy_stats['requests']}")
print(f"Bytes sent: {proxy_stats['bytes_sent']}")
print(f"Bytes received: {proxy_stats['bytes_received']}")
print(f"Errors: {proxy_stats['errors']}")

```

This integration allows monitoring of bandwidth consumption and error rates for the LLM proxy component specifically.

## Architecture and Implementation Details

### Request Flow

1. Client sends OpenAI-formatted request to local server (`/v1/chat/completions`)
2. `LLMProxy` receives request in [`src/speech_to_speech/api/openai_realtime/llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/llm_proxy.py)
3. Proxy reformats headers and body for the upstream provider
4. Async HTTP client forwards request with configured timeout
5. Response streams back through the proxy to the original client
6. Token counters and byte statistics are updated atomically

### Error Handling

The proxy implements retry logic for transient failures and maintains an error counter exposed through the usage endpoint. Connection timeouts are configurable via `--llm_proxy_connect_timeout_s` to accommodate slow or distant upstream services.

### Compression Support

Upstream responses using gzip or deflate compression are automatically detected and decompressed by the proxy before streaming to clients, ensuring compatibility with clients that don't support compressed SSE streams.

## When to Use the LLM Proxy

**Enable the LLM proxy** when you want to:

- Offload LLM inference to managed services (OpenAI, Azure, Groq, etc.)
- Reduce local GPU memory requirements
- Access specialized models not available in open weights
- Maintain OpenAI API compatibility across deployments

**Disable the proxy** (default behavior) when you:

- Require offline operation or data privacy
- Want to minimize latency through local inference
- Need to run on hardware without external network access

## Summary

- The **LLM Proxy** exposes any remote OpenAI-compatible service through a local FastAPI interface in the `huggingface/speech-to-speech` repository
- Core implementation resides in [`src/speech_to_speech/api/openai_realtime/llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/llm_proxy.py) with configuration through `LLMProxyConfig`
- Activate via `--enable_llm_proxy` CLI flag with optional timeout and backend selection
- Provides standard `/v1/chat/completions` and augmented `/v1/usage` endpoints
- Supports streaming, compression, and usage accounting for production monitoring

## Frequently Asked Questions

### How do I switch between local and remote LLM in the same deployment?

Toggle the `--enable_llm_proxy` flag when starting the server. Without this flag, [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) initializes a local LLM using `transformers` or `mlx-lm`. With the flag enabled, all LLM requests route to your configured `upstream_url`. No client code changes are required between modes since both present the same OpenAI-compatible interface.

### Can I use the LLM proxy with non-OpenAI providers?

Yes. Set `--upstream_url` to any OpenAI-compatible endpoint. The proxy expects standard chat completion request/response formats. Providers like Groq, Together AI, and self-hosted vLLM servers work transparently. Verify your provider's compatibility with the `responses-api` versus `chat-completions` backend modes.

### Does the LLM proxy add significant latency?

The proxy introduces minimal overhead—typically single-digit milliseconds for request forwarding plus network round-trip time to the upstream service. The async implementation in [`llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/llm_proxy.py) uses `httpx` with connection pooling. Compression handling and streaming pass-through are optimized to avoid buffering entire responses.

### What metrics are available for monitoring the LLM proxy?

The `/v1/usage` endpoint includes an `llm_proxy` object with: `requests` (total proxied calls), `bytes_sent` and `bytes_received` (network I/O), and `errors` (failed upstream connections). These counters persist for the server process lifetime and reset on restart. No external metrics service integration is required.