# Hugging Face Speech-to-Speech LLM Proxy Feature: Complete Usage Guide

> Learn how to use the Hugging Face Speech-to-Speech LLM proxy. This guide details how to forward requests to remote LLM backends with an OpenAI-compatible passthrough.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-05

---

**The LLM proxy in the Hugging Face Speech-to-Speech repository provides an OpenAI-compatible passthrough that lets a local Speech-to-Speech server forward chat-completion or responses-API requests to any remote LLM backend while keeping the client-side protocol unchanged.**

The **LLM proxy** is a specialized component in the Hugging Face Speech-to-Speech pipeline that solves a critical authentication and routing problem. When running in **realtime mode**, the FastAPI server accepts unauthenticated SDK calls but must still reach a secured upstream LLM. The proxy bridges this gap by enforcing a single server-side credential, guaranteeing consistent model configuration, and exposing detailed usage metrics through a `/v1/usage` endpoint.

---

## What the LLM Proxy Does

The proxy creates two local endpoints that mirror OpenAI's HTTP contract:

- `POST /v1/chat/completions` — for chat-completions backend
- `POST /v1/responses` — for responses-api backend

When enabled, every request to these routes is forwarded to a user-configured remote LLM endpoint. The proxy injects the server-side `model_name` and `upstream_api_key`, forces `store=False` for the responses API, and returns the upstream response verbatim—including streamed Server-Sent Events (SSE).

This design delivers three concrete benefits:

1. **Authentication isolation** — Clients need no credentials; the server holds the single upstream key.
2. **Configuration consistency** — The proxy reuses the same `responses_api_*` arguments as the pipeline's LM.
3. **Observability** — Live counters track requests, status codes, and token consumption.

---

## Core Components in the Source Code

The LLM proxy implementation spans four key files in the repository:

| File | Responsibility |
|------|--------------|
| [`src/speech_to_speech/api/openai_realtime/llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/llm_proxy.py) | Defines `LLMProxyConfig`, `LLMProxyUsage`, `mount_llm_proxy()`, and request-forwarding logic. |
| [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) | Builds the config via `build_llm_proxy_config()` from CLI arguments. |
| [`src/speech_to_speech/api/openai_realtime/server.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/server.py) | Creates the FastAPI app and invokes `mount_llm_proxy()` during server initialization. |
| [`tests/openai_realtime/test_llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/openai_realtime/test_llm_proxy.py) | Integration tests for passthrough, streaming, errors, and token accounting. |

In [`llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/llm_proxy.py), the `LLMProxyConfig` pydantic model captures all proxy settings:

```python
from pydantic import BaseModel

class LLMProxyConfig(BaseModel):
    enabled: bool
    llm_backend: str  # "chat-completions" or "responses-api"

    upstream_base_url: str
    upstream_api_key: str
    model_name: str
    connect_timeout_s: float

```

The `mount_llm_proxy(app, config)` function registers the routes. If `enabled=False` or an unsupported backend is selected, both routes return **501 Not Implemented** with a descriptive reason.

---

## Enabling the LLM Proxy from Command Line

The proxy activates only in **realtime mode**. Use these CLI flags to configure it:

```bash
python -m speech_to_speech.main \
    --mode realtime \
    --enable_llm_proxy \
    --llm_backend chat-completions \
    --llm_proxy_connect_timeout_s 5 \
    --responses_api_base_url https://my-remote-llm.com/v1 \
    --responses_api_api_key sk-remote-key \
    --responses_api_model_name my-remote-model

```

**Critical flag explanations:**

- `--enable_llm_proxy` — Toggles the feature on.
- `--llm_backend` — Must match your intended upstream protocol (`chat-completions` or `responses-api`).
- `--responses_api_*` — These arguments are shared with the pipeline LM. The `build_llm_proxy_config()` function in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) (lines 139–165) reads these to ensure the proxy mirrors the LM's upstream settings exactly.

---

## Programmatic Configuration and Mounting

For custom deployments, instantiate `LLMProxyConfig` directly:

```python
from speech_to_speech.api.openai_realtime.llm_proxy import LLMProxyConfig, mount_llm_proxy
from fastapi import FastAPI

proxy_cfg = LLMProxyConfig(
    enabled=True,
    llm_backend="chat-completions",
    upstream_base_url="https://my-remote-llm.com/v1",
    upstream_api_key="sk-remote-key",
    model_name="my-remote-model",
    connect_timeout_s=5.0,
)

app = FastAPI()
usage = mount_llm_proxy(app, proxy_cfg)

```

The `mount_llm_proxy()` function returns an `LLMProxyUsage` instance containing live counters. This object is updated on every proxied request and exposed through the server's `/v1/usage` endpoint.

---

## Error Handling Behavior

The proxy implements a layered error strategy as defined in [`llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/llm_proxy.py):

| Scenario | HTTP Status | Mechanism |
|----------|-------------|-----------|
| Proxy disabled or backend unsupported | 501 Not Implemented | Checked at route registration; returns immediate structured error. |
| Upstream unreachable (timeout, DNS failure, connection refused) | 502 Bad Gateway | `_upstream_unreachable()` helper wraps the `httpx` exception. |
| Upstream returns 4xx, 429, or 5xx | Passed through unchanged | Preserves original status code and body for client transparency. |

This design ensures clients receive meaningful diagnostics without leaking internal server details.

---

## Usage Statistics and Token Accounting

The proxy aggregates runtime metrics in `LLMProxyUsage`:

| Field | Description |
|-------|-------------|
| `requests` | Total proxied requests |
| `responses_2xx` | Successful upstream responses |
| `responses_4xx` | Client errors from upstream |
| `responses_429` | Rate-limited requests (separate for alerting) |
| `responses_5xx` | Server errors from upstream |
| `input_tokens` | Cumulative prompt tokens |
| `output_tokens` | Cumulative completion tokens |

Query the statistics via HTTP:

```http
GET /v1/usage HTTP/1.1
Host: localhost:8000

```

Example response excerpt:

```json
{
  "connections": { },
  "llm_proxy": {
    "requests": 12,
    "responses_2xx": 9,
    "responses_4xx": 2,
    "responses_429": 1,
    "responses_5xx": 0,
    "input_tokens": 342,
    "output_tokens": 210
  }
}

```

For streaming requests, the proxy automatically injects `"stream_options": {"include_usage": True}` to capture final token counts from the upstream SSE stream.

---

## Streaming Request Example

The LLM proxy transparently handles SSE streams. Here's a complete client example:

```python
import httpx

proxy_url = "http://localhost:8000/v1/chat/completions"
payload = {
    "messages": [{"role": "user", "content": "Explain the LLM proxy feature"}],
    "model": "any-model-name",  # Overridden by proxy

    "stream": True
}

with httpx.stream("POST", proxy_url, json=payload) as resp:
    for chunk in resp.iter_raw():
        print(chunk.decode(), end="")

```

Each SSE chunk passes through unmodified. After the stream completes, check `/v1/usage` for aggregated token totals.

---

## Integration with the Responses API

When using `llm_backend="responses-api"`, the proxy exposes `POST /v1/responses` while `POST /v1/chat/completions` returns 501. The proxy forces `store=False` to prevent upstream storage of conversation history, maintaining privacy consistent with the Speech-to-Speech pipeline's design.

```python
from fastapi import FastAPI
from speech_to_speech.api.openai_realtime.llm_proxy import LLMProxyConfig, mount_llm_proxy

app = FastAPI()
cfg = LLMProxyConfig(
    enabled=True,
    llm_backend="responses-api",
    upstream_base_url="https://api.openai.com/v1",
    upstream_api_key="sk-proxy-key",
    model_name="gpt-4o",
    connect_timeout_s=10.0,
)
mount_llm_proxy(app, cfg)  # Only /v1/responses is functional

```

---

## Summary

- The **LLM proxy** provides OpenAI-compatible passthrough routing for remote LLM backends in the Hugging Face Speech-to-Speech repository.
- It enforces server-side authentication, guarantees consistent model configuration, and records detailed usage statistics.
- Activation requires `--mode realtime` and `--enable_llm_proxy` via CLI, or manual `LLMProxyConfig` construction for custom deployments.
- Supported backends are `chat-completions` and `responses-api`; both cannot be active simultaneously on their respective routes.
- Error handling distinguishes configuration failures (501), upstream unreachability (502), and transparent passthrough of upstream HTTP statuses.
- Token accounting works for both streaming and non-streaming requests through automatic `include_usage` injection.

---

## Frequently Asked Questions

### What backends are compatible with the LLM proxy?

Any provider implementing the OpenAI HTTP contract works, including OpenAI, Azure OpenAI, and Hugging Face Inference API. The proxy supports `chat-completions` and `responses-api` protocol variants. You select one via the `llm_backend` configuration field; the other route returns 501 Not Implemented when active.

### Why does the proxy only work in realtime mode?

The Speech-to-Speech pipeline runs a FastAPI server exclusively during realtime operation. The `mount_llm_proxy()` call occurs inside the `RealtimeServer` initialization. In other modes (`--mode local`, `--mode api`), no server exists to host the proxy routes, so the feature is unavailable by design.

### How does token counting work for streaming responses?

The proxy automatically adds `"stream_options": {"include_usage": True}` to every streaming request. Upstream LLMs supporting this option emit a final SSE event containing `usage.prompt_tokens` and `usage.completion_tokens`. The proxy parses this event and increments `LLMProxyUsage.input_tokens` and `output_tokens` accordingly. Non-streaming requests extract token counts directly from the response body `usage` field.

### Can I use different credentials for the proxy and the pipeline LM?

No—and this is intentional. The `build_llm_proxy_config()` function in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) reads the identical `responses_api_base_url`, `responses_api_api_key`, and `responses_api_model_name` arguments used by the pipeline's LM. This guarantees that proxy requests and direct LM calls target the same upstream endpoint with the same authentication, preventing configuration drift.