# How the LLM Proxy Handles Authentication and Rate Limiting for Side Tasks in Speech-to-Speech

> Discover how the LLM proxy in speech-to-speech handles authentication and rate limiting by delegating to external gateways and using upstream API keys for secure, controlled access.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: internals
- Published: 2026-08-08

---

**The LLM proxy in the huggingface/speech-to-speech repository performs no authentication or rate limiting of its own, instead delegating these responsibilities to external gateway infrastructure while forwarding requests using a configurable upstream API key.**

The LLM proxy feature provides an OpenAI-compatible passthrough for side tasks, forwarding requests from the Speech-to-Speech server to remote LLM providers. According to the source code in [`src/speech_to_speech/api/openai_realtime/llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/llm_proxy.py), the proxy intentionally operates as a thin translation layer that relies on surrounding deployment components—such as reverse proxies, load balancers, or HF token gates—to enforce access control and throttling policies.

## Authentication: Bearer Token Forwarding Without Validation

The proxy never validates incoming client tokens. Instead, it injects a Bearer token into every upstream request using the `upstream_api_key` parameter from `LLMProxyConfig`.

In [`llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/llm_proxy.py) (lines 27-28), the Authorization header is constructed as follows:

```python
headers = {"Authorization": f"Bearer {config.upstream_api_key}"}

```

If `upstream_api_key` is omitted, requests are sent without an Authorization header, which upstream providers typically reject with a 401 error. The proxy assumes that client authentication is handled by the outer gateway before requests reach the proxy endpoint.

## Rate Limiting Strategy: Observability Without Enforcement

Rather than enforcing throttling, the proxy records request statistics in a **process-local** `LLMProxyUsage` model for observability purposes.

### The LLMProxyUsage Tracking Model

Located in [`llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/llm_proxy.py) (lines 51-58), this model tracks:

- Total request counts
- HTTP status code buckets (2xx, 4xx, 5xx, 429)
- Accumulated input and output tokens

The proxy updates these counters after each request:

```python
usage.record_status(response.status_code)           # count 2xx/4xx/5xx/429

usage.record_token_payload(upstream.json())        # add token usage from JSON payloads

usage.record_sse_event(event)                     # parse SSE chunks for token usage

```

**No throttling logic is applied** to incoming requests. The module docstring (lines 5-6) explicitly states that the server "performs no authentication and no throttling of its own," expecting external components like Cloudflare gateways or internal rate limiters to enforce limits.

## Configuring the LLM Proxy for Side Tasks

To enable the proxy with authentication and usage tracking, configure `LLMProxyConfig` through the CLI arguments defined in [`src/speech_to_speech/arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/module_arguments.py).

### Basic Setup with Upstream Authentication

```python
from speech_to_speech.s2s_pipeline import build_llm_proxy_config
from speech_to_speech.arguments_classes.module_arguments import ModuleArguments

module_kwargs = ModuleArguments(
    enable_llm_proxy=True,
    llm_backend="chat-completions",          # or "responses-api"

    llm_proxy_upstream_api_key="sk-my-api-key",
    llm_proxy_connect_timeout_s=15.0,
)
llm_proxy_cfg = build_llm_proxy_config(module_kwargs, args.llm_backend)

```

### Mounting the Proxy Endpoints

The `mount_llm_proxy` function registers the OpenAI-compatible endpoints and returns a usage tracker:

```python
from speech_to_speech.api.openai_realtime.llm_proxy import mount_llm_proxy
from fastapi import FastAPI

app = FastAPI()
proxy_usage = mount_llm_proxy(app, llm_proxy_cfg)   # registers /v1/chat/completions, /v1/responses

```

### Querying Usage Statistics

Access aggregated metrics via the `/v1/usage` endpoint:

```python
import httpx

resp = httpx.get("http://localhost:8000/v1/usage")
print(resp.json()["llm_proxy"])

# → {"requests": 12, "responses_2xx": 11, "responses_4xx": 1, "responses_429": 0,

#    "responses_5xx": 0, "input_tokens": 3450, "output_tokens": 780}

```

## Implementation Details in the Source Code

The proxy's responsibilities are limited to three core functions:

1. **Forwarding** request bodies while injecting parameters like `model`, `store=False`, and `include_usage=True` for streaming
2. **Recording** usage metrics via the `LLMProxyUsage` counters
3. **Returning** upstream responses verbatim, including error payloads

This design is validated by integration tests in [`tests/openai_realtime/test_llm_proxy.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/openai_realtime/test_llm_proxy.py), which verify the proxy's behavior including its lack of built-in authentication and rate limiting.

## Summary

- The LLM proxy delegates all authentication to external gateways, only forwarding a Bearer token from `LLMProxyConfig.upstream_api_key` to upstream providers.
- Rate limiting is not enforced by the proxy; instead, process-local usage statistics are recorded via `LLMProxyUsage` for observability.
- The proxy exposes metrics through the `/v1/usage` endpoint, tracking requests, status codes, and token counts without throttling traffic.
- Configuration occurs through `ModuleArguments` in [`arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/arguments_classes/module_arguments.py) and activation via `build_llm_proxy_config` in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py).

## Frequently Asked Questions

### Does the LLM proxy validate client tokens before forwarding requests?

No. The proxy does not perform any client token validation. It relies entirely on external infrastructure—such as the Hugging Face token gate, reverse proxies, or load balancers—to authenticate requests before they reach the proxy. The proxy simply forwards the configured `upstream_api_key` to the remote LLM provider.

### How can I enforce rate limits on LLM proxy requests?

You must implement rate limiting in the surrounding deployment infrastructure, such as a Cloudflare gateway, NGINX reverse proxy, or Kubernetes ingress controller. The proxy itself only records usage statistics in the `LLMProxyUsage` model and exposes them via the `/v1/usage` endpoint without throttling any requests.

### What happens if I don't provide an upstream API key?

If `llm_proxy_upstream_api_key` is omitted or empty, the proxy sends requests to the upstream provider without an Authorization header. The remote LLM provider will typically reject these requests with a 401 Unauthorized error, as the proxy does not generate or fallback to any default credentials.

### Where are the LLM proxy usage statistics stored?

Usage statistics are stored in a **process-local** `LLMProxyUsage` instance created when `mount_llm_proxy` is called. This in-memory model tracks request counts, HTTP status code distributions (2xx, 4xx, 429, 5xx), and token usage metrics. The data is not persisted to disk or shared across processes unless explicitly captured via the `/v1/usage` HTTP endpoint.