How the LLM Proxy Handles Authentication and Rate Limiting for Side Tasks in Speech-to-Speech

The LLM proxy in the huggingface/speech-to-speech repository performs no authentication or rate limiting of its own, instead delegating these responsibilities to external gateway infrastructure while forwarding requests using a configurable upstream API key.

The LLM proxy feature provides an OpenAI-compatible passthrough for side tasks, forwarding requests from the Speech-to-Speech server to remote LLM providers. According to the source code in src/speech_to_speech/api/openai_realtime/llm_proxy.py, the proxy intentionally operates as a thin translation layer that relies on surrounding deployment components—such as reverse proxies, load balancers, or HF token gates—to enforce access control and throttling policies.

Authentication: Bearer Token Forwarding Without Validation

The proxy never validates incoming client tokens. Instead, it injects a Bearer token into every upstream request using the upstream_api_key parameter from LLMProxyConfig.

In llm_proxy.py (lines 27-28), the Authorization header is constructed as follows:

headers = {"Authorization": f"Bearer {config.upstream_api_key}"}

If upstream_api_key is omitted, requests are sent without an Authorization header, which upstream providers typically reject with a 401 error. The proxy assumes that client authentication is handled by the outer gateway before requests reach the proxy endpoint.

Rate Limiting Strategy: Observability Without Enforcement

Rather than enforcing throttling, the proxy records request statistics in a process-local LLMProxyUsage model for observability purposes.

The LLMProxyUsage Tracking Model

Located in llm_proxy.py (lines 51-58), this model tracks:

• Total request counts
• HTTP status code buckets (2xx, 4xx, 5xx, 429)
• Accumulated input and output tokens

The proxy updates these counters after each request:

usage.record_status(response.status_code)           # count 2xx/4xx/5xx/429

usage.record_token_payload(upstream.json())        # add token usage from JSON payloads

usage.record_sse_event(event)                     # parse SSE chunks for token usage

No throttling logic is applied to incoming requests. The module docstring (lines 5-6) explicitly states that the server "performs no authentication and no throttling of its own," expecting external components like Cloudflare gateways or internal rate limiters to enforce limits.

Configuring the LLM Proxy for Side Tasks

To enable the proxy with authentication and usage tracking, configure LLMProxyConfig through the CLI arguments defined in src/speech_to_speech/arguments_classes/module_arguments.py.

Basic Setup with Upstream Authentication

from speech_to_speech.s2s_pipeline import build_llm_proxy_config
from speech_to_speech.arguments_classes.module_arguments import ModuleArguments

module_kwargs = ModuleArguments(
    enable_llm_proxy=True,
    llm_backend="chat-completions",          # or "responses-api"

    llm_proxy_upstream_api_key="sk-my-api-key",
    llm_proxy_connect_timeout_s=15.0,
)
llm_proxy_cfg = build_llm_proxy_config(module_kwargs, args.llm_backend)

Mounting the Proxy Endpoints

The mount_llm_proxy function registers the OpenAI-compatible endpoints and returns a usage tracker:

from speech_to_speech.api.openai_realtime.llm_proxy import mount_llm_proxy
from fastapi import FastAPI

app = FastAPI()
proxy_usage = mount_llm_proxy(app, llm_proxy_cfg)   # registers /v1/chat/completions, /v1/responses

Querying Usage Statistics

Access aggregated metrics via the /v1/usage endpoint:

import httpx

resp = httpx.get("http://localhost:8000/v1/usage")
print(resp.json()["llm_proxy"])

# → {"requests": 12, "responses_2xx": 11, "responses_4xx": 1, "responses_429": 0,

#    "responses_5xx": 0, "input_tokens": 3450, "output_tokens": 780}

Implementation Details in the Source Code

The proxy's responsibilities are limited to three core functions:

1. Forwarding request bodies while injecting parameters like model, store=False, and include_usage=True for streaming
2. Recording usage metrics via the LLMProxyUsage counters
3. Returning upstream responses verbatim, including error payloads

This design is validated by integration tests in tests/openai_realtime/test_llm_proxy.py, which verify the proxy's behavior including its lack of built-in authentication and rate limiting.

Summary

• The LLM proxy delegates all authentication to external gateways, only forwarding a Bearer token from LLMProxyConfig.upstream_api_key to upstream providers.
• Rate limiting is not enforced by the proxy; instead, process-local usage statistics are recorded via LLMProxyUsage for observability.
• The proxy exposes metrics through the /v1/usage endpoint, tracking requests, status codes, and token counts without throttling traffic.
• Configuration occurs through ModuleArguments in arguments_classes/module_arguments.py and activation via build_llm_proxy_config in s2s_pipeline.py.

Frequently Asked Questions

Does the LLM proxy validate client tokens before forwarding requests?

No. The proxy does not perform any client token validation. It relies entirely on external infrastructure—such as the Hugging Face token gate, reverse proxies, or load balancers—to authenticate requests before they reach the proxy. The proxy simply forwards the configured upstream_api_key to the remote LLM provider.

How can I enforce rate limits on LLM proxy requests?

You must implement rate limiting in the surrounding deployment infrastructure, such as a Cloudflare gateway, NGINX reverse proxy, or Kubernetes ingress controller. The proxy itself only records usage statistics in the LLMProxyUsage model and exposes them via the /v1/usage endpoint without throttling any requests.

What happens if I don't provide an upstream API key?

If llm_proxy_upstream_api_key is omitted or empty, the proxy sends requests to the upstream provider without an Authorization header. The remote LLM provider will typically reject these requests with a 401 Unauthorized error, as the proxy does not generate or fallback to any default credentials.

Where are the LLM proxy usage statistics stored?

Usage statistics are stored in a process-local LLMProxyUsage instance created when mount_llm_proxy is called. This in-memory model tracks request counts, HTTP status code distributions (2xx, 4xx, 429, 5xx), and token usage metrics. The data is not persisted to disk or shared across processes unless explicitly captured via the /v1/usage HTTP endpoint.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}