Hugging Face Speech-to-Speech LLM Proxy Feature: Complete Usage Guide

The LLM proxy in the Hugging Face Speech-to-Speech repository provides an OpenAI-compatible passthrough that lets a local Speech-to-Speech server forward chat-completion or responses-API requests to any remote LLM backend while keeping the client-side protocol unchanged.

The LLM proxy is a specialized component in the Hugging Face Speech-to-Speech pipeline that solves a critical authentication and routing problem. When running in realtime mode, the FastAPI server accepts unauthenticated SDK calls but must still reach a secured upstream LLM. The proxy bridges this gap by enforcing a single server-side credential, guaranteeing consistent model configuration, and exposing detailed usage metrics through a /v1/usage endpoint.

---

What the LLM Proxy Does

The proxy creates two local endpoints that mirror OpenAI's HTTP contract:

• POST /v1/chat/completions — for chat-completions backend
• POST /v1/responses — for responses-api backend

When enabled, every request to these routes is forwarded to a user-configured remote LLM endpoint. The proxy injects the server-side model_name and upstream_api_key, forces store=False for the responses API, and returns the upstream response verbatim—including streamed Server-Sent Events (SSE).

This design delivers three concrete benefits:

1. Authentication isolation — Clients need no credentials; the server holds the single upstream key.
2. Configuration consistency — The proxy reuses the same responses_api_* arguments as the pipeline's LM.
3. Observability — Live counters track requests, status codes, and token consumption.

---

Core Components in the Source Code

The LLM proxy implementation spans four key files in the repository:

File	Responsibility
src/speech_to_speech/api/openai_realtime/llm_proxy.py	Defines LLMProxyConfig, LLMProxyUsage, mount_llm_proxy(), and request-forwarding logic.
src/speech_to_speech/s2s_pipeline.py	Builds the config via build_llm_proxy_config() from CLI arguments.
src/speech_to_speech/api/openai_realtime/server.py	Creates the FastAPI app and invokes mount_llm_proxy() during server initialization.
tests/openai_realtime/test_llm_proxy.py	Integration tests for passthrough, streaming, errors, and token accounting.

In llm_proxy.py, the LLMProxyConfig pydantic model captures all proxy settings:

from pydantic import BaseModel

class LLMProxyConfig(BaseModel):
    enabled: bool
    llm_backend: str  # "chat-completions" or "responses-api"

    upstream_base_url: str
    upstream_api_key: str
    model_name: str
    connect_timeout_s: float

The mount_llm_proxy(app, config) function registers the routes. If enabled=False or an unsupported backend is selected, both routes return 501 Not Implemented with a descriptive reason.

---

Enabling the LLM Proxy from Command Line

The proxy activates only in realtime mode. Use these CLI flags to configure it:

python -m speech_to_speech.main \
    --mode realtime \
    --enable_llm_proxy \
    --llm_backend chat-completions \
    --llm_proxy_connect_timeout_s 5 \
    --responses_api_base_url https://my-remote-llm.com/v1 \
    --responses_api_api_key sk-remote-key \
    --responses_api_model_name my-remote-model

Critical flag explanations:

• --enable_llm_proxy — Toggles the feature on.
• --llm_backend — Must match your intended upstream protocol (chat-completions or responses-api).
• --responses_api_* — These arguments are shared with the pipeline LM. The build_llm_proxy_config() function in s2s_pipeline.py (lines 139–165) reads these to ensure the proxy mirrors the LM's upstream settings exactly.

---

Programmatic Configuration and Mounting

For custom deployments, instantiate LLMProxyConfig directly:

from speech_to_speech.api.openai_realtime.llm_proxy import LLMProxyConfig, mount_llm_proxy
from fastapi import FastAPI

proxy_cfg = LLMProxyConfig(
    enabled=True,
    llm_backend="chat-completions",
    upstream_base_url="https://my-remote-llm.com/v1",
    upstream_api_key="sk-remote-key",
    model_name="my-remote-model",
    connect_timeout_s=5.0,
)

app = FastAPI()
usage = mount_llm_proxy(app, proxy_cfg)

The mount_llm_proxy() function returns an LLMProxyUsage instance containing live counters. This object is updated on every proxied request and exposed through the server's /v1/usage endpoint.

---

Error Handling Behavior

The proxy implements a layered error strategy as defined in llm_proxy.py:

Scenario	HTTP Status	Mechanism
Proxy disabled or backend unsupported	501 Not Implemented	Checked at route registration; returns immediate structured error.
Upstream unreachable (timeout, DNS failure, connection refused)	502 Bad Gateway	_upstream_unreachable() helper wraps the httpx exception.
Upstream returns 4xx, 429, or 5xx	Passed through unchanged	Preserves original status code and body for client transparency.

This design ensures clients receive meaningful diagnostics without leaking internal server details.

---

Usage Statistics and Token Accounting

The proxy aggregates runtime metrics in LLMProxyUsage:

Field	Description
requests	Total proxied requests
responses_2xx	Successful upstream responses
responses_4xx	Client errors from upstream
responses_429	Rate-limited requests (separate for alerting)
responses_5xx	Server errors from upstream
input_tokens	Cumulative prompt tokens
output_tokens	Cumulative completion tokens

Query the statistics via HTTP:

GET /v1/usage HTTP/1.1
Host: localhost:8000

Example response excerpt:

{
  "connections": { },
  "llm_proxy": {
    "requests": 12,
    "responses_2xx": 9,
    "responses_4xx": 2,
    "responses_429": 1,
    "responses_5xx": 0,
    "input_tokens": 342,
    "output_tokens": 210
  }
}

For streaming requests, the proxy automatically injects "stream_options": {"include_usage": True} to capture final token counts from the upstream SSE stream.

---

Streaming Request Example

The LLM proxy transparently handles SSE streams. Here's a complete client example:

import httpx

proxy_url = "http://localhost:8000/v1/chat/completions"
payload = {
    "messages": [{"role": "user", "content": "Explain the LLM proxy feature"}],
    "model": "any-model-name",  # Overridden by proxy

    "stream": True
}

with httpx.stream("POST", proxy_url, json=payload) as resp:
    for chunk in resp.iter_raw():
        print(chunk.decode(), end="")

Each SSE chunk passes through unmodified. After the stream completes, check /v1/usage for aggregated token totals.

---

Integration with the Responses API

When using llm_backend="responses-api", the proxy exposes POST /v1/responses while POST /v1/chat/completions returns 501. The proxy forces store=False to prevent upstream storage of conversation history, maintaining privacy consistent with the Speech-to-Speech pipeline's design.

from fastapi import FastAPI
from speech_to_speech.api.openai_realtime.llm_proxy import LLMProxyConfig, mount_llm_proxy

app = FastAPI()
cfg = LLMProxyConfig(
    enabled=True,
    llm_backend="responses-api",
    upstream_base_url="https://api.openai.com/v1",
    upstream_api_key="sk-proxy-key",
    model_name="gpt-4o",
    connect_timeout_s=10.0,
)
mount_llm_proxy(app, cfg)  # Only /v1/responses is functional

---

Summary

• The LLM proxy provides OpenAI-compatible passthrough routing for remote LLM backends in the Hugging Face Speech-to-Speech repository.
• It enforces server-side authentication, guarantees consistent model configuration, and records detailed usage statistics.
• Activation requires --mode realtime and --enable_llm_proxy via CLI, or manual LLMProxyConfig construction for custom deployments.
• Supported backends are chat-completions and responses-api; both cannot be active simultaneously on their respective routes.
• Error handling distinguishes configuration failures (501), upstream unreachability (502), and transparent passthrough of upstream HTTP statuses.
• Token accounting works for both streaming and non-streaming requests through automatic include_usage injection.

---

Frequently Asked Questions

What backends are compatible with the LLM proxy?

Any provider implementing the OpenAI HTTP contract works, including OpenAI, Azure OpenAI, and Hugging Face Inference API. The proxy supports chat-completions and responses-api protocol variants. You select one via the llm_backend configuration field; the other route returns 501 Not Implemented when active.

Why does the proxy only work in realtime mode?

The Speech-to-Speech pipeline runs a FastAPI server exclusively during realtime operation. The mount_llm_proxy() call occurs inside the RealtimeServer initialization. In other modes (--mode local, --mode api), no server exists to host the proxy routes, so the feature is unavailable by design.

How does token counting work for streaming responses?

The proxy automatically adds "stream_options": {"include_usage": True} to every streaming request. Upstream LLMs supporting this option emit a final SSE event containing usage.prompt_tokens and usage.completion_tokens. The proxy parses this event and increments LLMProxyUsage.input_tokens and output_tokens accordingly. Non-streaming requests extract token counts directly from the response body usage field.

Can I use different credentials for the proxy and the pipeline LM?

No—and this is intentional. The build_llm_proxy_config() function in s2s_pipeline.py reads the identical responses_api_base_url, responses_api_api_key, and responses_api_model_name arguments used by the pipeline's LM. This guarantees that proxy requests and direct LM calls target the same upstream endpoint with the same authentication, preventing configuration drift.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}