How to Use the Chat Completions API for LLM Backend in Speech-to-Speech

The Speech-to-Speech repository provides an OpenAI-compatible Chat Completions backend that handles full LLM lifecycle management through ChatCompletionsApiModelHandler in speech_to_speech/LLM/chat_completions_language_model.py.

This backend enables you to connect any OpenAI-compatible endpoint—including vLLM, Qwen, or OpenAI itself—to your speech pipeline. The implementation wraps the standard /v1/chat/completions protocol with built-in streaming support, tool-call handling, and audio input capabilities.

What Is the Chat Completions Backend

The Chat Completions backend is registered under the identifier chat-completions and lives in src/speech_to_speech/LLM/chat_completions_language_model.py. It inherits from BaseOpenAICompatibleHandler in base_openai_compatible_language_model.py, which provides common infrastructure for OpenAI-compatible endpoints.

Key components include:

• ChatCompletionsApiModelHandler (lines 78-94): The main handler class implementing warmup, request building, and response parsing
• ChatCompletionsLanguageModelHandlerArguments (lines 9-21): Configuration dataclass exposing CLI flags like --responses_api_base_url and --responses_api_stream
• Serialization helpers: _chat_messages, _to_chat_content_part, _build_chat_optional_kwargs for converting internal Chat objects to OpenAI message format
• Streaming parser: _iter_chat_stream_events (lines 202-254) yielding TextDelta, AssistantMessage, ToolCall, and Usage events
• Non-streaming parser: _iter_chat_response_events (lines 56-76) for single-shot responses

The backend is registered in src/speech_to_speech/backend_registry.py (lines 96-104) and instantiated through _simple_handler_factory.

Configuring the Chat Completions Backend

All configuration flows through ChatCompletionsLanguageModelHandlerArguments, which extends ResponsesApiLanguageModelHandlerArguments. This design lets you reuse existing --responses_api_* flags across both backends.

Required parameters:

Flag	Purpose	Default
responses_api_base_url	Endpoint base URL with /v1 path	https://api.openai.com/v1
responses_api_api_key	Authentication key	None
responses_api_stream	Enable streaming responses	True
responses_api_reasoning_effort	Provider-specific reasoning level	"none"

Set responses_api_api_key="none" for local servers without authentication.

Running from Command Line

The fastest way to use the Chat Completions API for LLM backend is through the CLI:

speech-to-speech \
  --llm_backend chat-completions \
  --responses_api_base_url https://api.openai.com/v1 \
  --responses_api_api_key $OPENAI_API_KEY \
  --responses_api_stream true \
  --responses_api_reasoning_effort none \
  --tts_backend chatTTS \
  --audio_max_tokens 256

The --llm_backend chat-completions flag triggers the registry lookup in backend_registry.py, which instantiates ChatCompletionsApiModelHandler with your arguments.

Programmatic Usage

For custom integrations, instantiate the handler directly through the registry:

from speech_to_speech.arguments_classes.chat_completions_language_model_arguments import (
    ChatCompletionsLanguageModelHandlerArguments,
)
from speech_to_speech.backend_registry import LLM_BACKENDS
from speech_to_speech.pipeline.chat import Chat

# Build configuration

args = ChatCompletionsLanguageModelHandlerArguments(
    responses_api_base_url="http://localhost:8000/v1",
    responses_api_api_key="none",  # No auth for local vLLM

    responses_api_stream=True,
    responses_api_reasoning_effort="none",
)

# Retrieve handler from registry

spec = next(s for s in LLM_BACKENDS if s.name == "chat-completions")
handler = spec.handler_factory(args)  # Returns ChatCompletionsApiModelHandler

# Prepare conversation

chat = Chat()
chat.add_user_message("What's the weather in Paris?")

# Process streaming response

for event in handler.process(chat):
    if hasattr(event, 'text'):
        print(event.text, end="", flush=True)

The process method inherited from BaseHandler orchestrates: serialization via _chat_messages, dispatch through _request_chat_completions, and event iteration via _iter_chat_stream_events or _iter_chat_response_events.

Low-Level API Access

For fine-grained control, use the internal helpers directly from chat_completions_language_model.py:

from openai import OpenAI
from speech_to_speech.LLM.chat_completions_language_model import (
    _chat_messages,
    _build_chat_optional_kwargs,
    _request_chat_completions,
)

client = OpenAI(
    api_key="sk-...",
    base_url="https://api.openai.com/v1"
)

# Serialize Chat object to OpenAI format

messages = _chat_messages(chat)
optional = _build_chat_optional_kwargs(tools=None, tool_choice=None)

# Execute request

stream = _request_chat_completions(
    client=client,
    model_name="gpt-4o-mini",
    messages=messages,
    stream=True,
    extra_body=None,
    timeout=20.0,
    optional_kwargs=optional,
)

for chunk in stream:
    content = chunk.choices[0].delta.content
    print(content or "", end="")

This pattern is useful when you need custom tool configurations or direct access to raw ChatCompletionChunk objects.

Handling Audio Input and Tool Calls

The base handler in base_openai_compatible_language_model.py automatically manages:

• Audio token limits: Enforced via audio_max_tokens parameter
• Audio type selection: Chooses between input_audio and audio_url based on content
• History compaction: Prevents context window overflow
• Speculative-turn gating: Reduces latency for anticipated responses

Tool calls use the incremental choices[].delta.tool_calls protocol, with parsing implemented in _iter_chat_stream_events starting at line 202. Each ToolCall event contains tool_name, arguments, and id fields.

Architecture Flow

When processing a conversation, the handler executes this sequence:

1. Warmup: warmup method primes the connection with a trivial request
2. Serialization: _chat_messages converts Chat to OpenAI message array
3. Request: _request_chat_completions POSTs to /chat/completions
4. Parsing: Stream or response parser yields ProviderEvent objects
5. Integration: Events flow to TTS and other downstream pipeline stages

Summary

• The Chat Completions backend provides OpenAI-compatible LLM integration through ChatCompletionsApiModelHandler in src/speech_to_speech/LLM/chat_completions_language_model.py
• Configuration uses responses_api_* flags via ChatCompletionsLanguageModelHandlerArguments for CLI and programmatic access
• Streaming and non-streaming modes both produce ProviderEvent objects through dedicated parsers in the handler
• Tool calls and audio input are fully supported with automatic token management by the base handler
• The backend registers as chat-completions in backend_registry.py and instantiates through _simple_handler_factory

Frequently Asked Questions

What providers work with the Chat Completions backend?

Any OpenAI-compatible endpoint works, including OpenAI's official API, vLLM, and Qwen deployments. The handler is specifically recommended for providers that expose the mature Chat Completions streaming specification rather than the newer Responses API.

How do I disable streaming for the Chat Completions backend?

Set responses_api_stream=False in your arguments or pass --responses_api_stream false on the CLI. The handler automatically routes to _iter_chat_response_events instead of _iter_chat_stream_events for non-streaming operations.

What's the difference between Chat Completions and Responses API backends?

The Chat Completions backend uses the established /v1/chat/completions endpoint with full tool-call delta support, while the Responses API targets newer provider-specific protocols. Chat Completions has broader compatibility with local inference servers like vLLM. Both share configuration infrastructure through BaseOpenAICompatibleHandler.

How does audio input work with the Chat Completions handler?

Audio blobs in the Chat object are serialized through _to_chat_content_part into either input_audio (base64-encoded) or audio_url references. The base handler enforces audio_max_tokens limits and manages audio-type selection automatically based on content size and configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}