How to Use vLLM with the Responses API Backend for Speech-to-Speech Pipelines

Point the speech-to-speech library to a locally hosted vLLM server by setting --responses_api_base_url to your vLLM HTTP endpoint and matching the --responses_api_model_name to the value served by vLLM.

The Hugging Face speech-to-speech library orchestrates a modular voice agent pipeline (VAD → STT → LLM → TTS) where the LLM component communicates via any OpenAI-compatible Responses API server. Running this on a local vLLM instance requires only endpoint configuration—no code changes needed. This guide walks through CLI setup, Python API usage, and the alternative Chat Completions backend for tool-call reliability.

---

Start a vLLM Server with OpenAI-Compatible Responses API

Before connecting the pipeline, launch vLLM with the Responses API enabled. The server must expose the /v1/responses endpoint that the library expects.

vllm serve Qwen/Qwen-7B-Chat \
    --port 8000 \
    --api-key dummy \
    --served-model-name qwen

Key flags explained:

• --served-model-name qwen — registers the model under this identifier in the OpenAI-compatible API
• --api-key dummy — vLLM requires a key even for local use; match this value in pipeline configuration

The server now accepts requests at http://localhost:8000/v1.

---

Configure the Speech-to-Speech Pipeline for vLLM

The library's CLI arguments for the Responses API backend are defined in [src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py#L16-L53) (lines 16–53). These map directly to vLLM connection parameters.

CLI Configuration

speech_to_speech \
    --responses_api_base_url http://localhost:8000/v1 \
    --responses_api_api_key dummy \
    --responses_api_model_name qwen \
    --responses_api_stream true \
    --responses_api_audio_content_type input_audio

Argument	Purpose	vLLM Mapping
--responses_api_base_url	Root URL for the Responses API	http://localhost:8000/v1
--responses_api_api_key	Authentication token	Must match --api-key from vLLM launch
--responses_api_model_name	Model identifier to request	Must match --served-model-name
--responses_api_stream	Enable server-sent events for responses	Recommended for real-time pipelines
--responses_api_audio_content_type	Audio input format for multimodal models	input_audio for audio-native LLMs

The --responses_api_base_url parameter is documented in the repository README at line 382, which describes self-hosted server integration.

---

Python API Usage for vLLM Integration

For programmatic control, instantiate ResponsesApiLanguageModelHandlerArguments and pass it to the pipeline constructor.

from speech_to_speech import SpeechToSpeechPipeline
from speech_to_speech.arguments_classes.responses_api_language_model_arguments import (
    ResponsesApiLanguageModelHandlerArguments,
)

llm_args = ResponsesApiLanguageModelHandlerArguments(
    responses_api_base_url="http://localhost:8000/v1",
    responses_api_api_key="dummy",
    model_name="qwen",
    responses_api_stream=True,
    responses_api_audio_content_type="input_audio",
)

pipeline = SpeechToSpeechPipeline(
    llm_handler="responses_api",   # selects ResponsesApiModelHandler

    llm_handler_kwargs=llm_args.to_dict(),
)
pipeline.run()

The llm_handler="responses_api" string maps to ResponsesApiModelHandler via the backend registry in [src/speech_to_speech/backend_registry.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py#L389-L393) (lines 389–393). The actual HTTP client implementation lives in [src/speech_to_speech/LLM/responses_api_language_model.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/responses_api_language_model.py#L43-L50) (lines 43–50), which formats requests to the /v1/responses endpoint.

---

Tool-Call Streaming: When to Use Chat Completions Instead

The Responses API streaming implementation in some vLLM builds exhibits reliability issues with tool calls. The library maintainers recommend the Chat Completions backend for production tool-use scenarios, as noted in the README at line 431.

Switching to Chat Completions

speech_to_speech \
    --chat_completions_base_url http://localhost:8000/v1 \
    --chat_completions_model_name qwen \
    --chat_completions_stream true

This selects ChatCompletionsApiModelHandler from [src/speech_to_speech/LLM/chat_completions_language_model.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat_completions_language_model.py#L278-L285) (lines 278–285). The handler structure mirrors the Responses API implementation—see the parallel initialization code at line 283—ensuring identical behavior for non-tool-call use cases.

Both handlers are registered in [backend_registry.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py#L389-L393) under keys "responses_api" and "chat_completions" respectively.

---

Summary

• vLLM serves the Responses API at /v1/responses when launched with standard OpenAI-compatible flags
• Match four values between vLLM and the pipeline: host/port, API key, served model name, and endpoint version
• CLI flags for vLLM configuration are centralized in responses_api_language_model_arguments.py
• Python API uses ResponsesApiLanguageModelHandlerArguments with llm_handler="responses_api"
• Chat Completions fallback provides more reliable tool-call streaming via chat_completions_language_model.py

---

Frequently Asked Questions

What vLLM version supports the Responses API endpoint?

vLLM 0.5.0 and later include experimental Responses API support via the --enable-responses-api flag or automatic detection when /v1/responses is accessed. Verify your build with curl http://localhost:8000/v1/models and check that responses appear in the OpenAPI schema.

Why does my pipeline fail with "model not found" errors?

The --responses_api_model_name value must exactly match --served-model-name from your vLLM launch command. Mismatches cause 404 errors because vLLM validates model names against its registry before forwarding to the underlying weights.

Can I use vLLM with audio-native multimodal models in this pipeline?

Yes—set --responses_api_audio_content_type input_audio (as shown in the CLI example) when serving audio-native LLMs like GPT-4o-audio or compatible open counterparts. The handler passes audio bytes directly through the Responses API's native audio input fields.

How do I debug connection issues between the pipeline and vLLM?

Test the vLLM endpoint independently with: curl http://localhost:8000/v1/responses -H "Authorization: Bearer dummy" -d '{"model":"qwen","input":"test"}'. If this succeeds but the pipeline fails, verify --responses_api_api_key matches the vLLM --api-key exactly, including case sensitivity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}