How to Use Vision Language Models with the Speech-to-Speech Pipeline

Enable vision inputs in the Speech-to-Speech pipeline by setting llm_is_vlm=True and providing image URLs or data-URIs alongside text prompts.

The Hugging Face speech-to-speech repository supports vision language models (VLMs) natively, allowing you to process images alongside audio and text in a unified real-time conversation flow. This guide walks through the exact mechanism, configuration flags, and code patterns needed to activate VLM mode.

---

How VLMs Work in the Speech-to-Speech Pipeline

The pipeline uses a single abstraction layer that automatically swaps between text-only LLMs and multimodal VLMs based on runtime arguments. Here's the internal flow as implemented in the source code.

The llm_is_vlm Configuration Flag

In src/speech_to_speech/arguments_classes/language_model_arguments.py, the LanguageModelHandlerArguments dataclass exposes a boolean flag:

llm_is_vlm: bool = False  # Set to True to enable vision mode

This flag propagates through BaseLanguageModelHandler.setup() and determines which model class and processor get instantiated.

Model Loading: Text vs. Vision

In src/speech_to_speech/LLM/language_model.py, the _load_model method branches based on self.llm_is_vlm:

Mode	Model Class	Processor
llm_is_vlm=False	AutoModelForCausalLM	AutoTokenizer
llm_is_vlm=True	AutoModelForImageTextToText	AutoProcessor

The VLM stack uses Hugging Face's standard multimodal abstractions, so any model compatible with AutoProcessor and AutoModelForImageTextToText works out of the box.

Image Handling and URL Decoding

When a user message contains an image, the Realtime API-style message includes an input_image part with an image_url field. The utility image_url_to_pil in src/speech_to_speech/LLM/utils.py handles:

• HTTP/HTTPS URL downloads
• Base64 data-URI decoding
• Local file path loading

It returns a PIL.Image object ready for the VLM processor.

Prompt Construction for Multimodal Inputs

The method _prepare_mlx_vlm_inputs (or its Transformers equivalent in language_model.py) performs three steps:

1. Extract all image parts from the message list
2. Build a list of PIL.Image objects
3. Create a formatted prompt with image placeholders (typically <image> tokens)

The VLM receives (images, formatted_prompt) and generates text autoregressively.

Speculative Turn Compatibility

Per src/speech_to_speech/pipeline/speculative_turns.py, the speculative-turn tracking system works unchanged for VLM outputs. Turn revisions, interruptions, and buffer management operate on the text level, so vision inputs don't complicate the conversation state machine.

---

Command-Line Usage

The fastest way to start with VLMs is via the CLI. Pass --llm-is-vlm and select a vision-capable checkpoint:

python -m speech_to_speech \
    --llm-model-name microsoft/Phi-3-vision-128k-instruct \
    --llm-is-vlm \
    --audio-input-device default \
    --output-device default

Key parameters explained:

• --llm-model-name: Any Hugging Face model supporting AutoModelForImageTextToText (e.g., Qwen/Qwen-VL-Chat, llava-hf/llava-v1.6-mistral-7b-hf)
• --llm-is-vlm: Required flag; without it, the pipeline attempts to load as a text-only model and fails

---

Programmatic Usage

For custom integrations, instantiate SpeechToSpeechPipeline with LanguageModelHandlerArguments:

from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline
from speech_to_speech.arguments_classes.language_model_arguments import LanguageModelHandlerArguments

# Configure VLM arguments

llm_args = LanguageModelHandlerArguments(
    llm_model_name="Qwen/Qwen-VL-Chat",
    llm_is_vlm=True,          # Enable vision mode

    llm_device="cuda",        # Or "mps", "cpu"

    llm_backend="transformers",  # Or "mlx" for Apple Silicon

)

# Build pipeline (add STT and TTS arguments as needed)

pipeline = SpeechToSpeechPipeline(
    language_model_args=llm_args,
    # stt_args=..., tts_args=..., etc.

)

# Send multimodal message: text + image URL

pipeline.send_user_message(
    parts=[
        ("text", "What objects are visible in this image?"),
        ("image", "https://upload.wikimedia.org/wikipedia/commons/9/9a/Gull_portrait_ca_usa.jpg"),
    ]
)

# Execute

pipeline.run()

The parts list follows the OpenAI Realtime API convention: ("image", url) creates an input_image message part that the pipeline routes through image_url_to_pil.

---

Low-Level Handler Access (Advanced)

To bypass the full pipeline and work directly with the language model handler:

from speech_to_speech.LLM.language_model import LanguageModelHandler
from speech_to_speech.LLM.utils import image_url_to_pil

# Instantiate and configure

handler = LanguageModelHandler()
handler.setup(
    model_name="Qwen/Qwen-VL-Chat",
    llm_is_vlm=True,
    backend="transformers",
)

# Construct raw Realtime-style message

messages = [
    {"type": "input_image", "image_url": "https://example.com/chart.png"},
    {"type": "input_text", "text": "Summarize the trends in this chart."},
]

# Process and stream output

for token in handler.process_stream(messages):
    print(token, end="", flush=True)

This pattern is useful for:

• Custom preprocessing of image inputs
• Non-audio applications of the VLM backend
• Unit testing and debugging

---

Supported Model Architectures

The following VLM families are confirmed compatible based on the AutoModelForImageTextToText requirement:

• Qwen-VL (Qwen/Qwen-VL-Chat, Qwen/Qwen2-VL-*)
• LLaVA (llava-hf/llava-*)
• Phi-3 Vision (microsoft/Phi-3-vision-*)
• Idefics (HuggingFaceM4/idefics2-*)
• PaliGemma (google/paligemma-*)

Check your target model's config.json for "architectures" containing ImageTextToText or VisionEncoderDecoder variants.

---

Performance Considerations

Factor	Impact
Image resolution	Higher resolution increases encoder compute; most VLMs resize to 336×336 or 448×448 internally
Backend selection	mlx (Apple Silicon) vs. transformers (CUDA/CPU) affects image encoder throughput
Batching	The pipeline processes single images per turn; batching multiple images increases latency linearly
Speculative turns	VLM text generation works with speculative decoding; image encoding is always eager

For production deployments, pre-cache image embeddings if the same image appears across multiple turns—though the current speech-to-speech implementation does not expose this optimization.

---

Summary

• Set llm_is_vlm=True in LanguageModelHandlerArguments to activate VLM mode
• Provide images via input_image message parts with URL or data-URI sources
• The pipeline automatically loads AutoProcessor + AutoModelForImageTextToText and handles image decoding via image_url_to_pil
• Speculative turn tracking and conversation state management work identically for text and vision outputs
• Supported models include Qwen-VL, LLaVA, Phi-3 Vision, and any Hugging Face ImageTextToText architecture

---

Frequently Asked Questions

What image formats are supported for VLM inputs?

The image_url_to_pil utility in src/speech_to_speech/LLM/utils.py uses PIL/Pillow under the hood, so any format Pillow supports—PNG, JPEG, GIF, WebP, BMP—works. For base64 data-URIs, the standard data:image/png;base64, or data:image/jpeg;base64, prefixes are required.

Can I use multiple images in a single turn?

Yes. The _prepare_mlx_vlm_inputs method in language_model.py iterates over all message parts and collects every input_image into a list. The formatted prompt receives multiple <image> tokens in sequence. Verify your specific VLM's context window and image token budget—most support 1-4 images per prompt.

Does the pipeline cache image embeddings between turns?

No. Per the current implementation in speech-to-speech, each turn's images are freshly encoded. The conversation history in src/speech_to_speech/api/openai_realtime/handlers/conversation.py strips or retains image references according to the chat lifecycle, but embeddings are not reused. For repeated images, this adds encoder latency on every reference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}