# How to Use Vision Language Models with the Speech-to-Speech Pipeline

> Integrate vision language models into the Speech-to-Speech pipeline with ease. Learn how to set llm_is_vlm=True and provide image inputs for enhanced speech generation. Explore the huggingface speech-to-speech repository.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-07

---

**Enable vision inputs in the Speech-to-Speech pipeline by setting `llm_is_vlm=True` and providing image URLs or data-URIs alongside text prompts.**

The Hugging Face `speech-to-speech` repository supports **vision language models (VLMs)** natively, allowing you to process images alongside audio and text in a unified real-time conversation flow. This guide walks through the exact mechanism, configuration flags, and code patterns needed to activate VLM mode.

---

## How VLMs Work in the Speech-to-Speech Pipeline

The pipeline uses a single abstraction layer that automatically swaps between text-only LLMs and multimodal VLMs based on runtime arguments. Here's the internal flow as implemented in the source code.

### The `llm_is_vlm` Configuration Flag

In [`src/speech_to_speech/arguments_classes/language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/language_model_arguments.py), the `LanguageModelHandlerArguments` dataclass exposes a boolean flag:

```python
llm_is_vlm: bool = False  # Set to True to enable vision mode

```

This flag propagates through `BaseLanguageModelHandler.setup()` and determines which model class and processor get instantiated.

### Model Loading: Text vs. Vision

In [`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py), the `_load_model` method branches based on `self.llm_is_vlm`:

| Mode | Model Class | Processor |
|------|-------------|-----------|
| `llm_is_vlm=False` | `AutoModelForCausalLM` | `AutoTokenizer` |
| `llm_is_vlm=True` | `AutoModelForImageTextToText` | `AutoProcessor` |

The VLM stack uses Hugging Face's standard multimodal abstractions, so any model compatible with `AutoProcessor` and `AutoModelForImageTextToText` works out of the box.

### Image Handling and URL Decoding

When a user message contains an image, the Realtime API-style message includes an `input_image` part with an `image_url` field. The utility `image_url_to_pil` in [`src/speech_to_speech/LLM/utils.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/utils.py) handles:

- HTTP/HTTPS URL downloads
- Base64 data-URI decoding
- Local file path loading

It returns a `PIL.Image` object ready for the VLM processor.

### Prompt Construction for Multimodal Inputs

The method `_prepare_mlx_vlm_inputs` (or its Transformers equivalent in [`language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/language_model.py)) performs three steps:

1. Extract all image parts from the message list
2. Build a list of `PIL.Image` objects
3. Create a **formatted prompt** with image placeholders (typically `<image>` tokens)

The VLM receives `(images, formatted_prompt)` and generates text autoregressively.

### Speculative Turn Compatibility

Per [`src/speech_to_speech/pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py), the speculative-turn tracking system works **unchanged** for VLM outputs. Turn revisions, interruptions, and buffer management operate on the text level, so vision inputs don't complicate the conversation state machine.

---

## Command-Line Usage

The fastest way to start with VLMs is via the CLI. Pass `--llm-is-vlm` and select a vision-capable checkpoint:

```bash
python -m speech_to_speech \
    --llm-model-name microsoft/Phi-3-vision-128k-instruct \
    --llm-is-vlm \
    --audio-input-device default \
    --output-device default

```

**Key parameters explained:**

- `--llm-model-name`: Any Hugging Face model supporting `AutoModelForImageTextToText` (e.g., `Qwen/Qwen-VL-Chat`, `llava-hf/llava-v1.6-mistral-7b-hf`)
- `--llm-is-vlm`: Required flag; without it, the pipeline attempts to load as a text-only model and fails

---

## Programmatic Usage

For custom integrations, instantiate `SpeechToSpeechPipeline` with `LanguageModelHandlerArguments`:

```python
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline
from speech_to_speech.arguments_classes.language_model_arguments import LanguageModelHandlerArguments

# Configure VLM arguments

llm_args = LanguageModelHandlerArguments(
    llm_model_name="Qwen/Qwen-VL-Chat",
    llm_is_vlm=True,          # Enable vision mode

    llm_device="cuda",        # Or "mps", "cpu"

    llm_backend="transformers",  # Or "mlx" for Apple Silicon

)

# Build pipeline (add STT and TTS arguments as needed)

pipeline = SpeechToSpeechPipeline(
    language_model_args=llm_args,
    # stt_args=..., tts_args=..., etc.

)

# Send multimodal message: text + image URL

pipeline.send_user_message(
    parts=[
        ("text", "What objects are visible in this image?"),
        ("image", "https://upload.wikimedia.org/wikipedia/commons/9/9a/Gull_portrait_ca_usa.jpg"),
    ]
)

# Execute

pipeline.run()

```

The `parts` list follows the **OpenAI Realtime API convention**: `("image", url)` creates an `input_image` message part that the pipeline routes through `image_url_to_pil`.

---

## Low-Level Handler Access (Advanced)

To bypass the full pipeline and work directly with the language model handler:

```python
from speech_to_speech.LLM.language_model import LanguageModelHandler
from speech_to_speech.LLM.utils import image_url_to_pil

# Instantiate and configure

handler = LanguageModelHandler()
handler.setup(
    model_name="Qwen/Qwen-VL-Chat",
    llm_is_vlm=True,
    backend="transformers",
)

# Construct raw Realtime-style message

messages = [
    {"type": "input_image", "image_url": "https://example.com/chart.png"},
    {"type": "input_text", "text": "Summarize the trends in this chart."},
]

# Process and stream output

for token in handler.process_stream(messages):
    print(token, end="", flush=True)

```

This pattern is useful for:
- Custom preprocessing of image inputs
- Non-audio applications of the VLM backend
- Unit testing and debugging

---

## Supported Model Architectures

The following VLM families are confirmed compatible based on the `AutoModelForImageTextToText` requirement:

- **Qwen-VL** (`Qwen/Qwen-VL-Chat`, `Qwen/Qwen2-VL-*`)
- **LLaVA** (`llava-hf/llava-*`)
- **Phi-3 Vision** (`microsoft/Phi-3-vision-*`)
- **Idefics** (`HuggingFaceM4/idefics2-*`)
- **PaliGemma** (`google/paligemma-*`)

Check your target model's [`config.json`](https://github.com/huggingface/speech-to-speech/blob/main/config.json) for `"architectures"` containing `ImageTextToText` or `VisionEncoderDecoder` variants.

---

## Performance Considerations

| Factor | Impact |
|--------|--------|
| **Image resolution** | Higher resolution increases encoder compute; most VLMs resize to 336×336 or 448×448 internally |
| **Backend selection** | `mlx` (Apple Silicon) vs. `transformers` (CUDA/CPU) affects image encoder throughput |
| **Batching** | The pipeline processes single images per turn; batching multiple images increases latency linearly |
| **Speculative turns** | VLM text generation works with speculative decoding; image encoding is always eager |

For production deployments, pre-cache image embeddings if the same image appears across multiple turns—though the current `speech-to-speech` implementation does not expose this optimization.

---

## Summary

- **Set `llm_is_vlm=True`** in `LanguageModelHandlerArguments` to activate VLM mode
- **Provide images** via `input_image` message parts with URL or data-URI sources
- The pipeline **automatically loads** `AutoProcessor` + `AutoModelForImageTextToText` and handles image decoding via `image_url_to_pil`
- **Speculative turn tracking** and conversation state management work identically for text and vision outputs
- Supported models include Qwen-VL, LLaVA, Phi-3 Vision, and any Hugging Face `ImageTextToText` architecture

---

## Frequently Asked Questions

### What image formats are supported for VLM inputs?

The `image_url_to_pil` utility in [`src/speech_to_speech/LLM/utils.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/utils.py) uses PIL/Pillow under the hood, so any format Pillow supports—PNG, JPEG, GIF, WebP, BMP—works. For base64 data-URIs, the standard `data:image/png;base64,` or `data:image/jpeg;base64,` prefixes are required.

### Can I use multiple images in a single turn?

Yes. The `_prepare_mlx_vlm_inputs` method in [`language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/language_model.py) iterates over all message parts and collects every `input_image` into a list. The formatted prompt receives multiple `<image>` tokens in sequence. Verify your specific VLM's context window and image token budget—most support 1-4 images per prompt.

### Does the pipeline cache image embeddings between turns?

No. Per the current implementation in `speech-to-speech`, each turn's images are freshly encoded. The conversation history in [`src/speech_to_speech/api/openai_realtime/handlers/conversation.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/handlers/conversation.py) strips or retains image references according to the chat lifecycle, but embeddings are not reused. For repeated images, this adds encoder latency on every reference.