# How to Apply Chat Templates in MLX-VLM: Complete Guide

> Learn how to apply chat templates in MLX-VLM with this complete guide. Discover how to use apply_chat_template() for seamless prompt construction with multimodal inputs.

- Repository: [Prince Canuma/mlx-vlm](https://github.com/Blaizzy/mlx-vlm)
- Tags: how-to-guide
- Published: 2026-04-05

---

**MLX-VLM constructs model-ready prompts from conversational messages using the `apply_chat_template()` function in [`mlx_vlm/prompt_utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/prompt_utils.py), which automatically detects model-specific formats and handles multimodal inputs including images, video, and audio.**

The `Blaizzy/mlx-vlm` library provides a unified interface for converting chat-style messages into the specific JSON or text formats required by different vision-language models. Understanding how to apply chat templates in MLX-VLM ensures your prompts render correctly whether you're using Qwen2-VL, Llama Vision, or other supported architectures.

## Core Chat Template Architecture

### Message Conversion Pipeline

The templating system begins with `get_message_json()` in [`mlx_vlm/prompt_utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/prompt_utils.py). This function converts raw message dictionaries into model-specific JSON payloads through the `MessageFormatter` class.

`MessageFormatter` selects the appropriate representation from its `formatter_map` based on the `model_type` extracted from the model configuration. The mapping resides in the `MODEL_CONFIG` dictionary, which defines whether a model expects image tokens, list-with-image-first structures, or other specialized formats. For single-image-only models, the system references `SINGLE_IMAGE_ONLY_MODELS` to enforce constraints.

### Template Execution Flow

The `apply_chat_template()` function (lines 633-640 in [`mlx_vlm/prompt_utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/prompt_utils.py)) serves as the primary entry point. It accepts a processor (or tokenizer), model config, and prompt input, then executes the following logic:

1. Detects if the processor provides an `apply_chat_template` method
2. If available, forwards messages to `processor.apply_chat_template`
3. If unavailable, falls back to `_messages_to_plain_prompt` (lines 354-375)

For multimodal content, `_flatten_content` (lines 887-925) processes image, audio, and video entries, replacing them with appropriate tokens like `<image>` while preserving surrounding whitespace.

## How to Apply Chat Templates in MLX-VLM

### Basic Usage with String Prompts

For text-only interactions, pass a string directly to `apply_chat_template`:

```python
from mlx_vlm import load, apply_chat_template

model, processor = load("mlx-community/Qwen2-VL-7B")
config = model.config

prompt = "Describe the picture."
templated = apply_chat_template(
    processor,
    config,
    prompt,
    num_images=0,
    add_generation_prompt=True,
)

print(templated)   # → "User: Describe the picture.\nAssistant:"

```

This example demonstrates the plain-text fallback when no chat template is defined in the processor.

### Multimodal Chat Templates with Images

When working with vision models, structure your messages as dictionaries with content lists:

```python
messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What is shown in the image?"},
            {"type": "image", "image": "examples/images/dog.jpg"},
        ],
    }
]

templated = apply_chat_template(
    processor,
    config,
    messages,
    num_images=1,               # tells the formatter to inject an image token

    add_generation_prompt=True,
)

print(templated)

# For a LIST_WITH_IMAGE_FIRST model (e.g., Qwen2‑VL) you get:

# [{'role': 'user', 'content': [{'type': 'image'}, {'type': 'text', 'text': 'What is shown in the image?'}]}, ...]

```

The `MessageFormatter._format_list_with_image` method (lines 777-801 in [`prompt_utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/prompt_utils.py)) handles the reordering based on the model's expected input structure.

### High-Level Generation API Integration

The `generate()` function in [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) (lines 1400-1449) internally calls `apply_chat_template`, but you can also preprocess prompts manually:

```python
from mlx_vlm import generate, load, process_vision_info

model, processor = load("mlx-community/Qwen2-VL-7B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Explain the diagram."},
            {"type": "image", "image": "examples/images/diagram.png"},
        ],
    }
]

# Split vision inputs and obtain the textual prompt

image_inputs, _ = process_vision_info(messages)
input_prompt = processor.tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

response = generate(
    model,
    processor,
    prompt=input_prompt,
    images=image_inputs,
)

print(response)

```

## Server-Side Chat Template Application

For API implementations, the MLX-VLM server utilizes `apply_chat_template` to format incoming requests. In [`mlx_vlm/server.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py) (lines 841-851), the function processes chat messages from REST API clients:

```python

# Inside mlx_vlm/server.py (excerpt)

formatted_prompt = apply_chat_template(
    processor,
    model.config,
    request.messages,                # list of chat messages from the API client

    num_images=request.num_images,
    add_generation_prompt=True,
    enable_thinking=request.enable_thinking,
)

```

## Fallback Mechanisms and Plain-Text Rendering

When a processor lacks a chat template, MLX-VLM defaults to `_messages_to_plain_prompt`. This function concatenates roles with standard prefixes (`User:`, `Assistant:`) and appends a generation prompt when `add_generation_prompt=True`. The fallback ensures compatibility with base models that haven't been fine-tuned for chat, while the `return_messages=True` parameter allows retrieval of raw message structures for custom processing.

## Summary

- **Primary entry point**: Use `apply_chat_template()` in [`mlx_vlm/prompt_utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/prompt_utils.py) to convert messages into model-ready prompts.
- **Automatic format selection**: The `MessageFormatter` class maps `model_type` to specific JSON structures via `MODEL_CONFIG`.
- **Multimodal support**: Pass `num_images` and `num_audios` parameters to trigger token injection handled by `_flatten_content`.
- **Graceful degradation**: The system falls back to `_messages_to_plain_prompt` when processors lack native chat templates.
- **Server integration**: REST API endpoints in [`server.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/server.py) demonstrate production-ready chat template application.

## Frequently Asked Questions

### What happens if my model processor lacks a chat template?

MLX-VLM detects the absence of `apply_chat_template` on the processor and automatically falls back to `_messages_to_plain_prompt` (lines 354-375 in [`prompt_utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/prompt_utils.py)). This function renders messages as plain text with role prefixes like "User:" and "Assistant:", ensuring compatibility with base models while maintaining conversational structure.

### How does MLX-VLM handle image tokens in chat templates?

The library uses `_flatten_content` (lines 887-925 in [`prompt_utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/prompt_utils.py)) to traverse message content and replace image entries with model-specific tokens (e.g., `<image>`). The `MessageFormatter` then selects the appropriate layout—such as `MessageFormat.IMAGE_TOKEN` or `MessageFormat.LIST_WITH_IMAGE_FIRST`—based on the `MODEL_CONFIG` lookup for your specific `model_type`.

### Can I retrieve raw message structures instead of templated strings?

Yes. Set `return_messages=True` when calling `apply_chat_template()`. This bypasses template rendering and returns the processed message list with proper JSON formatting for multimodal inputs, which is useful when you need to inspect or modify the intermediate representation before tokenization.

### Which models support multimodal chat templates?

MLX-VLM supports chat templates across its vision-language model suite, including Qwen2-VL, Llama Vision, and other architectures defined in `MODEL_CONFIG`. The `SINGLE_IMAGE_ONLY_MODELS` constant identifies models with single-image constraints, while the formatter automatically handles variations in how images, video, and audio tokens are positioned within the prompt structure.