How to Apply Chat Templates in MLX-VLM: Complete Guide

MLX-VLM constructs model-ready prompts from conversational messages using the apply_chat_template() function in mlx_vlm/prompt_utils.py, which automatically detects model-specific formats and handles multimodal inputs including images, video, and audio.

The Blaizzy/mlx-vlm library provides a unified interface for converting chat-style messages into the specific JSON or text formats required by different vision-language models. Understanding how to apply chat templates in MLX-VLM ensures your prompts render correctly whether you're using Qwen2-VL, Llama Vision, or other supported architectures.

Core Chat Template Architecture

Message Conversion Pipeline

The templating system begins with get_message_json() in mlx_vlm/prompt_utils.py. This function converts raw message dictionaries into model-specific JSON payloads through the MessageFormatter class.

MessageFormatter selects the appropriate representation from its formatter_map based on the model_type extracted from the model configuration. The mapping resides in the MODEL_CONFIG dictionary, which defines whether a model expects image tokens, list-with-image-first structures, or other specialized formats. For single-image-only models, the system references SINGLE_IMAGE_ONLY_MODELS to enforce constraints.

Template Execution Flow

The apply_chat_template() function (lines 633-640 in mlx_vlm/prompt_utils.py) serves as the primary entry point. It accepts a processor (or tokenizer), model config, and prompt input, then executes the following logic:

1. Detects if the processor provides an apply_chat_template method
2. If available, forwards messages to processor.apply_chat_template
3. If unavailable, falls back to _messages_to_plain_prompt (lines 354-375)

For multimodal content, _flatten_content (lines 887-925) processes image, audio, and video entries, replacing them with appropriate tokens like <image> while preserving surrounding whitespace.

How to Apply Chat Templates in MLX-VLM

Basic Usage with String Prompts

For text-only interactions, pass a string directly to apply_chat_template:

from mlx_vlm import load, apply_chat_template

model, processor = load("mlx-community/Qwen2-VL-7B")
config = model.config

prompt = "Describe the picture."
templated = apply_chat_template(
    processor,
    config,
    prompt,
    num_images=0,
    add_generation_prompt=True,
)

print(templated)   # → "User: Describe the picture.\nAssistant:"

This example demonstrates the plain-text fallback when no chat template is defined in the processor.

Multimodal Chat Templates with Images

When working with vision models, structure your messages as dictionaries with content lists:

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What is shown in the image?"},
            {"type": "image", "image": "examples/images/dog.jpg"},
        ],
    }
]

templated = apply_chat_template(
    processor,
    config,
    messages,
    num_images=1,               # tells the formatter to inject an image token

    add_generation_prompt=True,
)

print(templated)

# For a LIST_WITH_IMAGE_FIRST model (e.g., Qwen2‑VL) you get:

# [{'role': 'user', 'content': [{'type': 'image'}, {'type': 'text', 'text': 'What is shown in the image?'}]}, ...]

The MessageFormatter._format_list_with_image method (lines 777-801 in prompt_utils.py) handles the reordering based on the model's expected input structure.

High-Level Generation API Integration

The generate() function in mlx_vlm/generate.py (lines 1400-1449) internally calls apply_chat_template, but you can also preprocess prompts manually:

from mlx_vlm import generate, load, process_vision_info

model, processor = load("mlx-community/Qwen2-VL-7B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Explain the diagram."},
            {"type": "image", "image": "examples/images/diagram.png"},
        ],
    }
]

# Split vision inputs and obtain the textual prompt

image_inputs, _ = process_vision_info(messages)
input_prompt = processor.tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

response = generate(
    model,
    processor,
    prompt=input_prompt,
    images=image_inputs,
)

print(response)

Server-Side Chat Template Application

For API implementations, the MLX-VLM server utilizes apply_chat_template to format incoming requests. In mlx_vlm/server.py (lines 841-851), the function processes chat messages from REST API clients:

# Inside mlx_vlm/server.py (excerpt)

formatted_prompt = apply_chat_template(
    processor,
    model.config,
    request.messages,                # list of chat messages from the API client

    num_images=request.num_images,
    add_generation_prompt=True,
    enable_thinking=request.enable_thinking,
)

Fallback Mechanisms and Plain-Text Rendering

When a processor lacks a chat template, MLX-VLM defaults to _messages_to_plain_prompt. This function concatenates roles with standard prefixes (User:, Assistant:) and appends a generation prompt when add_generation_prompt=True. The fallback ensures compatibility with base models that haven't been fine-tuned for chat, while the return_messages=True parameter allows retrieval of raw message structures for custom processing.

Summary

• Primary entry point: Use apply_chat_template() in mlx_vlm/prompt_utils.py to convert messages into model-ready prompts.
• Automatic format selection: The MessageFormatter class maps model_type to specific JSON structures via MODEL_CONFIG.
• Multimodal support: Pass num_images and num_audios parameters to trigger token injection handled by _flatten_content.
• Graceful degradation: The system falls back to _messages_to_plain_prompt when processors lack native chat templates.
• Server integration: REST API endpoints in server.py demonstrate production-ready chat template application.

Frequently Asked Questions

What happens if my model processor lacks a chat template?

MLX-VLM detects the absence of apply_chat_template on the processor and automatically falls back to _messages_to_plain_prompt (lines 354-375 in prompt_utils.py). This function renders messages as plain text with role prefixes like "User:" and "Assistant:", ensuring compatibility with base models while maintaining conversational structure.

How does MLX-VLM handle image tokens in chat templates?

The library uses _flatten_content (lines 887-925 in prompt_utils.py) to traverse message content and replace image entries with model-specific tokens (e.g., <image>). The MessageFormatter then selects the appropriate layout—such as MessageFormat.IMAGE_TOKEN or MessageFormat.LIST_WITH_IMAGE_FIRST—based on the MODEL_CONFIG lookup for your specific model_type.

Can I retrieve raw message structures instead of templated strings?

Yes. Set return_messages=True when calling apply_chat_template(). This bypasses template rendering and returns the processed message list with proper JSON formatting for multimodal inputs, which is useful when you need to inspect or modify the intermediate representation before tokenization.

Which models support multimodal chat templates?

MLX-VLM supports chat templates across its vision-language model suite, including Qwen2-VL, Llama Vision, and other architectures defined in MODEL_CONFIG. The SINGLE_IMAGE_ONLY_MODELS constant identifies models with single-image constraints, while the formatter automatically handles variations in how images, video, and audio tokens are positioned within the prompt structure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}