Which Models Are Supported by MLX-VLM? Complete List and Usage Guide

MLX-VLM officially supports four core vision-language architectures—Qwen2-VL, Qwen2.5-VL, Idefics3, and LLaVA—for image, video, and multi-modal chat inference on Apple Silicon.

The Blaizzy/mlx-vlm repository delivers optimized inference for vision-language models (VLMs) using Apple's MLX framework. Understanding exactly which models are supported by MLX-VLM is essential for developers building multimodal applications on macOS and iOS devices. This guide breaks down the officially supported architectures, provides runnable code examples for each, and explains how the library validates model compatibility through its source code.

Officially Supported Models

According to the README's "Supported Models" section, MLX-VLM provides validated support for the following architectures:

• Qwen2-VL: Alibaba's vision-language model (e.g., mlx-community/Qwen2-VL-2B-Instruct-4bit)
• Qwen2.5-VL: Enhanced successor with improved video understanding capabilities
• Idefics3: Hugging Face's multimodal architecture for image and video reasoning
• LLaVA: Large Language and Vision Assistant for general vision-language tasks

While additional models such as Pixtral and llava-interleaved appear in example notebooks, they are not part of the formal supported list and may have limited functionality.

Loading and Generating With Supported Models

Each supported model follows the same initialization pattern using mlx_vlm.load() and mlx_vlm.generate(), with apply_chat_template() handling prompt formatting. Below are implementation examples for each officially supported architecture.

Qwen2-VL for Image and Video

Qwen2-VL supports both static images and video sequences. The implementation uses max_pixels and fps parameters to control video preprocessing:

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model_path = "mlx-community/Qwen2-VL-2B-Instruct-4bit"
model, processor = load(model_path)
config = model.config

# Image generation

image_url = "http://images.cocodataset.org/val2017/000000039769.jpg"
prompt = "Describe this image."
formatted = apply_chat_template(processor, config, prompt, num_images=1)
output = generate(model, processor, formatted, [image_url])
print(output)

# Video generation

video_path = "path/to/video.mp4"
video_prompt = "Summarize what happens in this video."
formatted_vid = apply_chat_template(
    processor, config, video_prompt, num_images=0
)
output_vid = generate(
    model, processor, formatted_vid, video=video_path, max_pixels=(224, 224), fps=1.0
)
print(output_vid)

Qwen2.5-VL for Video Analysis

Qwen2.5-VL extends the architecture with refined temporal understanding, making it ideal for detailed video description tasks:

model_path = "mlx-community/Qwen2.5-VL-32B-Instruct-8bit"
model, processor = load(model_path)
config = model.config

video_path = "path/to/video.mp4"
prompt = "Explain the main events in this clip."

formatted = apply_chat_template(processor, config, prompt)
output = generate(
    model,
    processor,
    formatted,
    video=video_path,
    max_pixels=(224, 224),
    fps=1.0,
)
print(output)

Idefics3 for Video Summarization

Idefics3 provides competitive performance for vision-language tasks with efficient memory usage on Apple Silicon:

model_path = "mlx-community/Idefics3-8B-Instruct-4bit"
model, processor = load(model_path)

video_path = "path/to/video.mp4"
prompt = "Provide a concise summary of the video."

formatted = apply_chat_template(processor, model.config, prompt)
output = generate(
    model,
    processor,
    formatted,
    video=video_path,
    max_pixels=(224, 224),
    fps=1.0,
)
print(output)

LLaVA for Video Description

LLaVA remains a standard for general-purpose vision-language inference, supporting both single images and video sequences:

model_path = "mlx-community/LLaVA-13B-V1.5-4bit"
model, processor = load(model_path)

video_path = "path/to/video.mp4"
prompt = "Describe the scene and any actions happening."

formatted = apply_chat_template(processor, model.config, prompt)
output = generate(
    model,
    processor,
    formatted,
    video=video_path,
    max_pixels=(224, 224),
    fps=1.0,
)
print(output)

Model Validation and Architecture

MLX-VLM determines model capabilities through specific validation logic in the source code. The is_video_model function in mlx_vlm/video_generate.py validates whether a loaded architecture supports video inference, while mlx_vlm/utils.py provides get_model_and_args to handle model instantiation and configuration loading.

Abstract base classes defining the interface for these models reside in mlx_vlm/models/base.py, which establishes the contract for vision, language, and multimodal components. Reference implementations, such as the lightweight SmolVLM example in mlx_vlm/models/smolvlm/smolvlm.py, demonstrate how new architectures can extend these base classes to join the supported model roster.

Summary

• MLX-VLM officially supports Qwen2-VL, Qwen2.5-VL, Idefics3, and LLaVA for multimodal inference.
• All four models support video, image, and text inputs through the unified generate() API.
• Model validation occurs via is_video_model() in video_generate.py and configuration handling in utils.py.
• Experimental models like Pixtral exist in examples but lack official support documentation.
• Each model loads via mlx_vlm.load() and processes prompts through apply_chat_template() before generation.

Frequently Asked Questions

What distinguishes Qwen2.5-VL from Qwen2-VL in MLX-VLM?

Both models share the same initialization pattern through mlx_vlm.load(), but Qwen2.5-VL represents an updated architecture with enhanced video understanding capabilities. While Qwen2-VL handles both images and videos effectively, Qwen2.5-VL typically offers improved temporal reasoning for complex video analysis tasks according to the repository documentation.

Are experimental models like Pixtral fully supported?

Pixtral and llava-interleaved appear in example notebooks within the repository, but they are not part of the formal "Supported Models" list in the README. These implementations may work for specific use cases but lack the guaranteed compatibility and validation provided for the four official architectures.

How does MLX-VLM validate video support for a model?

The library checks video compatibility through the is_video_model function located in mlx_vlm/video_generate.py. This validation occurs before generation, ensuring that the loaded architecture can properly process video inputs using the max_pixels and fps parameters specified in the generate() call.

Where are the base classes for supported models defined?

Abstract base classes that define the interface for all supported vision-language models reside in mlx_vlm/models/base.py. These classes establish the contract for vision, language, and multimodal components, while specific implementations like SmolVLM demonstrate the extension pattern in mlx_vlm/models/smolvlm/smolvlm.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}