How to Load a Model and Processor in MLX-VLM: A Complete Guide

To load a model and processor in MLX-VLM, call mlx_vlm.utils.load(path_or_hf_repo) which returns a tuple of (model, processor) ready for inference.

The MLX-VLM library provides a unified loading pipeline that orchestrates model weights, configuration files, image processors, and tokenizers through a single entry point. Whether you are pulling a multimodal model from the Hugging Face Hub or loading a fine-tuned checkpoint from local storage, understanding the internal loading mechanism ensures efficient memory usage and correct processor initialization.

The load Entry Point

The primary interface for loading models is the load function defined in mlx_vlm/utils.py (lines 363‑379). This helper abstracts four distinct operations: path resolution, model instantiation, image processor construction, and tokenizer assembly.

from mlx_vlm.utils import load

model, processor = load(
    path_or_hf_repo="Qwen/Qwen2-VL-Chat",
    adapter_path=None,
    lazy=False,
    quantize_activations=False
)

The function accepts either a local directory path or a Hugging Face repository identifier. It returns a tuple containing the instantiated model and a fully configured processor object that handles both text tokenization and image preprocessing.

Step-by-Step Loading Pipeline

Understanding the internal pipeline helps debug loading errors and optimize performance for large vision-language models.

Path Resolution and Model Weights

First, load invokes get_model_path to resolve the input argument to a concrete directory on disk. It then calls load_model (lines 160‑188 in mlx_vlm/utils.py) to:

1. Read the model configuration JSON
2. Locate .safetensors weight files
3. Instantiate the appropriate model class
4. Optionally quantize weights
5. Evaluate parameters eagerly unless lazy=True

The lazy parameter controls whether model parameters are materialized immediately (lazy=False, default) or loaded on-demand during the first forward pass (lazy=True).

Image Processor Construction

MLX-VLM supports vision models with specialized image or video preprocessing requirements. The load_image_processor function (lines 459‑479) inspects the model class for a dedicated ImageProcessor or VideoProcessor implementation and constructs it if present. This component handles resizing, normalization, and tensor conversion for visual inputs.

Tokenizer and Processor Assembly

Finally, load_processor (lines 482‑514) builds a Hugging Face AutoProcessor or model-specific subclass. This step wires up:

• The tokenizer and detokenizer
• Stopping criteria for generation
• The eos_token_id extracted from the model configuration

If an image processor was created in the previous step, it is attached via processor.image_processor = image_processor before the function returns.

Optional LoRA Adapters

For parameter-efficient fine-tuning, MLX-VLM supports loading LoRA adapters alongside base models. When the adapter_path parameter is provided, load invokes apply_lora_layers (lines 404‑407 in mlx_vlm/utils.py) after instantiating the base model. The adapter weights are automatically merged into the model parameters during this phase.

model, processor = load(
    "base-model-repo",
    adapter_path="/path/to/lora/adapter",
    lazy=False
)

Lazy vs Eager Loading

The lazy parameter offers a trade-off between startup time and initial latency:

• lazy=False (default) – Calls mx.eval on all parameters immediately after loading. This guarantees the model resides fully in memory before returning, eliminating compilation delays during the first inference call.
• lazy=True – Defers weight materialization until the first forward pass. This reduces initial memory pressure for very large models but may cause a noticeable pause when processing the first request.

The lazy evaluation logic appears in mlx_vlm/utils.py within the load_model implementation.

Practical Code Examples

Loading from Hugging Face Hub

Load a Qwen2-VL model directly from the Hugging Face repository with eager initialization:

from mlx_vlm.utils import load

model, processor = load(
    "Qwen/Qwen2-VL-Chat",
    revision="main",
    lazy=False
)

prompt = "Describe this image: <image>"
outputs = model.generate(processor(prompt))
print(processor.decode(outputs[0]))

Loading Local Checkpoints

For locally saved models or custom fine-tunes, provide the absolute path to the directory containing config.json and .safetensors files:

from pathlib import Path
from mlx_vlm.utils import load

checkpoint_dir = Path("/home/user/models/my-vlm-checkpoint")
model, processor = load(str(checkpoint_dir), lazy=True)

# Parameters will materialize on first use

response = model.generate(processor("What is shown in this picture? <image>"))

Loading with LoRA Adapters

Combine a base model with task-specific LoRA weights:

model, processor = load(
    "mlx-community/Qwen2-VL-7B-Instruct",
    adapter_path="/home/user/checkpoints/lora-vision",
    lazy=False
)

result = model.generate(processor("Explain the chart data. <image>"))
print(processor.decode(result[0]))

Inspecting Processor Components

Verify that the image processor and tokenizer loaded correctly:

print(type(processor.image_processor))

# Output: <class 'mlx_vlm.models.qwen2_vl.processing_qwen2_vl.AutoImageProcessor'>

print(type(processor.tokenizer))

# Output: <class 'transformers.Qwen2TokenizerFast'>

Summary

• The mlx_vlm.utils.load function serves as the single entry point for model and processor initialization, handling both local directories and Hugging Face repositories.
• The loading pipeline explicitly separates concerns into load_model (weights), load_image_processor (vision preprocessing), and load_processor (tokenization).
• Set lazy=True to defer weight materialization and reduce startup memory for large models, or use lazy=False (default) for immediate readiness.
• Pass adapter_path to automatically merge LoRA adapters into the base model during loading.
• Source implementations reside primarily in mlx_vlm/utils.py with specific line ranges for each sub-operation.

Frequently Asked Questions

How do I load a model from a local directory instead of Hugging Face?

Pass the absolute path to the directory containing your config.json and weight files as the first argument to load. For example: load("/path/to/local/model", lazy=False). The function detects local paths automatically and skips remote resolution.

What is the difference between the processor and the model in MLX-VLM?

The model object contains the neural network weights and forward logic, while the processor handles input preparation (tokenizing text, preprocessing images) and output decoding. The processor typically wraps a Hugging Face tokenizer and may include a custom image processor attached at processor.image_processor.

Can I use lazy loading with LoRA adapters?

Yes. When you specify both lazy=True and adapter_path, the base model loads lazily while the LoRA weights are still applied correctly. However, note that the first forward pass will incur the combined overhead of materializing base parameters and merging adapter layers.

Where does MLX-VLM store downloaded Hugging Face models?

The library uses standard Hugging Face Hub caching mechanisms. When you provide a repository ID like "Qwen/Qwen2-VL-Chat", the get_model_path helper downloads files to the HF cache directory (typically ~/.cache/huggingface/hub) and returns the local path for subsequent loading operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}