# How to Load a Model and Processor in MLX-VLM: A Complete Guide

> Effortlessly load model and processor in MLX-VLM using mlx_vlm.utils.load. Get your MLX-VLM setup for inference quickly with this simple guide.

- Repository: [Prince Canuma/mlx-vlm](https://github.com/Blaizzy/mlx-vlm)
- Tags: how-to-guide
- Published: 2026-04-05

---

**To load a model and processor in MLX-VLM, call `mlx_vlm.utils.load(path_or_hf_repo)` which returns a tuple of `(model, processor)` ready for inference.**

The **MLX-VLM** library provides a unified loading pipeline that orchestrates model weights, configuration files, image processors, and tokenizers through a single entry point. Whether you are pulling a multimodal model from the Hugging Face Hub or loading a fine-tuned checkpoint from local storage, understanding the internal loading mechanism ensures efficient memory usage and correct processor initialization.

## The `load` Entry Point

The primary interface for loading models is the `load` function defined in [`mlx_vlm/utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/utils.py) (lines 363‑379). This helper abstracts four distinct operations: path resolution, model instantiation, image processor construction, and tokenizer assembly.

```python
from mlx_vlm.utils import load

model, processor = load(
    path_or_hf_repo="Qwen/Qwen2-VL-Chat",
    adapter_path=None,
    lazy=False,
    quantize_activations=False
)

```

The function accepts either a local directory path or a Hugging Face repository identifier. It returns a tuple containing the instantiated model and a fully configured processor object that handles both text tokenization and image preprocessing.

## Step-by-Step Loading Pipeline

Understanding the internal pipeline helps debug loading errors and optimize performance for large vision-language models.

### Path Resolution and Model Weights

First, `load` invokes `get_model_path` to resolve the input argument to a concrete directory on disk. It then calls `load_model` (lines 160‑188 in [`mlx_vlm/utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/utils.py)) to:

1. Read the model configuration JSON
2. Locate `.safetensors` weight files
3. Instantiate the appropriate model class
4. Optionally quantize weights
5. Evaluate parameters eagerly unless `lazy=True`

The `lazy` parameter controls whether model parameters are materialized immediately (`lazy=False`, default) or loaded on-demand during the first forward pass (`lazy=True`).

### Image Processor Construction

MLX-VLM supports vision models with specialized image or video preprocessing requirements. The `load_image_processor` function (lines 459‑479) inspects the model class for a dedicated `ImageProcessor` or `VideoProcessor` implementation and constructs it if present. This component handles resizing, normalization, and tensor conversion for visual inputs.

### Tokenizer and Processor Assembly

Finally, `load_processor` (lines 482‑514) builds a Hugging Face `AutoProcessor` or model-specific subclass. This step wires up:

- The **tokenizer** and **detokenizer**
- **Stopping criteria** for generation
- The **eos_token_id** extracted from the model configuration

If an image processor was created in the previous step, it is attached via `processor.image_processor = image_processor` before the function returns.

## Optional LoRA Adapters

For parameter-efficient fine-tuning, MLX-VLM supports loading LoRA adapters alongside base models. When the `adapter_path` parameter is provided, `load` invokes `apply_lora_layers` (lines 404‑407 in [`mlx_vlm/utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/utils.py)) after instantiating the base model. The adapter weights are automatically merged into the model parameters during this phase.

```python
model, processor = load(
    "base-model-repo",
    adapter_path="/path/to/lora/adapter",
    lazy=False
)

```

## Lazy vs Eager Loading

The `lazy` parameter offers a trade-off between startup time and initial latency:

- **`lazy=False` (default)** – Calls `mx.eval` on all parameters immediately after loading. This guarantees the model resides fully in memory before returning, eliminating compilation delays during the first inference call.
- **`lazy=True`** – Defers weight materialization until the first forward pass. This reduces initial memory pressure for very large models but may cause a noticeable pause when processing the first request.

The lazy evaluation logic appears in [`mlx_vlm/utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/utils.py) within the `load_model` implementation.

## Practical Code Examples

### Loading from Hugging Face Hub

Load a Qwen2-VL model directly from the Hugging Face repository with eager initialization:

```python
from mlx_vlm.utils import load

model, processor = load(
    "Qwen/Qwen2-VL-Chat",
    revision="main",
    lazy=False
)

prompt = "Describe this image: <image>"
outputs = model.generate(processor(prompt))
print(processor.decode(outputs[0]))

```

### Loading Local Checkpoints

For locally saved models or custom fine-tunes, provide the absolute path to the directory containing [`config.json`](https://github.com/Blaizzy/mlx-vlm/blob/main/config.json) and `.safetensors` files:

```python
from pathlib import Path
from mlx_vlm.utils import load

checkpoint_dir = Path("/home/user/models/my-vlm-checkpoint")
model, processor = load(str(checkpoint_dir), lazy=True)

# Parameters will materialize on first use

response = model.generate(processor("What is shown in this picture? <image>"))

```

### Loading with LoRA Adapters

Combine a base model with task-specific LoRA weights:

```python
model, processor = load(
    "mlx-community/Qwen2-VL-7B-Instruct",
    adapter_path="/home/user/checkpoints/lora-vision",
    lazy=False
)

result = model.generate(processor("Explain the chart data. <image>"))
print(processor.decode(result[0]))

```

### Inspecting Processor Components

Verify that the image processor and tokenizer loaded correctly:

```python
print(type(processor.image_processor))

# Output: <class 'mlx_vlm.models.qwen2_vl.processing_qwen2_vl.AutoImageProcessor'>

print(type(processor.tokenizer))

# Output: <class 'transformers.Qwen2TokenizerFast'>

```

## Summary

- The **`mlx_vlm.utils.load`** function serves as the single entry point for model and processor initialization, handling both local directories and Hugging Face repositories.
- The loading pipeline explicitly separates concerns into **`load_model`** (weights), **`load_image_processor`** (vision preprocessing), and **`load_processor`** (tokenization).
- Set **`lazy=True`** to defer weight materialization and reduce startup memory for large models, or use **`lazy=False`** (default) for immediate readiness.
- Pass **`adapter_path`** to automatically merge LoRA adapters into the base model during loading.
- Source implementations reside primarily in **[`mlx_vlm/utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/utils.py)** with specific line ranges for each sub-operation.

## Frequently Asked Questions

### How do I load a model from a local directory instead of Hugging Face?

Pass the absolute path to the directory containing your [`config.json`](https://github.com/Blaizzy/mlx-vlm/blob/main/config.json) and weight files as the first argument to `load`. For example: `load("/path/to/local/model", lazy=False)`. The function detects local paths automatically and skips remote resolution.

### What is the difference between the processor and the model in MLX-VLM?

The **model** object contains the neural network weights and forward logic, while the **processor** handles input preparation (tokenizing text, preprocessing images) and output decoding. The processor typically wraps a Hugging Face tokenizer and may include a custom image processor attached at `processor.image_processor`.

### Can I use lazy loading with LoRA adapters?

Yes. When you specify both `lazy=True` and `adapter_path`, the base model loads lazily while the LoRA weights are still applied correctly. However, note that the first forward pass will incur the combined overhead of materializing base parameters and merging adapter layers.

### Where does MLX-VLM store downloaded Hugging Face models?

The library uses standard Hugging Face Hub caching mechanisms. When you provide a repository ID like `"Qwen/Qwen2-VL-Chat"`, the `get_model_path` helper downloads files to the HF cache directory (typically `~/.cache/huggingface/hub`) and returns the local path for subsequent loading operations.