# How to Integrate MLX-VLM into Custom Applications: A Complete Developer's Guide

> Integrate MLX-VLM into custom applications with this developer's guide. Learn to load models, prepare inputs, and generate outputs for Vision-Language inference on Apple Silicon using Python.

- Repository: [Prince Canuma/mlx-vlm](https://github.com/Blaizzy/mlx-vlm)
- Tags: how-to-guide
- Published: 2026-04-05

---

**Integrate MLX-VLM into custom applications by loading models with `mlx_vlm.utils.load()`, preparing inputs via `prepare_inputs()`, and generating outputs through `stream_generate()` or `generate()`—enabling Vision-Language inference on Apple Silicon with just a few lines of Python.**

MLX-VLM is a lightweight, Apple-silicon-optimized library that runs Vision-Language and Omni models directly on macOS GPU/CPU via the MLX framework. Because the public API is intentionally minimal—centered around three core functions in [`mlx_vlm/utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/utils.py) and [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py)—you can embed multimodal AI capabilities into Python scripts, CLI tools, web services, or desktop applications without managing complex dependencies.

## Understanding the MLX-VLM Architecture

The library implements a **three-stage inference pipeline** that remains consistent across image, audio, and video modalities:

1. **Model & Processor Loading** – The `load()` function in [`mlx_vlm/utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/utils.py) downloads weights from HuggingFace (or loads local checkpoints), instantiates the MLX model as an `nn.Module`, and returns a processor wrapper that handles tokenization and multimodal encoding.

2. **Input Preparation** – `prepare_inputs()` converts raw file paths, URLs, or `PIL.Image` objects into the tensor formats expected by the vision tower, managing resizing, padding, and special token insertion.

3. **Generation** – `stream_generate()` in [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) runs token-by-token inference with optional KV-cache quantization and TurboQuant compression, yielding `GenerationResult` objects. The convenience wrapper `generate()` simply consumes this stream and returns the final text.

This architecture allows you to intercept and customize any stage—whether caching vision features between chat turns or injecting custom stopping criteria—while the core engine handles MLX-specific optimizations automatically.

## Step-by-Step Integration Guide

### 1. Install and Load the Model

Begin by installing the package and loading a quantized model from the MLX community or a local path:

```python
from mlx_vlm import load

# Auto-downloads from HuggingFace if not cached locally

model, processor = load("mlx-community/Qwen2-VL-2B-Instruct-4bit")

```

The `load()` function (located in [`mlx_vlm/utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/utils.py) lines 60-84) returns a tuple containing the MLX model and a HuggingFace-compatible processor. For configuration-aware prompting, fetch the model config separately:

```python
from mlx_vlm.utils import load_config

config = load_config("mlx-community/Qwen2-VL-2B-Instruct-4bit")

```

### 2. Prepare Multimodal Inputs

Use `apply_chat_template()` from [`mlx_vlm/prompt_utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/prompt_utils.py) to format prompts with correct image/audio tokens, then pass raw media files directly to the generation functions:

```python
from mlx_vlm.prompt_utils import apply_chat_template

prompt = "Describe what is happening in this picture."
formatted_prompt = apply_chat_template(
    processor, 
    config, 
    prompt, 
    num_images=1
)

# Supports local paths, URLs, or PIL.Image objects

image = ["https://example.com/photo.jpg"]

```

Under the hood, `prepare_inputs()` (in [`mlx_vlm/utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/utils.py) lines 89-135) handles tensor conversion, ensuring images are resized to the model's expected patch grid and audio is resampled to the correct sampling rate.

### 3. Generate Outputs

**Blocking generation** (simplest for scripts):

```python
from mlx_vlm import generate

result = generate(
    model, 
    processor, 
    formatted_prompt, 
    image=image, 
    verbose=True
)
print(result.text)

```

**Streaming generation** (for real-time UIs):

```python
from mlx_vlm import stream_generate

for step in stream_generate(
    model, 
    processor, 
    formatted_prompt, 
    image=image, 
    max_tokens=256
):
    print(step.text, end="", flush=True)

```

The `stream_generate()` function (in [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) lines 75-84) yields intermediate results immediately, enabling progressive display in web interfaces or command-line tools.

## Advanced Integration Patterns

### Multi-Turn Conversations with VisionFeatureCache

For chat applications where users ask multiple questions about the same image, instantiate `VisionFeatureCache` to avoid recomputing expensive vision tower embeddings:

```python
from mlx_vlm import load, stream_generate
from mlx_vlm.vision_cache import VisionFeatureCache

model, processor = load("mlx-community/Qwen2-VL-2B-Instruct-4bit")
cache = VisionFeatureCache(max_size=8)  # LRU cache for up to 8 images

# First turn: encodes and caches vision features

for chunk in stream_generate(
    model, processor, 
    "Describe this scene.", 
    image="park.jpg",
    vision_cache=cache
):
    print(chunk.text, end="")

# Second turn: retrieves cached embeddings instantly

for chunk in stream_generate(
    model, processor, 
    "What colors are dominant?", 
    image="park.jpg",
    vision_cache=cache
):
    print(chunk.text, end="")

```

The `VisionFeatureCache` class (defined in [`mlx_vlm/vision_cache.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/vision_cache.py)) stores the output of `model.encode_image(pixel_values)` keyed by content hash, reducing latency by skipping the vision forward pass on cache hits.

### Audio-Enabled Applications

MLX-VLM supports Omni models that process audio alongside text. Load audio files using the same API pattern:

```python
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("mlx-community/gemma-3n-E2B-it-4bit")

audio_files = ["speech.wav"]
prompt = apply_chat_template(
    processor, 
    model.config, 
    "Summarize what you hear.", 
    num_audios=1
)

result = generate(
    model, processor, 
    prompt, 
    audio=audio_files,
    verbose=True
)

```

Audio preprocessing is handled internally by `load_audio()` and `resample_audio()` functions in [`mlx_vlm/utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/utils.py).

### Video Processing Workflows

For video understanding, the library extracts frames and processes them as sequences of images. Use the video-aware generation pipeline:

```python
from mlx_vlm import load, generate
from mlx_vlm.video_generate import process_vision_info

model, processor = load("mlx-community/Qwen2-VL-2B-Instruct-4bit")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "video", "video": "sample.mp4", "max_pixels": 50176, "fps": 1.0},
            {"type": "text", "text": "Provide a brief description of the video."},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs, _ = process_vision_info(messages, return_video=True)

result = generate(
    model, processor,
    prompt=text,
    image=image_inputs,
    video=video_inputs,
    verbose=True
)

```

The heavy lifting for frame extraction and resizing occurs in [`mlx_vlm/video_generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/video_generate.py), which intelligently reduces frame rates and resolution to fit model constraints.

### Building REST APIs with FastAPI

Deploy MLX-VLM as a microservice by wrapping the streaming generator in an async endpoint:

```python
import uvicorn
from fastapi import FastAPI
from mlx_vlm import load, stream_generate

app = FastAPI()
model, processor = load("mlx-community/Qwen2-VL-2B-Instruct-4bit")

@app.post("/v1/chat/completions")
def chat_completion(messages: list[dict], max_tokens: int = 256):
    prompt = processor.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
    )
    
    # Extract image URL from OpenAI-style message format

    image = next(
        (c["image_url"] for m in messages for c in m["content"] 
         if c.get("type") == "input_image"), None
    )
    
    text = "".join([
        step.text for step in stream_generate(
            model, processor, prompt, 
            image=image, 
            max_tokens=max_tokens
        )
    ])
    
    return {"choices": [{"message": {"content": text}}]}

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8080)

```

This pattern maintains the `VisionFeatureCache` across requests when implemented as a singleton, ensuring repeated queries against the same media are served from RAM rather than recomputed.

## Key Configuration Options

When integrating into production applications, consider these performance optimizations available in the generation functions:

- **KV-Cache Quantization**: Pass `kv_bits=4` and `kv_quant_scheme="uniform"` (or `"turboquant"`) to `stream_generate()` to compress attention memory, enabling longer context windows on limited RAM.
- **Thinking Budget**: For reasoning models that emit `<think>` blocks, use `thinking_budget=1024` to force exit from the thinking phase after a specified token limit via `ThinkingBudgetCriteria` in [`mlx_vlm/utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/utils.py).
- **Batch Generation**: Use `batch_generate()` in [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) for processing multiple prompts efficiently by grouping variable-size images to minimize padding waste.

## Summary

- **Load models** with `mlx_vlm.utils.load()` to get a model-processor pair compatible with HuggingFace conventions.
- **Prepare inputs** using `apply_chat_template()` for chat formatting and `prepare_inputs()` for tensor conversion.
- **Generate text** via `stream_generate()` for real-time applications or `generate()` for blocking calls.
- **Optimize performance** with `VisionFeatureCache` for multi-turn chats and KV-cache quantization for long contexts.
- **Extend to audio/video** using the same API patterns, with preprocessing handled in [`mlx_vlm/utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/utils.py) and [`mlx_vlm/video_generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/video_generate.py).

## Frequently Asked Questions

### How do I cache image embeddings across multiple API calls?

Instantiate `VisionFeatureCache` from [`mlx_vlm/vision_cache.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/vision_cache.py) and pass it to `stream_generate()` via the `vision_cache` parameter. The cache stores encoded image tensors keyed by file hash, eliminating redundant vision tower computation when the same image appears in subsequent requests.

### Can I run MLX-VLM on Intel Macs or Linux?

MLX-VLM is optimized for Apple Silicon (M1/M2/M3/M4) using the MLX framework. While MLX itself is Apple-specific, the library's Python API will import on other platforms but will fail when attempting GPU/CPU acceleration without MLX backend support. For cross-platform deployment, consider containerizing on macOS or using alternative VLM frameworks for Linux.

### What is the difference between `generate()` and `stream_generate()`?

`generate()` is a synchronous wrapper that accumulates all tokens from `stream_generate()` and returns a single `GenerationResult` object. Use `stream_generate()` when building interactive applications that need to display partial outputs immediately, or when implementing custom stopping logic that interrupts generation mid-stream.

### How do I handle video files with different frame rates?

The video processing pipeline in [`mlx_vlm/video_generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/video_generate.py) automatically handles frame extraction with configurable `fps` and `max_pixels` parameters. Pass these in the message dictionary when building your prompt, and the library will resample video frames to match the model's expected input dimensions while preserving temporal information.