How to Integrate MLX-VLM into Custom Applications: A Complete Developer's Guide

Integrate MLX-VLM into custom applications by loading models with mlx_vlm.utils.load(), preparing inputs via prepare_inputs(), and generating outputs through stream_generate() or generate()—enabling Vision-Language inference on Apple Silicon with just a few lines of Python.

MLX-VLM is a lightweight, Apple-silicon-optimized library that runs Vision-Language and Omni models directly on macOS GPU/CPU via the MLX framework. Because the public API is intentionally minimal—centered around three core functions in mlx_vlm/utils.py and mlx_vlm/generate.py—you can embed multimodal AI capabilities into Python scripts, CLI tools, web services, or desktop applications without managing complex dependencies.

Understanding the MLX-VLM Architecture

The library implements a three-stage inference pipeline that remains consistent across image, audio, and video modalities:

1. Model & Processor Loading – The load() function in mlx_vlm/utils.py downloads weights from HuggingFace (or loads local checkpoints), instantiates the MLX model as an nn.Module, and returns a processor wrapper that handles tokenization and multimodal encoding.

2. Input Preparation – prepare_inputs() converts raw file paths, URLs, or PIL.Image objects into the tensor formats expected by the vision tower, managing resizing, padding, and special token insertion.

3. Generation – stream_generate() in mlx_vlm/generate.py runs token-by-token inference with optional KV-cache quantization and TurboQuant compression, yielding GenerationResult objects. The convenience wrapper generate() simply consumes this stream and returns the final text.

This architecture allows you to intercept and customize any stage—whether caching vision features between chat turns or injecting custom stopping criteria—while the core engine handles MLX-specific optimizations automatically.

Step-by-Step Integration Guide

1. Install and Load the Model

Begin by installing the package and loading a quantized model from the MLX community or a local path:

from mlx_vlm import load

# Auto-downloads from HuggingFace if not cached locally

model, processor = load("mlx-community/Qwen2-VL-2B-Instruct-4bit")

The load() function (located in mlx_vlm/utils.py lines 60-84) returns a tuple containing the MLX model and a HuggingFace-compatible processor. For configuration-aware prompting, fetch the model config separately:

from mlx_vlm.utils import load_config

config = load_config("mlx-community/Qwen2-VL-2B-Instruct-4bit")

2. Prepare Multimodal Inputs

Use apply_chat_template() from mlx_vlm/prompt_utils.py to format prompts with correct image/audio tokens, then pass raw media files directly to the generation functions:

from mlx_vlm.prompt_utils import apply_chat_template

prompt = "Describe what is happening in this picture."
formatted_prompt = apply_chat_template(
    processor, 
    config, 
    prompt, 
    num_images=1
)

# Supports local paths, URLs, or PIL.Image objects

image = ["https://example.com/photo.jpg"]

Under the hood, prepare_inputs() (in mlx_vlm/utils.py lines 89-135) handles tensor conversion, ensuring images are resized to the model's expected patch grid and audio is resampled to the correct sampling rate.

3. Generate Outputs

Blocking generation (simplest for scripts):

from mlx_vlm import generate

result = generate(
    model, 
    processor, 
    formatted_prompt, 
    image=image, 
    verbose=True
)
print(result.text)

Streaming generation (for real-time UIs):

from mlx_vlm import stream_generate

for step in stream_generate(
    model, 
    processor, 
    formatted_prompt, 
    image=image, 
    max_tokens=256
):
    print(step.text, end="", flush=True)

The stream_generate() function (in mlx_vlm/generate.py lines 75-84) yields intermediate results immediately, enabling progressive display in web interfaces or command-line tools.

Advanced Integration Patterns

Multi-Turn Conversations with VisionFeatureCache

For chat applications where users ask multiple questions about the same image, instantiate VisionFeatureCache to avoid recomputing expensive vision tower embeddings:

from mlx_vlm import load, stream_generate
from mlx_vlm.vision_cache import VisionFeatureCache

model, processor = load("mlx-community/Qwen2-VL-2B-Instruct-4bit")
cache = VisionFeatureCache(max_size=8)  # LRU cache for up to 8 images

# First turn: encodes and caches vision features

for chunk in stream_generate(
    model, processor, 
    "Describe this scene.", 
    image="park.jpg",
    vision_cache=cache
):
    print(chunk.text, end="")

# Second turn: retrieves cached embeddings instantly

for chunk in stream_generate(
    model, processor, 
    "What colors are dominant?", 
    image="park.jpg",
    vision_cache=cache
):
    print(chunk.text, end="")

The VisionFeatureCache class (defined in mlx_vlm/vision_cache.py) stores the output of model.encode_image(pixel_values) keyed by content hash, reducing latency by skipping the vision forward pass on cache hits.

Audio-Enabled Applications

MLX-VLM supports Omni models that process audio alongside text. Load audio files using the same API pattern:

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("mlx-community/gemma-3n-E2B-it-4bit")

audio_files = ["speech.wav"]
prompt = apply_chat_template(
    processor, 
    model.config, 
    "Summarize what you hear.", 
    num_audios=1
)

result = generate(
    model, processor, 
    prompt, 
    audio=audio_files,
    verbose=True
)

Audio preprocessing is handled internally by load_audio() and resample_audio() functions in mlx_vlm/utils.py.

Video Processing Workflows

For video understanding, the library extracts frames and processes them as sequences of images. Use the video-aware generation pipeline:

from mlx_vlm import load, generate
from mlx_vlm.video_generate import process_vision_info

model, processor = load("mlx-community/Qwen2-VL-2B-Instruct-4bit")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "video", "video": "sample.mp4", "max_pixels": 50176, "fps": 1.0},
            {"type": "text", "text": "Provide a brief description of the video."},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs, _ = process_vision_info(messages, return_video=True)

result = generate(
    model, processor,
    prompt=text,
    image=image_inputs,
    video=video_inputs,
    verbose=True
)

The heavy lifting for frame extraction and resizing occurs in mlx_vlm/video_generate.py, which intelligently reduces frame rates and resolution to fit model constraints.

Building REST APIs with FastAPI

Deploy MLX-VLM as a microservice by wrapping the streaming generator in an async endpoint:

import uvicorn
from fastapi import FastAPI
from mlx_vlm import load, stream_generate

app = FastAPI()
model, processor = load("mlx-community/Qwen2-VL-2B-Instruct-4bit")

@app.post("/v1/chat/completions")
def chat_completion(messages: list[dict], max_tokens: int = 256):
    prompt = processor.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
    )
    
    # Extract image URL from OpenAI-style message format

    image = next(
        (c["image_url"] for m in messages for c in m["content"] 
         if c.get("type") == "input_image"), None
    )
    
    text = "".join([
        step.text for step in stream_generate(
            model, processor, prompt, 
            image=image, 
            max_tokens=max_tokens
        )
    ])
    
    return {"choices": [{"message": {"content": text}}]}

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8080)

This pattern maintains the VisionFeatureCache across requests when implemented as a singleton, ensuring repeated queries against the same media are served from RAM rather than recomputed.

Key Configuration Options

When integrating into production applications, consider these performance optimizations available in the generation functions:

• KV-Cache Quantization: Pass kv_bits=4 and kv_quant_scheme="uniform" (or "turboquant") to stream_generate() to compress attention memory, enabling longer context windows on limited RAM.
• Thinking Budget: For reasoning models that emit <think> blocks, use thinking_budget=1024 to force exit from the thinking phase after a specified token limit via ThinkingBudgetCriteria in mlx_vlm/utils.py.
• Batch Generation: Use batch_generate() in mlx_vlm/generate.py for processing multiple prompts efficiently by grouping variable-size images to minimize padding waste.

Summary

• Load models with mlx_vlm.utils.load() to get a model-processor pair compatible with HuggingFace conventions.
• Prepare inputs using apply_chat_template() for chat formatting and prepare_inputs() for tensor conversion.
• Generate text via stream_generate() for real-time applications or generate() for blocking calls.
• Optimize performance with VisionFeatureCache for multi-turn chats and KV-cache quantization for long contexts.
• Extend to audio/video using the same API patterns, with preprocessing handled in mlx_vlm/utils.py and mlx_vlm/video_generate.py.

Frequently Asked Questions

How do I cache image embeddings across multiple API calls?

Instantiate VisionFeatureCache from mlx_vlm/vision_cache.py and pass it to stream_generate() via the vision_cache parameter. The cache stores encoded image tensors keyed by file hash, eliminating redundant vision tower computation when the same image appears in subsequent requests.

Can I run MLX-VLM on Intel Macs or Linux?

MLX-VLM is optimized for Apple Silicon (M1/M2/M3/M4) using the MLX framework. While MLX itself is Apple-specific, the library's Python API will import on other platforms but will fail when attempting GPU/CPU acceleration without MLX backend support. For cross-platform deployment, consider containerizing on macOS or using alternative VLM frameworks for Linux.

What is the difference between generate() and stream_generate()?

generate() is a synchronous wrapper that accumulates all tokens from stream_generate() and returns a single GenerationResult object. Use stream_generate() when building interactive applications that need to display partial outputs immediately, or when implementing custom stopping logic that interrupts generation mid-stream.

How do I handle video files with different frame rates?

The video processing pipeline in mlx_vlm/video_generate.py automatically handles frame extraction with configurable fps and max_pixels parameters. Pass these in the message dictionary when building your prompt, and the library will resample video frames to match the model's expected input dimensions while preserving temporal information.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}