How to Perform Video Understanding with MLX-VLM: Architecture and API Guide

MLX-VLM converts video files into sequences of image frames using OpenCV-based sampling, processes them through smart_resize() to maintain token budgets, and feeds them into vision-language models via the generate() function—available through both CLI and Python APIs.

Developers can perform video understanding with MLX-VLM to run temporal visual analysis on Apple Silicon hardware using the same efficient MLX backend designed for static images. The mlx-vlm repository by Blaizzy implements this capability by treating input clips as batched frame tensors, enabling models like Qwen2.5-VL to generate natural language descriptions from temporal visual content without requiring native video encoders.

Video Processing Architecture in MLX-VLM

The framework handles video inputs through a five-stage pipeline defined in mlx_vlm/video_generate.py and supporting modules. This architecture extracts frames using OpenCV, constrains resolution to prevent token overflow, and prepares tensors for the vision encoder.

Frame Extraction and Sampling Logic

In mlx_vlm/video_generate.py, the load_video() method (lines [199-235]) opens video files via cv2.VideoCapture and extracts metadata including total frame count and FPS. The helper smart_nframes() (lines [61-88]) determines sampling strategy by either respecting a user-provided nframes parameter or calculating nframes = total_frames / video_fps * fps, clamping the result between FPS_MIN_FRAMES and FPS_MAX_FRAMES to prevent memory overflow.

Selected frames undergo BGR-to-RGB conversion and are returned as a NumPy tensor of shape (T, C, H, W) representing time, channels, height, and width.

Smart Resizing and Token Budget Management

Before entering the model, frames pass through smart_resize() in video_generate.py (lines [66-94]), which enforces two critical constraints: dimensions must be multiples of IMAGE_FACTOR (28), and the total pixel count must remain within [MIN_PIXELS, MAX_PIXELS]. This guarantees that the resulting visual tokens never exceed the transformer's attention budget, preserving generation performance on hardware-limited devices.

Each frame is resized using bicubic interpolation via cv2.resize, then stacked and cast to float32 (lines [70-85]) before returning to the calling function.

Multimodal Input Preparation

The process_vision_info() function in mlx_vlm/utils.py (lines [29-45] and [55-71]) detects video entries in message payloads and routes them through fetch_video(). For models lacking native video support, the VideoFrameExtractor class samples frames at 1 FPS and injects them as static images into the conversation context. The processed video tensor is stored under the "videos" key in the input dictionary.

Generation Pipeline Integration

The stream_generate() function in mlx_vlm/generate.py (lines [126-147] and [311-334]) receives pixel_values (or pixel_values_videos) and passes them to the model via model.get_input_embeddings(). The subsequent generation loop—including KV-cache handling, sampling, and token prediction—remains identical to the image-only code path, demonstrating that the vision encoder processes video frames as a batch of independent images without architectural modifications.

Methods to Perform Video Understanding with MLX-VLM

Developers can perform video understanding through three primary interfaces, ranging from simple command-line usage to low-level tensor manipulation.

Command-Line Interface

The fastest method uses the built-in CLI module mlx_vlm.video_generate:

python -m mlx_vlm.video_generate \
    --model mlx-community/Qwen2.5-VL-7B-Instruct-4bit \
    --video path/to/clip.mp4 \
    --prompt "Summarize what is happening in this video." \
    --fps 2.0

This entry point (defined in video_generate.py lines [28-62] and [126-140]) handles argument parsing, model loading via load(), and message construction including the "type": "video" element required by the chat template.

Python API Integration

For programmatic control, import the generate() function directly:

from mlx_vlm import load, generate

model, processor = load(
    "mlx-community/Qwen2.5-VL-7B-Instruct-4bit"
)

result = generate(
    model,
    processor,
    prompt="What actions are the people performing?",
    video="https://example.com/my_video.mp4",
    fps=1.5,
    max_tokens=200,
    verbose=True,
)

print(result.text)

The generate() wrapper (lines [124-150] and [91-112] in generate.py) automatically invokes process_vision_info() to handle video tensors and manages the stream_generate execution loop.

Advanced Manual Tensor Control

For custom sampling strategies, access the low-level video utilities directly:

from mlx_vlm.utils import fetch_video

video_tensor, sample_fps = fetch_video(
    {
        "video": "my_clip.mov",
        "fps": 3,
        "max_pixels": 500_000,
    },
    image_factor=28,
    return_video_sample_fps=True,
)

This approach (implemented in video_generate.py lines [38-86]) returns the raw video tensor suitable for direct feeding into generate_step() when implementing custom generation loops.

Key Source Files

Understanding the implementation requires familiarity with these specific modules:

• mlx_vlm/video_generate.py: Contains load_video(), smart_nframes(), and smart_resize(); serves as the CLI entry point for video processing.
• mlx_vlm/utils.py: Implements process_vision_info() and fetch_video() for multimodal input extraction and video tensor preparation.
• mlx_vlm/generate.py: Houses stream_generate() and generate_step(), managing the actual token generation and KV-cache operations for video inputs.
• mlx_vlm/vision_cache.py: Optional utilities for reusing video embeddings across multiple generation turns.
• mlx_vlm/models/*/vision.py: Model-specific vision encoders (e.g., Qwen2.5-VL, Jina-VLM) that consume the preprocessed frame tensors.

Summary

• MLX-VLM treats video as sequences of image frames, sampled via OpenCV and processed through the same vision encoders as static images.
• Frame sampling is controlled by the fps parameter or explicit nframes, constrained by FPS_MIN_FRAMES and FPS_MAX_FRAMES in smart_nframes().
• Resolution constraints via smart_resize() ensure total pixels remain within MIN_PIXELS and MAX_PIXELS while maintaining multiples of IMAGE_FACTOR (28).
• Input preparation occurs through process_vision_info() in utils.py, with fallback to VideoFrameExtractor for models lacking native video support.
• Generation happens through stream_generate() in generate.py, handling video tensors via pixel_values_videos using identical logic to image generation.

Frequently Asked Questions

What is the maximum video length supported by MLX-VLM?

Video length is constrained by the VIDEO_TOTAL_PIXELS budget and available memory rather than a hard time limit. The smart_nframes() function in video_generate.py automatically clamps frame counts between FPS_MIN_FRAMES and FPS_MAX_FRAMES, while smart_resize() enforces pixel budgets. For long videos, increase the fps interval (e.g., sample at 0.5 FPS instead of 2.0) to reduce the total frame count processed by the vision encoder.

How does MLX-VLM handle videos for models without native video support?

For models that expect only image inputs, the VideoFrameExtractor class (utilized within process_vision_info() in utils.py) automatically extracts frames at 1 FPS and presents them as separate image messages in the conversation context. This allows any vision-language model in the MLX-VLM ecosystem to process video content without architectural changes.

Can I customize the frame sampling strategy beyond the FPS parameter?

Yes. Advanced users can bypass the high-level generate() function and call fetch_video() directly from mlx_vlm/utils, then manipulate the resulting tensor before passing it to generate_step(). The video dictionary accepts nframes (absolute count) or fps (rate-based), and max_pixels controls the per-frame resolution ceiling processed by smart_resize().

Why does MLX-VLM require OpenCV (cv2) for video processing?

OpenCV provides the VideoCapture backend that decodes video files into NumPy arrays and extracts frame-level metadata (original FPS, total frame count) required by load_video(). The library handles format variations and color space conversions (BGR to RGB) before the frames enter the MLX tensor pipeline, ensuring compatibility with standard video codecs while maintaining the (T, C, H, W) tensor shape expected by vision encoders.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}