# How to Perform Video Understanding with MLX-VLM: Architecture and API Guide

> Master video understanding with MLX-VLM. Learn the architecture and API guide to process video frames efficiently using Python or CLI. Explore the power of vision-language models.

- Repository: [Prince Canuma/mlx-vlm](https://github.com/Blaizzy/mlx-vlm)
- Tags: architecture
- Published: 2026-04-05

---

**MLX-VLM converts video files into sequences of image frames using OpenCV-based sampling, processes them through `smart_resize()` to maintain token budgets, and feeds them into vision-language models via the `generate()` function—available through both CLI and Python APIs.**

Developers can perform video understanding with MLX-VLM to run temporal visual analysis on Apple Silicon hardware using the same efficient MLX backend designed for static images. The `mlx-vlm` repository by Blaizzy implements this capability by treating input clips as batched frame tensors, enabling models like Qwen2.5-VL to generate natural language descriptions from temporal visual content without requiring native video encoders.

## Video Processing Architecture in MLX-VLM

The framework handles video inputs through a five-stage pipeline defined in [`mlx_vlm/video_generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/video_generate.py) and supporting modules. This architecture extracts frames using OpenCV, constrains resolution to prevent token overflow, and prepares tensors for the vision encoder.

### Frame Extraction and Sampling Logic

In [`mlx_vlm/video_generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/video_generate.py), the `load_video()` method (lines [199-235]) opens video files via `cv2.VideoCapture` and extracts metadata including total frame count and FPS. The helper `smart_nframes()` (lines [61-88]) determines sampling strategy by either respecting a user-provided `nframes` parameter or calculating `nframes = total_frames / video_fps * fps`, clamping the result between `FPS_MIN_FRAMES` and `FPS_MAX_FRAMES` to prevent memory overflow.

Selected frames undergo BGR-to-RGB conversion and are returned as a NumPy tensor of shape **(T, C, H, W)** representing time, channels, height, and width.

### Smart Resizing and Token Budget Management

Before entering the model, frames pass through `smart_resize()` in [`video_generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/video_generate.py) (lines [66-94]), which enforces two critical constraints: dimensions must be multiples of `IMAGE_FACTOR` (28), and the total pixel count must remain within `[MIN_PIXELS, MAX_PIXELS]`. This guarantees that the resulting visual tokens never exceed the transformer's attention budget, preserving generation performance on hardware-limited devices.

Each frame is resized using bicubic interpolation via `cv2.resize`, then stacked and cast to `float32` (lines [70-85]) before returning to the calling function.

### Multimodal Input Preparation

The `process_vision_info()` function in [`mlx_vlm/utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/utils.py) (lines [29-45] and [55-71]) detects video entries in message payloads and routes them through `fetch_video()`. For models lacking native video support, the `VideoFrameExtractor` class samples frames at 1 FPS and injects them as static images into the conversation context. The processed video tensor is stored under the `"videos"` key in the input dictionary.

### Generation Pipeline Integration

The `stream_generate()` function in [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) (lines [126-147] and [311-334]) receives `pixel_values` (or `pixel_values_videos`) and passes them to the model via `model.get_input_embeddings()`. The subsequent generation loop—including KV-cache handling, sampling, and token prediction—remains identical to the image-only code path, demonstrating that the vision encoder processes video frames as a batch of independent images without architectural modifications.

## Methods to Perform Video Understanding with MLX-VLM

Developers can perform video understanding through three primary interfaces, ranging from simple command-line usage to low-level tensor manipulation.

### Command-Line Interface

The fastest method uses the built-in CLI module `mlx_vlm.video_generate`:

```bash
python -m mlx_vlm.video_generate \
    --model mlx-community/Qwen2.5-VL-7B-Instruct-4bit \
    --video path/to/clip.mp4 \
    --prompt "Summarize what is happening in this video." \
    --fps 2.0

```

This entry point (defined in [`video_generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/video_generate.py) lines [28-62] and [126-140]) handles argument parsing, model loading via `load()`, and message construction including the `"type": "video"` element required by the chat template.

### Python API Integration

For programmatic control, import the `generate()` function directly:

```python
from mlx_vlm import load, generate

model, processor = load(
    "mlx-community/Qwen2.5-VL-7B-Instruct-4bit"
)

result = generate(
    model,
    processor,
    prompt="What actions are the people performing?",
    video="https://example.com/my_video.mp4",
    fps=1.5,
    max_tokens=200,
    verbose=True,
)

print(result.text)

```

The `generate()` wrapper (lines [124-150] and [91-112] in [`generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/generate.py)) automatically invokes `process_vision_info()` to handle video tensors and manages the `stream_generate` execution loop.

### Advanced Manual Tensor Control

For custom sampling strategies, access the low-level video utilities directly:

```python
from mlx_vlm.utils import fetch_video

video_tensor, sample_fps = fetch_video(
    {
        "video": "my_clip.mov",
        "fps": 3,
        "max_pixels": 500_000,
    },
    image_factor=28,
    return_video_sample_fps=True,
)

```

This approach (implemented in [`video_generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/video_generate.py) lines [38-86]) returns the raw video tensor suitable for direct feeding into `generate_step()` when implementing custom generation loops.

## Key Source Files

Understanding the implementation requires familiarity with these specific modules:

- **[`mlx_vlm/video_generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/video_generate.py)**: Contains `load_video()`, `smart_nframes()`, and `smart_resize()`; serves as the CLI entry point for video processing.
- **[`mlx_vlm/utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/utils.py)**: Implements `process_vision_info()` and `fetch_video()` for multimodal input extraction and video tensor preparation.
- **[`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py)**: Houses `stream_generate()` and `generate_step()`, managing the actual token generation and KV-cache operations for video inputs.
- **[`mlx_vlm/vision_cache.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/vision_cache.py)**: Optional utilities for reusing video embeddings across multiple generation turns.
- **`mlx_vlm/models/*/vision.py`**: Model-specific vision encoders (e.g., Qwen2.5-VL, Jina-VLM) that consume the preprocessed frame tensors.

## Summary

- **MLX-VLM treats video as sequences of image frames**, sampled via OpenCV and processed through the same vision encoders as static images.
- **Frame sampling** is controlled by the `fps` parameter or explicit `nframes`, constrained by `FPS_MIN_FRAMES` and `FPS_MAX_FRAMES` in `smart_nframes()`.
- **Resolution constraints** via `smart_resize()` ensure total pixels remain within `MIN_PIXELS` and `MAX_PIXELS` while maintaining multiples of `IMAGE_FACTOR` (28).
- **Input preparation** occurs through `process_vision_info()` in [`utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/utils.py), with fallback to `VideoFrameExtractor` for models lacking native video support.
- **Generation** happens through `stream_generate()` in [`generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/generate.py), handling video tensors via `pixel_values_videos` using identical logic to image generation.

## Frequently Asked Questions

### What is the maximum video length supported by MLX-VLM?

Video length is constrained by the `VIDEO_TOTAL_PIXELS` budget and available memory rather than a hard time limit. The `smart_nframes()` function in [`video_generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/video_generate.py) automatically clamps frame counts between `FPS_MIN_FRAMES` and `FPS_MAX_FRAMES`, while `smart_resize()` enforces pixel budgets. For long videos, increase the `fps` interval (e.g., sample at 0.5 FPS instead of 2.0) to reduce the total frame count processed by the vision encoder.

### How does MLX-VLM handle videos for models without native video support?

For models that expect only image inputs, the `VideoFrameExtractor` class (utilized within `process_vision_info()` in [`utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/utils.py)) automatically extracts frames at 1 FPS and presents them as separate image messages in the conversation context. This allows any vision-language model in the MLX-VLM ecosystem to process video content without architectural changes.

### Can I customize the frame sampling strategy beyond the FPS parameter?

Yes. Advanced users can bypass the high-level `generate()` function and call `fetch_video()` directly from `mlx_vlm/utils`, then manipulate the resulting tensor before passing it to `generate_step()`. The video dictionary accepts `nframes` (absolute count) or `fps` (rate-based), and `max_pixels` controls the per-frame resolution ceiling processed by `smart_resize()`.

### Why does MLX-VLM require OpenCV (cv2) for video processing?

OpenCV provides the `VideoCapture` backend that decodes video files into NumPy arrays and extracts frame-level metadata (original FPS, total frame count) required by `load_video()`. The library handles format variations and color space conversions (BGR to RGB) before the frames enter the MLX tensor pipeline, ensuring compatibility with standard video codecs while maintaining the (T, C, H, W) tensor shape expected by vision encoders.