Vision Feature Caching in MLX-VLM: How It Speeds Up Multi-Turn Image Conversations

Vision Feature Caching in MLX-VLM is an LRU-based memory cache that stores pre-computed vision embeddings to eliminate redundant forward passes through the vision tower when the same image is referenced across multiple conversation turns.

MLX-VLM processes visual inputs by running images through a vision tower and projecting the resulting embeddings into the language model's space via embed_vision. Because these computations are expensive—especially in multi-turn dialogues that revisit the same image—Blaizzy/mlx-vlm implements a deterministic caching layer to reuse projected features instantly.

How Vision Feature Caching Works in MLX-VLM

The caching mechanism intercepts vision feature computation after the initial projection, storing mx.array tensors in memory for rapid retrieval. This avoids recomputing embeddings for identical image inputs, significantly reducing latency in chat interfaces and API servers.

The VisionFeatureCache Class

The core implementation resides in [mlx_vlm/vision_cache.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/vision_cache.py). The VisionFeatureCache class wraps Python's OrderedDict to provide LRU semantics while storing MLX array objects.

from mlx_vlm.vision_cache import VisionFeatureCache
import mlx.core as mx

# Initialize with default max_size=20

cache = VisionFeatureCache()

# Store projected vision features

features = mx.ones((1, 280, 1536))
cache.put("cat.jpg", features)

The class exposes get, put, clear, __len__, and __contains__ methods for standard dictionary-like operations with bounded memory.

Cache Key Generation Strategy

Cache keys are deterministic identifiers derived from the image source via the _make_key method:

• String or Path objects: Used directly as keys (file paths or URLs)
• List of images: Keys are concatenated in order to support multi-image prompts
• PIL Images or array-like objects: SHA-256 hash of raw bytes with a pil: prefix (e.g., pil:a1b2c3...)

This ensures that identical image content or paths map to the same cache entry across sessions.

LRU Eviction Policy

When the cache reaches its max_size (default 20 entries), the least-recently-used item is automatically evicted using popitem(last=False). This bounded memory approach prevents unbounded growth during long-running server processes or extended chat sessions.

Integration Points Across the MLX-VLM Ecosystem

Vision Feature Caching is woven into the entire MLX-VLM stack, from the HTTP server to the interactive CLI.

Server Implementation

In [mlx_vlm/server.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py) (lines 88-90), each loaded model maintains its own VisionFeatureCache instance inside the model_cache dictionary:

# Per-model cache storage

model_cache["vision_cache"] = VisionFeatureCache()

When a model is unloaded or swapped, the cache is explicitly cleared to prevent stale embeddings from persisting across different vision-language models.

CLI and Chat Interfaces

The generation pipeline in [mlx_vlm/generate.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) (lines 94-96) instantiates a cache and injects it into stream_generate via the vision_cache keyword argument:

vision_cache = VisionFeatureCache()
stream_kwargs = {"vision_cache": vision_cache, ...}
for chunk in stream_generate(..., **stream_kwargs):
    ...

Similarly, [mlx_vlm/chat_ui.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/chat_ui.py) (line 44) maintains a global state.vision_cache that persists across messages but clears when a new model is loaded.

Model-Level Integration

Every vision-capable model inspects the cached_image_features keyword argument within its get_input_embeddings implementation. If a cache hit occurs, the model bypasses the vision tower entirely. The test suite in [tests/test_vision_cache.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/tests/test_vision_cache.py) verifies this keyword is present across supported model architectures.

Practical Code Examples

Basic Cache Operations

Direct instantiation allows fine-grained control over cache behavior:

from mlx_vlm.vision_cache import VisionFeatureCache
import mlx.core as mx

cache = VisionFeatureCache(max_size=50)

# Simulate vision tower output

features = mx.random.normal((1, 576, 2048))
cache.put("/path/to/image.png", features)

# Retrieval (cache hit)

cached_features = cache.get("/path/to/image.png")
assert cached_features is not None

Using Vision Cache in Generation Workflows

When using the CLI chat mode, the cache is automatically managed:

python -m mlx_vlm generate \
    --model qnguyen3/nanoLLaVA \
    --chat \
    --image screenshot.png

Programmatically, pass the cache to stream_generate:

from mlx_vlm import stream_generate
from mlx_vlm.vision_cache import VisionFeatureCache

vision_cache = VisionFeatureCache()
for chunk in stream_generate(
    model, 
    processor, 
    image="screenshot.png",
    prompt="Describe this image.",
    vision_cache=vision_cache
):
    print(chunk.text, end="", flush=True)

Managing Cache Lifecycle

Explicitly clear the cache when switching contexts to prevent memory leaks:

# When unloading a model (as implemented in server.py)

if "vision_cache" in model_cache:
    model_cache["vision_cache"].clear()
    unload_model_sync()

Multi-Image Caching

For prompts containing multiple images, concatenate identifiers:

# Store features for image batches

cache.put(["chart.png", "table.png"], batch_features)

# Retrieval requires identical order

assert cache.get(["chart.png", "table.png"]) is not None

Performance Benefits

• Latency Reduction: Eliminates redundant vision tower forward passes, cutting response time in multi-turn conversations referencing the same image
• Memory Efficiency: Bounded max_size prevents unbounded memory growth; automatic LRU eviction handles cache management
• Deterministic Behavior: Stable key generation guarantees cache hits for identical file paths, URLs, or image byte content across process restarts

Summary

• Vision Feature Caching stores projected vision embeddings in an LRU cache to avoid recomputing expensive vision tower outputs
• The VisionFeatureCache class in mlx_vlm/vision_cache.py provides get/put operations with automatic eviction at the default limit of 20 entries
• Keys are generated deterministically from file paths, URLs, or SHA-256 hashes of PIL image bytes
• The cache integrates seamlessly across the server (server.py), CLI (generate.py), and web UI (chat_ui.py) components
• Memory usage remains bounded through LRU eviction, while multi-turn chat performance improves significantly by reusing cached embeddings

Frequently Asked Questions

What is the default cache size limit in MLX-VLM?

The default max_size is 20 entries. You can customize this when instantiating VisionFeatureCache(max_size=100) for workloads involving many repeated images, though higher values increase memory consumption proportionally.

How does MLX-VLM generate cache keys for images?

The _make_key method generates deterministic keys based on input type: file paths and URLs are used as-is; PIL images and arrays are hashed using SHA-256 and prefixed with pil:; lists of images are concatenated in order. This ensures identical inputs always resolve to the same cache slot.

When is the vision cache cleared in the MLX-VLM server?

The cache clears automatically when a model is unloaded or swapped, as implemented in server.py where model_cache["vision_cache"].clear() is called before loading a new model. This prevents cross-model contamination and frees memory for the next vision-language model.

Can Vision Feature Caching be used with multiple images in one prompt?

Yes. Pass a list of image paths or identifiers to cache.put(), which concatenates them into a single key. Retrieval requires the same list order and content. This is particularly useful for visual question answering tasks comparing multiple images or analyzing image sequences.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}