How VisionFeatureCache Speeds Up Multi-Turn Conversations in MLX-VLM

The VisionFeatureCache stores pre-computed image features from the vision encoder in an LRU cache, eliminating redundant forward passes when the same image is referenced across multiple conversation turns.

Multi-turn conversations with vision-language models repeatedly process the same image, yet the vision tower computation dominates inference time. The mlx-vlm repository implements a VisionFeatureCache class in mlx_vlm/vision_cache.py to memoize these expensive features, reducing latency from hundreds of milliseconds to near-instant dictionary lookups.

Why Vision Encoding Is the Bottleneck in Multi-Turn Chats

Vision-language models process images through a vision encoder (the "vision tower") before the language model generates tokens. In mlx-vlm, this happens via model.encode_image(pixel_values), which can take over 100ms per image depending on resolution and hardware.

Without caching, every user message that references an image—whether asking "What color is the car?" followed by "How many doors does it have?"—triggers a full re-encode. The VisionFeatureCache breaks this cycle by persisting the output of encode_image() across turns.

How VisionFeatureCache Works

The cache implementation uses Python's OrderedDict to provide O(1) lookups while maintaining recency order for eviction.

LRU Eviction Strategy

The cache maintains a bounded size (default 20 entries) to prevent unbounded memory growth. When max_size is exceeded, the least-recently used entry is discarded.

In mlx_vlm/vision_cache.py (lines 31-68), the storage is declared as:

self._cache: OrderedDict[str, mx.array]

The put() method handles eviction logic at lines 66-68:

if len(self._cache) >= self.max_size:
    self._cache.popitem(last=False)  # Remove oldest

Deterministic Key Generation

The cache generates stable keys for images using _make_key() (lines 35-50), supporting:

• File paths or URLs: Used directly as string keys
• Composite inputs: List of keys hashed together
• PIL Image objects: SHA-256 hash of image.tobytes()

This ensures that identical images produce identical cache keys regardless of how they are loaded.

Fast Lookup with OrderedDict

The get() method (lines 52-58) implements the LRU pattern:

def get(self, key: Union[str, Image.Image, List]) -> Optional[mx.array]:
    key = self._make_key(key)
    if key in self._cache:
        self._cache.move_to_end(key)  # Mark as recently used

        return self._cache[key]
    return None

On a cache hit, move_to_end() reorders the entry to prevent eviction, then returns the cached mx.array containing the vision features.

Integration with the Generation Pipeline

The cache integrates directly into the generation loop in mlx_vlm/generate.py (lines 55-65). Before encoding an image, the code checks for cached features:

if vision_cache is not None and image is not None and pixel_values is not None:
    cached = vision_cache.get(image)
    if cached is not None:
        kwargs["cached_image_features"] = cached  # Reuse existing features

    else:
        features = model.encode_image(pixel_values)
        mx.eval(features)  # Materialize on device

        vision_cache.put(image, features)  # Store for future turns

        kwargs["cached_image_features"] = features

When cached_image_features is present in kwargs, the model skips the vision tower forward pass and uses the pre-computed tensors directly.

Each chat session maintains its own cache instance via ModelState in mlx_vlm/chat_ui.py (line 44):

self.vision_cache = VisionFeatureCache()

This isolation ensures that different conversations do not pollute each other's caches, and memory is reclaimed when the session ends.

Practical Usage Examples

You can leverage the cache directly in custom inference loops:

from mlx_vlm.vision_cache import VisionFeatureCache
import mlx.core as mx

vision_cache = VisionFeatureCache(max_size=5)

def encode_image_with_cache(model, image, pixel_values):
    # Try to fetch cached features

    cached = vision_cache.get(image)
    if cached is not None:
        return cached  # Fast path: skip encoder

    
    # Cache miss: run vision tower

    features = model.encode_image(pixel_values)
    mx.eval(features)  # Ensure computation completes

    vision_cache.put(image, features)  # Store for next turn

    return features

When using the high-level generate() API, caching happens automatically if you pass the vision_cache parameter:

from mlx_vlm import generate, load

model, processor = load("mlx-community/llava-phi-3")
vision_cache = VisionFeatureCache(max_size=10)

# First turn: encodes and caches

response_1 = generate(model, processor, image="photo.jpg", 
                     prompt="Describe this image.", vision_cache=vision_cache)

# Second turn: instant retrieval from cache

response_2 = generate(model, processor, image="photo.jpg", 
                     prompt="What is the mood?", vision_cache=vision_cache)

Performance Impact

The VisionFeatureCache delivers three critical optimizations for multi-turn dialogues:

• Eliminates redundant computation: Subsequent turns referencing the same image avoid the vision encoder entirely, reducing per-turn latency by 100ms or more.
• Stabilizes memory footprint: By reusing materialized mx.array features instead of recomputing intermediate tensors, GPU memory usage remains constant across turns.
• Optimizes for conversational patterns: The LRU policy keeps frequently referenced images (like a document being analyzed) in cache, matching typical user behavior where questions cluster around single images.

Summary

• The VisionFeatureCache class in mlx_vlm/vision_cache.py implements an LRU cache with a default capacity of 20 images.
• Keys are generated deterministically via _make_key(), supporting file paths, URLs, and PIL Image objects.
• Cache hits bypass model.encode_image(), injecting features directly via kwargs["cached_image_features"] in the generation pipeline.
• Each chat session maintains an isolated cache instance through ModelState in mlx_vlm/chat_ui.py.
• The cache reduces multi-turn conversation latency by avoiding repeated vision tower forward passes.

Frequently Asked Questions

How does the cache handle different image formats or preprocessing?

The cache key depends on the image source identifier (file path, URL, or byte hash) rather than the processed tensor. As implemented in _make_key(), two images with identical pixel content but different formats will generate different keys unless they produce the same hash. The actual preprocessing (pixel_values) occurs before the cache check in generate.py, meaning the cache stores post-processed features ready for the language model.

What happens when the cache reaches its maximum size?

When the number of cached images exceeds max_size (default 20), the cache automatically evicts the least-recently accessed entry using OrderedDict.popitem(last=False). This LRU eviction occurs inside the put() method at lines 66-68 of mlx_vlm/vision_cache.py, ensuring that active conversations retain their working set of images while dormant entries are freed.

Can I disable the vision cache if I need deterministic memory usage?

Yes. The cache is optional throughout the codebase. If you do not pass a vision_cache argument to generate(), or pass None, the code executes the standard model.encode_image(pixel_values) path on every turn. To explicitly disable caching in a chat application, simply omit the cache when initializing ModelState or set vision_cache=None in your generation calls.

Does the cache persist across different model instances?

No. The VisionFeatureCache is instantiated per-session in ModelState.__init__ (line 44 of mlx_vlm/chat_ui.py). When you load a new model or restart the server, the cache is garbage collected. There is no disk persistence or shared memory between separate load() calls, preventing cache key collisions between different vision encoders with incompatible output dimensions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}