# How VisionFeatureCache Speeds Up Multi-Turn Conversations in MLX-VLM

> Discover how VisionFeatureCache accelerates multi-turn conversations in MLX-VLM by caching image features and avoiding repeated vision encoder passes.

- Repository: [Prince Canuma/mlx-vlm](https://github.com/Blaizzy/mlx-vlm)
- Tags: performance
- Published: 2026-04-05

---

**The `VisionFeatureCache` stores pre-computed image features from the vision encoder in an LRU cache, eliminating redundant forward passes when the same image is referenced across multiple conversation turns.**

Multi-turn conversations with vision-language models repeatedly process the same image, yet the vision tower computation dominates inference time. The `mlx-vlm` repository implements a `VisionFeatureCache` class in [`mlx_vlm/vision_cache.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/vision_cache.py) to memoize these expensive features, reducing latency from hundreds of milliseconds to near-instant dictionary lookups.

## Why Vision Encoding Is the Bottleneck in Multi-Turn Chats

Vision-language models process images through a vision encoder (the "vision tower") before the language model generates tokens. In `mlx-vlm`, this happens via `model.encode_image(pixel_values)`, which can take over 100ms per image depending on resolution and hardware.

Without caching, every user message that references an image—whether asking "What color is the car?" followed by "How many doors does it have?"—triggers a full re-encode. The `VisionFeatureCache` breaks this cycle by persisting the output of `encode_image()` across turns.

## How VisionFeatureCache Works

The cache implementation uses Python's `OrderedDict` to provide O(1) lookups while maintaining recency order for eviction.

### LRU Eviction Strategy

The cache maintains a bounded size (default 20 entries) to prevent unbounded memory growth. When `max_size` is exceeded, the least-recently used entry is discarded.

In [`mlx_vlm/vision_cache.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/vision_cache.py) (lines 31-68), the storage is declared as:

```python
self._cache: OrderedDict[str, mx.array]

```

The `put()` method handles eviction logic at lines 66-68:

```python
if len(self._cache) >= self.max_size:
    self._cache.popitem(last=False)  # Remove oldest

```

### Deterministic Key Generation

The cache generates stable keys for images using `_make_key()` (lines 35-50), supporting:

- **File paths or URLs**: Used directly as string keys
- **Composite inputs**: List of keys hashed together
- **PIL Image objects**: SHA-256 hash of `image.tobytes()`

This ensures that identical images produce identical cache keys regardless of how they are loaded.

### Fast Lookup with OrderedDict

The `get()` method (lines 52-58) implements the LRU pattern:

```python
def get(self, key: Union[str, Image.Image, List]) -> Optional[mx.array]:
    key = self._make_key(key)
    if key in self._cache:
        self._cache.move_to_end(key)  # Mark as recently used

        return self._cache[key]
    return None

```

On a cache hit, `move_to_end()` reorders the entry to prevent eviction, then returns the cached `mx.array` containing the vision features.

## Integration with the Generation Pipeline

The cache integrates directly into the generation loop in [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) (lines 55-65). Before encoding an image, the code checks for cached features:

```python
if vision_cache is not None and image is not None and pixel_values is not None:
    cached = vision_cache.get(image)
    if cached is not None:
        kwargs["cached_image_features"] = cached  # Reuse existing features

    else:
        features = model.encode_image(pixel_values)
        mx.eval(features)  # Materialize on device

        vision_cache.put(image, features)  # Store for future turns

        kwargs["cached_image_features"] = features

```

When `cached_image_features` is present in `kwargs`, the model skips the vision tower forward pass and uses the pre-computed tensors directly.

Each chat session maintains its own cache instance via `ModelState` in [`mlx_vlm/chat_ui.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/chat_ui.py) (line 44):

```python
self.vision_cache = VisionFeatureCache()

```

This isolation ensures that different conversations do not pollute each other's caches, and memory is reclaimed when the session ends.

## Practical Usage Examples

You can leverage the cache directly in custom inference loops:

```python
from mlx_vlm.vision_cache import VisionFeatureCache
import mlx.core as mx

vision_cache = VisionFeatureCache(max_size=5)

def encode_image_with_cache(model, image, pixel_values):
    # Try to fetch cached features

    cached = vision_cache.get(image)
    if cached is not None:
        return cached  # Fast path: skip encoder

    
    # Cache miss: run vision tower

    features = model.encode_image(pixel_values)
    mx.eval(features)  # Ensure computation completes

    vision_cache.put(image, features)  # Store for next turn

    return features

```

When using the high-level `generate()` API, caching happens automatically if you pass the `vision_cache` parameter:

```python
from mlx_vlm import generate, load

model, processor = load("mlx-community/llava-phi-3")
vision_cache = VisionFeatureCache(max_size=10)

# First turn: encodes and caches

response_1 = generate(model, processor, image="photo.jpg", 
                     prompt="Describe this image.", vision_cache=vision_cache)

# Second turn: instant retrieval from cache

response_2 = generate(model, processor, image="photo.jpg", 
                     prompt="What is the mood?", vision_cache=vision_cache)

```

## Performance Impact

The `VisionFeatureCache` delivers three critical optimizations for multi-turn dialogues:

- **Eliminates redundant computation**: Subsequent turns referencing the same image avoid the vision encoder entirely, reducing per-turn latency by 100ms or more.
- **Stabilizes memory footprint**: By reusing materialized `mx.array` features instead of recomputing intermediate tensors, GPU memory usage remains constant across turns.
- **Optimizes for conversational patterns**: The LRU policy keeps frequently referenced images (like a document being analyzed) in cache, matching typical user behavior where questions cluster around single images.

## Summary

- The `VisionFeatureCache` class in [`mlx_vlm/vision_cache.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/vision_cache.py) implements an LRU cache with a default capacity of 20 images.
- Keys are generated deterministically via `_make_key()`, supporting file paths, URLs, and PIL Image objects.
- Cache hits bypass `model.encode_image()`, injecting features directly via `kwargs["cached_image_features"]` in the generation pipeline.
- Each chat session maintains an isolated cache instance through `ModelState` in [`mlx_vlm/chat_ui.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/chat_ui.py).
- The cache reduces multi-turn conversation latency by avoiding repeated vision tower forward passes.

## Frequently Asked Questions

### How does the cache handle different image formats or preprocessing?

The cache key depends on the image source identifier (file path, URL, or byte hash) rather than the processed tensor. As implemented in `_make_key()`, two images with identical pixel content but different formats will generate different keys unless they produce the same hash. The actual preprocessing (`pixel_values`) occurs before the cache check in [`generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/generate.py), meaning the cache stores post-processed features ready for the language model.

### What happens when the cache reaches its maximum size?

When the number of cached images exceeds `max_size` (default 20), the cache automatically evicts the least-recently accessed entry using `OrderedDict.popitem(last=False)`. This LRU eviction occurs inside the `put()` method at lines 66-68 of [`mlx_vlm/vision_cache.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/vision_cache.py), ensuring that active conversations retain their working set of images while dormant entries are freed.

### Can I disable the vision cache if I need deterministic memory usage?

Yes. The cache is optional throughout the codebase. If you do not pass a `vision_cache` argument to `generate()`, or pass `None`, the code executes the standard `model.encode_image(pixel_values)` path on every turn. To explicitly disable caching in a chat application, simply omit the cache when initializing `ModelState` or set `vision_cache=None` in your generation calls.

### Does the cache persist across different model instances?

No. The `VisionFeatureCache` is instantiated per-session in `ModelState.__init__` (line 44 of [`mlx_vlm/chat_ui.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/chat_ui.py)). When you load a new model or restart the server, the cache is garbage collected. There is no disk persistence or shared memory between separate `load()` calls, preventing cache key collisions between different vision encoders with incompatible output dimensions.