# Vision Feature Caching in MLX-VLM: How It Speeds Up Multi-Turn Image Conversations

> Discover Vision Feature Caching in MLX-VLM. This LRU cache stores pre-computed vision embeddings to speed up multi-turn conversations by preventing redundant image processing.

- Repository: [Prince Canuma/mlx-vlm](https://github.com/Blaizzy/mlx-vlm)
- Tags: internals
- Published: 2026-04-05

---

**Vision Feature Caching in MLX-VLM is an LRU-based memory cache that stores pre-computed vision embeddings to eliminate redundant forward passes through the vision tower when the same image is referenced across multiple conversation turns.**

MLX-VLM processes visual inputs by running images through a vision tower and projecting the resulting embeddings into the language model's space via `embed_vision`. Because these computations are expensive—especially in multi-turn dialogues that revisit the same image—Blaizzy/mlx-vlm implements a deterministic caching layer to reuse projected features instantly.

## How Vision Feature Caching Works in MLX-VLM

The caching mechanism intercepts vision feature computation after the initial projection, storing `mx.array` tensors in memory for rapid retrieval. This avoids recomputing embeddings for identical image inputs, significantly reducing latency in chat interfaces and API servers.

### The VisionFeatureCache Class

The core implementation resides in [[`mlx_vlm/vision_cache.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/vision_cache.py)](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/vision_cache.py). The `VisionFeatureCache` class wraps Python's `OrderedDict` to provide LRU semantics while storing MLX array objects.

```python
from mlx_vlm.vision_cache import VisionFeatureCache
import mlx.core as mx

# Initialize with default max_size=20

cache = VisionFeatureCache()

# Store projected vision features

features = mx.ones((1, 280, 1536))
cache.put("cat.jpg", features)

```

The class exposes `get`, `put`, `clear`, `__len__`, and `__contains__` methods for standard dictionary-like operations with bounded memory.

### Cache Key Generation Strategy

Cache keys are deterministic identifiers derived from the image source via the `_make_key` method:

- **String or Path objects**: Used directly as keys (file paths or URLs)
- **List of images**: Keys are concatenated in order to support multi-image prompts
- **PIL Images or array-like objects**: SHA-256 hash of raw bytes with a `pil:` prefix (e.g., `pil:a1b2c3...`)

This ensures that identical image content or paths map to the same cache entry across sessions.

### LRU Eviction Policy

When the cache reaches its `max_size` (default **20 entries**), the least-recently-used item is automatically evicted using `popitem(last=False)`. This bounded memory approach prevents unbounded growth during long-running server processes or extended chat sessions.

## Integration Points Across the MLX-VLM Ecosystem

Vision Feature Caching is woven into the entire MLX-VLM stack, from the HTTP server to the interactive CLI.

### Server Implementation

In [[`mlx_vlm/server.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py)](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py) (lines 88-90), each loaded model maintains its own `VisionFeatureCache` instance inside the `model_cache` dictionary:

```python

# Per-model cache storage

model_cache["vision_cache"] = VisionFeatureCache()

```

When a model is unloaded or swapped, the cache is explicitly cleared to prevent stale embeddings from persisting across different vision-language models.

### CLI and Chat Interfaces

The generation pipeline in [[`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py)](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) (lines 94-96) instantiates a cache and injects it into `stream_generate` via the `vision_cache` keyword argument:

```python
vision_cache = VisionFeatureCache()
stream_kwargs = {"vision_cache": vision_cache, ...}
for chunk in stream_generate(..., **stream_kwargs):
    ...

```

Similarly, [[`mlx_vlm/chat_ui.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/chat_ui.py)](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/chat_ui.py) (line 44) maintains a global `state.vision_cache` that persists across messages but clears when a new model is loaded.

### Model-Level Integration

Every vision-capable model inspects the `cached_image_features` keyword argument within its `get_input_embeddings` implementation. If a cache hit occurs, the model bypasses the vision tower entirely. The test suite in [[`tests/test_vision_cache.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/tests/test_vision_cache.py)](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/tests/test_vision_cache.py) verifies this keyword is present across supported model architectures.

## Practical Code Examples

### Basic Cache Operations

Direct instantiation allows fine-grained control over cache behavior:

```python
from mlx_vlm.vision_cache import VisionFeatureCache
import mlx.core as mx

cache = VisionFeatureCache(max_size=50)

# Simulate vision tower output

features = mx.random.normal((1, 576, 2048))
cache.put("/path/to/image.png", features)

# Retrieval (cache hit)

cached_features = cache.get("/path/to/image.png")
assert cached_features is not None

```

### Using Vision Cache in Generation Workflows

When using the CLI chat mode, the cache is automatically managed:

```bash
python -m mlx_vlm generate \
    --model qnguyen3/nanoLLaVA \
    --chat \
    --image screenshot.png

```

Programmatically, pass the cache to `stream_generate`:

```python
from mlx_vlm import stream_generate
from mlx_vlm.vision_cache import VisionFeatureCache

vision_cache = VisionFeatureCache()
for chunk in stream_generate(
    model, 
    processor, 
    image="screenshot.png",
    prompt="Describe this image.",
    vision_cache=vision_cache
):
    print(chunk.text, end="", flush=True)

```

### Managing Cache Lifecycle

Explicitly clear the cache when switching contexts to prevent memory leaks:

```python

# When unloading a model (as implemented in server.py)

if "vision_cache" in model_cache:
    model_cache["vision_cache"].clear()
    unload_model_sync()

```

### Multi-Image Caching

For prompts containing multiple images, concatenate identifiers:

```python

# Store features for image batches

cache.put(["chart.png", "table.png"], batch_features)

# Retrieval requires identical order

assert cache.get(["chart.png", "table.png"]) is not None

```

## Performance Benefits

- **Latency Reduction**: Eliminates redundant vision tower forward passes, cutting response time in multi-turn conversations referencing the same image
- **Memory Efficiency**: Bounded `max_size` prevents unbounded memory growth; automatic LRU eviction handles cache management
- **Deterministic Behavior**: Stable key generation guarantees cache hits for identical file paths, URLs, or image byte content across process restarts

## Summary

- Vision Feature Caching stores projected vision embeddings in an LRU cache to avoid recomputing expensive vision tower outputs
- The `VisionFeatureCache` class in [`mlx_vlm/vision_cache.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/vision_cache.py) provides `get`/`put` operations with automatic eviction at the default limit of 20 entries
- Keys are generated deterministically from file paths, URLs, or SHA-256 hashes of PIL image bytes
- The cache integrates seamlessly across the server ([`server.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/server.py)), CLI ([`generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/generate.py)), and web UI ([`chat_ui.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/chat_ui.py)) components
- Memory usage remains bounded through LRU eviction, while multi-turn chat performance improves significantly by reusing cached embeddings

## Frequently Asked Questions

### What is the default cache size limit in MLX-VLM?

The default `max_size` is **20 entries**. You can customize this when instantiating `VisionFeatureCache(max_size=100)` for workloads involving many repeated images, though higher values increase memory consumption proportionally.

### How does MLX-VLM generate cache keys for images?

The `_make_key` method generates deterministic keys based on input type: file paths and URLs are used as-is; PIL images and arrays are hashed using SHA-256 and prefixed with `pil:`; lists of images are concatenated in order. This ensures identical inputs always resolve to the same cache slot.

### When is the vision cache cleared in the MLX-VLM server?

The cache clears automatically when a model is unloaded or swapped, as implemented in [`server.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/server.py) where `model_cache["vision_cache"].clear()` is called before loading a new model. This prevents cross-model contamination and frees memory for the next vision-language model.

### Can Vision Feature Caching be used with multiple images in one prompt?

Yes. Pass a list of image paths or identifiers to `cache.put()`, which concatenates them into a single key. Retrieval requires the same list order and content. This is particularly useful for visual question answering tasks comparing multiple images or analyzing image sequences.