# How to Configure Hot Cache Size vs Cold Cache Size in oMLX

> Learn how to configure oMLX hot cache size and cold cache size using environment variables or CLI flags for optimal performance. Set RAM and disk allocations effectively.

- Repository: [Jun Kim/omlx](https://github.com/jundot/omlx)
- Tags: how-to-guide
- Published: 2026-05-11

---

**Set `OMLX_HOT_CACHE_MAX_SIZE` for RAM allocation and `OMLX_PAGED_SSD_CACHE_MAX_SIZE` for disk allocation using environment variables or CLI flags like `--hot-cache-max-size` and `--paged-ssd-cache-max-size`.**

The oMLX inference engine implements a **two-tier caching system** that separates frequently accessed data from bulk storage. When you configure the **hot cache size vs cold cache size in oMLX**, you control how much KV-cache data resides in ultra-fast RAM versus paged SSD storage. This configuration directly impacts inference latency and hardware requirements, with settings defined in [`omlx/config.py`](https://github.com/jundot/omlx/blob/main/omlx/config.py) and applied through [`omlx/cache/tiered_manager.py`](https://github.com/jundot/omlx/blob/main/omlx/cache/tiered_manager.py).

## Understanding the Tiered Cache Architecture

oMLX splits KV-cache storage into distinct performance tiers. The **hot cache** operates entirely in-process RAM for sub-millisecond access, while the **cold cache** persists data to disk using a paged-SSD backend.

### Hot Cache (RAM Tier)

The hot cache stores frequently accessed KV blocks in system memory. By default, this tier is disabled (`0 GB`) to avoid unexpected memory pressure. When enabled, the size limit is stored in `OMLXConfig.paged_ssd_cache.hot_cache_max_size` and converted to bytes via `PagedSSDCacheConfig.hot_cache_max_size_bytes` within [`omlx/config.py`](https://github.com/jundot/omlx/blob/main/omlx/config.py) (lines 1110–1121).

### Cold Cache (SSD Tier)

The cold cache holds the full KV dataset on disk with a default allocation of `100 GB`. This value maps to `PagedSSDCacheConfig.max_size` and is accessed programmatically through `max_size_bytes` (lines 1107–1112). When the hot cache fills, LRU-evicted blocks flush to this SSD tier.

## Configuration Methods

You can define cache sizes through environment variables, CLI arguments, or the Python API. All methods ultimately populate the `PagedSSDCacheConfig` class.

### Environment Variables

Set `OMLX_HOT_CACHE_MAX_SIZE` and `OMLX_PAGED_SSD_CACHE_MAX_SIZE` before starting the server:

```bash
export OMLX_HOT_CACHE_MAX_SIZE="8GB"
export OMLX_PAGED_SSD_CACHE_MAX_SIZE="200GB"
export OMLX_PAGED_SSD_CACHE_DIR="/var/omlx/ssd_cache"

omlx serve

```

### CLI Flags

Pass human-readable size strings directly to `omlx serve` via flags defined in [`omlx/cli.py`](https://github.com/jundot/omlx/blob/main/omlx/cli.py):

```bash
omlx serve \
    --hot-cache-max-size 8GB \
    --paged-ssd-cache-max-size 200GB \
    --paged-ssd-cache-dir /var/omlx/ssd_cache

```

### Python API

Programmatically construct an `OMLXConfig` instance for embedded deployments:

```python
from omlx.config import OMLXConfig
from pathlib import Path

cfg = OMLXConfig.from_env()
cfg.paged_ssd_cache.hot_cache_max_size = "8GB"
cfg.paged_ssd_cache.max_size = "200GB"
cfg.paged_ssd_cache.cache_dir = Path("/var/omlx/ssd_cache")

# Initialize server with custom config

server = OMLXServer(cfg)
server.run()

```

## How Data Flows Between Tiers

The `TieredCacheManager` (initialized in [`omlx/cache/tiered_manager.py`](https://github.com/jundot/omlx/blob/main/omlx/cache/tiered_manager.py) lines 71–78) coordinates movement between tiers. It receives the `hot_cache_max_bytes` parameter, which determines the RAM budget.

When the hot cache reaches its byte limit, the manager evicts the least-recently-used blocks to the cold SSD cache. If you disable the hot cache by setting `hot_cache_max_size` to `"0"`, all KV blocks route directly to SSD storage, bypassing RAM caching entirely.

## Verifying Your Configuration

Inspect effective byte values at runtime using the configuration object:

```python
cfg = OMLXConfig.from_env()
print("Hot cache bytes:", cfg.paged_ssd_cache.hot_cache_max_size_bytes)
print("Cold cache bytes:", cfg.paged_ssd_cache.max_size_bytes)

```

## Key Source Files

- **[`omlx/config.py`](https://github.com/jundot/omlx/blob/main/omlx/config.py)**: Defines `PagedSSDCacheConfig` with `hot_cache_max_size` and `max_size` properties (lines 1107–1121).
- **[`omlx/cache/tiered_manager.py`](https://github.com/jundot/omlx/blob/main/omlx/cache/tiered_manager.py)**: Implements tier coordination via `TieredCacheManager` and the `hot_cache_max_bytes` parameter (lines 71–78).
- **[`omlx/cli.py`](https://github.com/jundot/omlx/blob/main/omlx/cli.py)**: Parses `--hot-cache-max-size` and `--paged-ssd-cache-max-size` arguments.
- **[`tests/test_hot_cache.py`](https://github.com/jundot/omlx/blob/main/tests/test_hot_cache.py)**: Contains unit tests verifying hot-cache enablement and eviction logic.

## Summary

- **Hot cache** resides in RAM and defaults to `0 GB` (disabled); set via `OMLX_HOT_CACHE_MAX_SIZE` or `--hot-cache-max-size`.
- **Cold cache** resides on SSD and defaults to `100 GB`; set via `OMLX_PAGED_SSD_CACHE_MAX_SIZE` or `--paged-ssd-cache-max-size`.
- Configuration values are parsed by `PagedSSDCacheConfig` in [`omlx/config.py`](https://github.com/jundot/omlx/blob/main/omlx/config.py) and applied through `TieredCacheManager` in [`omlx/cache/tiered_manager.py`](https://github.com/jundot/omlx/blob/main/omlx/cache/tiered_manager.py).
- When enabled, full blocks evict from hot to cold via LRU; when disabled, data writes directly to SSD.

## Frequently Asked Questions

### What is the default hot cache size in oMLX?

The default hot cache size is `0 GB`, which means the tier is disabled by default. All KV-cache data writes directly to the cold SSD cache unless you explicitly set `OMLX_HOT_CACHE_MAX_SIZE` or the `--hot-cache-max-size` CLI flag to a positive value like `8GB`.

### How does oMLX handle eviction between cache tiers?

oMLX uses an LRU (Least Recently Used) eviction policy. When the hot cache reaches its `hot_cache_max_size_bytes` limit, the `TieredCacheManager` flushes the least recently accessed blocks to the cold SSD cache. This logic is implemented in [`omlx/cache/tiered_manager.py`](https://github.com/jundot/omlx/blob/main/omlx/cache/tiered_manager.py) during the initialization of the tiered storage pipeline.

### Can I run oMLX with only the hot cache and no cold cache?

No, the cold cache is mandatory when using the paged SSD caching feature because it serves as the backing store. However, you can minimize the cold cache footprint by setting `--paged-ssd-cache-max-size` to a small value, though the hot cache cannot function as a standalone persistent store.

### Where are the cache size limits converted from strings to bytes?

The conversion happens in [`omlx/config.py`](https://github.com/jundot/omlx/blob/main/omlx/config.py) within the `PagedSSDCacheConfig` class. The properties `hot_cache_max_size_bytes` and `max_size_bytes` parse human-readable strings (e.g., `8GB`) into integer byte counts for internal use by the `TieredCacheManager`.