How to Configure Hot Cache Size vs Cold Cache Size in oMLX

Set OMLX_HOT_CACHE_MAX_SIZE for RAM allocation and OMLX_PAGED_SSD_CACHE_MAX_SIZE for disk allocation using environment variables or CLI flags like --hot-cache-max-size and --paged-ssd-cache-max-size.

The oMLX inference engine implements a two-tier caching system that separates frequently accessed data from bulk storage. When you configure the hot cache size vs cold cache size in oMLX, you control how much KV-cache data resides in ultra-fast RAM versus paged SSD storage. This configuration directly impacts inference latency and hardware requirements, with settings defined in omlx/config.py and applied through omlx/cache/tiered_manager.py.

Understanding the Tiered Cache Architecture

oMLX splits KV-cache storage into distinct performance tiers. The hot cache operates entirely in-process RAM for sub-millisecond access, while the cold cache persists data to disk using a paged-SSD backend.

Hot Cache (RAM Tier)

The hot cache stores frequently accessed KV blocks in system memory. By default, this tier is disabled (0 GB) to avoid unexpected memory pressure. When enabled, the size limit is stored in OMLXConfig.paged_ssd_cache.hot_cache_max_size and converted to bytes via PagedSSDCacheConfig.hot_cache_max_size_bytes within omlx/config.py (lines 1110–1121).

Cold Cache (SSD Tier)

The cold cache holds the full KV dataset on disk with a default allocation of 100 GB. This value maps to PagedSSDCacheConfig.max_size and is accessed programmatically through max_size_bytes (lines 1107–1112). When the hot cache fills, LRU-evicted blocks flush to this SSD tier.

Configuration Methods

You can define cache sizes through environment variables, CLI arguments, or the Python API. All methods ultimately populate the PagedSSDCacheConfig class.

Environment Variables

Set OMLX_HOT_CACHE_MAX_SIZE and OMLX_PAGED_SSD_CACHE_MAX_SIZE before starting the server:

export OMLX_HOT_CACHE_MAX_SIZE="8GB"
export OMLX_PAGED_SSD_CACHE_MAX_SIZE="200GB"
export OMLX_PAGED_SSD_CACHE_DIR="/var/omlx/ssd_cache"

omlx serve

CLI Flags

Pass human-readable size strings directly to omlx serve via flags defined in omlx/cli.py:

omlx serve \
    --hot-cache-max-size 8GB \
    --paged-ssd-cache-max-size 200GB \
    --paged-ssd-cache-dir /var/omlx/ssd_cache

Python API

Programmatically construct an OMLXConfig instance for embedded deployments:

from omlx.config import OMLXConfig
from pathlib import Path

cfg = OMLXConfig.from_env()
cfg.paged_ssd_cache.hot_cache_max_size = "8GB"
cfg.paged_ssd_cache.max_size = "200GB"
cfg.paged_ssd_cache.cache_dir = Path("/var/omlx/ssd_cache")

# Initialize server with custom config

server = OMLXServer(cfg)
server.run()

How Data Flows Between Tiers

The TieredCacheManager (initialized in omlx/cache/tiered_manager.py lines 71–78) coordinates movement between tiers. It receives the hot_cache_max_bytes parameter, which determines the RAM budget.

When the hot cache reaches its byte limit, the manager evicts the least-recently-used blocks to the cold SSD cache. If you disable the hot cache by setting hot_cache_max_size to "0", all KV blocks route directly to SSD storage, bypassing RAM caching entirely.

Verifying Your Configuration

Inspect effective byte values at runtime using the configuration object:

cfg = OMLXConfig.from_env()
print("Hot cache bytes:", cfg.paged_ssd_cache.hot_cache_max_size_bytes)
print("Cold cache bytes:", cfg.paged_ssd_cache.max_size_bytes)

Key Source Files

• omlx/config.py: Defines PagedSSDCacheConfig with hot_cache_max_size and max_size properties (lines 1107–1121).
• omlx/cache/tiered_manager.py: Implements tier coordination via TieredCacheManager and the hot_cache_max_bytes parameter (lines 71–78).
• omlx/cli.py: Parses --hot-cache-max-size and --paged-ssd-cache-max-size arguments.
• tests/test_hot_cache.py: Contains unit tests verifying hot-cache enablement and eviction logic.

Summary

• Hot cache resides in RAM and defaults to 0 GB (disabled); set via OMLX_HOT_CACHE_MAX_SIZE or --hot-cache-max-size.
• Cold cache resides on SSD and defaults to 100 GB; set via OMLX_PAGED_SSD_CACHE_MAX_SIZE or --paged-ssd-cache-max-size.
• Configuration values are parsed by PagedSSDCacheConfig in omlx/config.py and applied through TieredCacheManager in omlx/cache/tiered_manager.py.
• When enabled, full blocks evict from hot to cold via LRU; when disabled, data writes directly to SSD.

Frequently Asked Questions

What is the default hot cache size in oMLX?

The default hot cache size is 0 GB, which means the tier is disabled by default. All KV-cache data writes directly to the cold SSD cache unless you explicitly set OMLX_HOT_CACHE_MAX_SIZE or the --hot-cache-max-size CLI flag to a positive value like 8GB.

How does oMLX handle eviction between cache tiers?

oMLX uses an LRU (Least Recently Used) eviction policy. When the hot cache reaches its hot_cache_max_size_bytes limit, the TieredCacheManager flushes the least recently accessed blocks to the cold SSD cache. This logic is implemented in omlx/cache/tiered_manager.py during the initialization of the tiered storage pipeline.

Can I run oMLX with only the hot cache and no cold cache?

No, the cold cache is mandatory when using the paged SSD caching feature because it serves as the backing store. However, you can minimize the cold cache footprint by setting --paged-ssd-cache-max-size to a small value, though the hot cache cannot function as a standalone persistent store.

Where are the cache size limits converted from strings to bytes?

The conversion happens in omlx/config.py within the PagedSSDCacheConfig class. The properties hot_cache_max_size_bytes and max_size_bytes parse human-readable strings (e.g., 8GB) into integer byte counts for internal use by the TieredCacheManager.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}