How to Set Up Per-Model TTL and Memory Limits in oMLX

To configure per-model TTL and memory limits in oMLX, use the ModelSettings dataclass in omlx/model_settings.py to define ttl_seconds and max_model_memory, which the EnginePool enforces via periodic TTL checks and memory monitoring.

The jundot/omlx repository provides fine-grained resource control through per-model configuration, allowing you to specify exactly how long models remain idle in memory and how much RAM each can consume. This prevents resource exhaustion in multi-model deployments by automatically unloading inactive models and enforcing hard memory caps.

How Model Settings Work in oMLX

oMLX stores every model's runtime configuration in the ModelSettings dataclass located in [omlx/model_settings.py](https://github.com/jundot/omlx/blob/main/omlx/model_settings.py). This system allows each model to maintain independent TTL and memory constraints while sharing the same server process.

The ModelSettings Dataclass

The core configuration fields are defined in the ModelSettings class:

• ttl_seconds: An Optional[int] specifying idle time before automatic unload. When set to None, TTL is disabled for that model.
• is_pinned: A boolean that prevents TTL-based unloading when True.

According to the source code at lines 48-51, ttl_seconds controls automatic eviction based on inactivity.

Persistence Layer

The ModelSettingsManager class handles thread-safe read/write operations to model_settings.json. The manager guarantees that the file always reflects the latest in-memory configuration, with _load (lines 59-84) and _save (lines 101-115) methods ensuring atomic updates.

Configuring Per-Model TTL

TTL (time-to-live) determines how long a model stays loaded after its last request. The EnginePool in [omlx/engine_pool.py](https://github.com/jundot/omlx/blob/main/omlx/engine_pool.py) runs periodic checks via _check_ttl (lines 989-1012) to unload models that exceed their idle time.

Set a 5-minute TTL for a specific model:

from pathlib import Path
from omlx.model_settings import ModelSettingsManager

settings_mgr = ModelSettingsManager(Path("/var/omlx/config"))
model_id = "gemma-7b"

current = settings_mgr.get_settings(model_id)
current.ttl_seconds = 300  # 5 minutes

settings_mgr.set_settings(model_id, current)

The updated configuration persists to model_settings.json. The engine pool sweeps every ~30 seconds and unloads the model after 300 seconds of inactivity.

Pinning Models to Bypass TTL

To keep a model resident indefinitely, set is_pinned = True:

current.is_pinned = True
current.ttl_seconds = None  # Explicitly disable TTL

settings_mgr.set_settings(model_id, current)

Pinned models remain loaded for the server's lifetime unless explicitly unloaded via the admin API.

Setting Per-Model Memory Limits

The max_model_memory field sets an upper bound in bytes for RAM consumption per model. As implemented in EnginePool at lines 991-1004, this limit prevents individual models from monopolizing system memory.

Apply an 8 GiB cap to the engine pool:

# Access the running EnginePool instance

pool = get_engine_pool()  # From omlx.server

pool.max_model_memory = 8 * 1024**3  # 8 GiB in bytes

When a model attempts to exceed this limit, the engine refuses new requests until memory is freed through TTL eviction of other idle models. The ProcessMemoryEnforcer in [omlx/process_memory_enforcer.py](https://github.com/jundot/omlx/blob/main/omlx/process_memory_enforcer.py) (lines 191-210) monitors global process memory and triggers unloading when limits are breached, consulting per-model caps first.

Using the CLI and REST API

Both settings are exposed through the CLI and admin REST API for operational flexibility.

Configure via CLI:

# Set 2-minute TTL for llama-2-13b

omlx set-model-settings llama-2-13b --ttl 120

# Limit each model to 6 GiB RAM

omlx set-engine-pool --max-memory 6G

The CLI commands in [omlx/cli.py](https://github.com/jundot/omlx/blob/main/omlx/cli.py) (lines 98-112) write to the same JSON files as the Python API.

For programmatic access, the REST API endpoint /admin/models/:id/settings in [omlx/admin/routes.py](https://github.com/jundot/omlx/blob/main/omlx/admin/routes.py) (lines 4875-4902) accepts JSON bodies:

{
  "ttl_seconds": 120,
  "max_model_memory": 6442450944
}

Using Configuration Profiles

For reusable resource policies across multiple models, define profiles stored in model_profiles.json:

{
  "name": "low-mem-fast-ttl",
  "display_name": "Low Memory + Fast TTL",
  "settings": {
    "max_model_memory": 8589934592,
    "ttl_seconds": 120
  }
}

Apply the profile using the manager:

settings_mgr.save_profile(
    model_id="llama-2-13b",
    name="low-mem-fast-ttl",
    display_name="Low Memory + Fast TTL",
    description="8 GB limit, 2-min TTL",
    settings={"max_model_memory": 8 * 1024**3, "ttl_seconds": 120},
)
settings_mgr.apply_profile("llama-2-13b", "low-mem-fast-ttl")

Summary

• Per-model configuration lives in the ModelSettings dataclass within omlx/model_settings.py, providing independent ttl_seconds and memory constraints for each model.
• TTL enforcement occurs through EnginePool._check_ttl, automatically unloading models after idle periods unless is_pinned is True.
• Memory limits are enforced by the EnginePool and ProcessMemoryEnforcer, preventing models from exceeding their max_model_memory allocation in bytes.
• Persistence is handled by ModelSettingsManager, which atomically updates model_settings.json and supports reusable profiles.
• Multiple interfaces include the Python API, CLI commands (omlx set-model-settings), and REST endpoints (/admin/models/:id/settings).

Frequently Asked Questions

What happens when a model exceeds its memory limit?

When a model attempts to allocate beyond its max_model_memory cap, the EnginePool refuses new inference requests for that model until sufficient memory is freed. The ProcessMemoryEnforcer may trigger eviction of idle models based on their TTL settings to reclaim space, as implemented in omlx/process_memory_enforcer.py lines 191-210.

How does pinning affect TTL settings?

Setting is_pinned = True in ModelSettings completely disables TTL-based unloading for that model, regardless of the ttl_seconds value. Pinned models remain resident in RAM for the server's lifetime or until explicitly unloaded via the admin API.

Can I set different TTL values for different models?

Yes. Since ttl_seconds is a per-model setting stored in model_settings.json, each model can have its own idle timeout. One model can use a 300-second TTL while another uses 120 seconds, allowing granular control over memory retention based on usage patterns.

Where are model settings stored on disk?

oMLX persists model configuration to model_settings.json in the base configuration directory (typically /var/omlx/config). The ModelSettingsManager class handles thread-safe atomic writes to this file, with profiles stored separately in model_profiles.json.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}