# How to Set Up Per-Model TTL and Memory Limits in oMLX

> Learn to set per-model TTL and memory limits in oMLX using ModelSettings. Ensure efficient resource management with TTL checks and memory monitoring for your models.

- Repository: [Jun Kim/omlx](https://github.com/jundot/omlx)
- Tags: how-to-guide
- Published: 2026-05-11

---

**To configure per-model TTL and memory limits in oMLX, use the `ModelSettings` dataclass in [`omlx/model_settings.py`](https://github.com/jundot/omlx/blob/main/omlx/model_settings.py) to define `ttl_seconds` and `max_model_memory`, which the `EnginePool` enforces via periodic TTL checks and memory monitoring.**

The jundot/omlx repository provides fine-grained resource control through per-model configuration, allowing you to specify exactly how long models remain idle in memory and how much RAM each can consume. This prevents resource exhaustion in multi-model deployments by automatically unloading inactive models and enforcing hard memory caps.

## How Model Settings Work in oMLX

oMLX stores every model's runtime configuration in the **`ModelSettings`** dataclass located in [[`omlx/model_settings.py`](https://github.com/jundot/omlx/blob/main/omlx/model_settings.py)](https://github.com/jundot/omlx/blob/main/omlx/model_settings.py). This system allows each model to maintain independent TTL and memory constraints while sharing the same server process.

### The ModelSettings Dataclass

The core configuration fields are defined in the `ModelSettings` class:

- **`ttl_seconds`**: An `Optional[int]` specifying idle time before automatic unload. When set to `None`, TTL is disabled for that model.
- **`is_pinned`**: A boolean that prevents TTL-based unloading when `True`.

According to the source code at lines 48-51, `ttl_seconds` controls automatic eviction based on inactivity.

### Persistence Layer

The **`ModelSettingsManager`** class handles thread-safe read/write operations to [`model_settings.json`](https://github.com/jundot/omlx/blob/main/model_settings.json). The manager guarantees that the file always reflects the latest in-memory configuration, with `_load` (lines 59-84) and `_save` (lines 101-115) methods ensuring atomic updates.

## Configuring Per-Model TTL

TTL (time-to-live) determines how long a model stays loaded after its last request. The **`EnginePool`** in [[`omlx/engine_pool.py`](https://github.com/jundot/omlx/blob/main/omlx/engine_pool.py)](https://github.com/jundot/omlx/blob/main/omlx/engine_pool.py) runs periodic checks via `_check_ttl` (lines 989-1012) to unload models that exceed their idle time.

Set a 5-minute TTL for a specific model:

```python
from pathlib import Path
from omlx.model_settings import ModelSettingsManager

settings_mgr = ModelSettingsManager(Path("/var/omlx/config"))
model_id = "gemma-7b"

current = settings_mgr.get_settings(model_id)
current.ttl_seconds = 300  # 5 minutes

settings_mgr.set_settings(model_id, current)

```

The updated configuration persists to [`model_settings.json`](https://github.com/jundot/omlx/blob/main/model_settings.json). The engine pool sweeps every ~30 seconds and unloads the model after 300 seconds of inactivity.

### Pinning Models to Bypass TTL

To keep a model resident indefinitely, set `is_pinned = True`:

```python
current.is_pinned = True
current.ttl_seconds = None  # Explicitly disable TTL

settings_mgr.set_settings(model_id, current)

```

Pinned models remain loaded for the server's lifetime unless explicitly unloaded via the admin API.

## Setting Per-Model Memory Limits

The **`max_model_memory`** field sets an upper bound in bytes for RAM consumption per model. As implemented in [`EnginePool`](https://github.com/jundot/omlx/blob/main/omlx/engine_pool.py) at lines 991-1004, this limit prevents individual models from monopolizing system memory.

Apply an 8 GiB cap to the engine pool:

```python

# Access the running EnginePool instance

pool = get_engine_pool()  # From omlx.server

pool.max_model_memory = 8 * 1024**3  # 8 GiB in bytes

```

When a model attempts to exceed this limit, the engine refuses new requests until memory is freed through TTL eviction of other idle models. The **`ProcessMemoryEnforcer`** in [[`omlx/process_memory_enforcer.py`](https://github.com/jundot/omlx/blob/main/omlx/process_memory_enforcer.py)](https://github.com/jundot/omlx/blob/main/omlx/process_memory_enforcer.py) (lines 191-210) monitors global process memory and triggers unloading when limits are breached, consulting per-model caps first.

## Using the CLI and REST API

Both settings are exposed through the CLI and admin REST API for operational flexibility.

Configure via CLI:

```bash

# Set 2-minute TTL for llama-2-13b

omlx set-model-settings llama-2-13b --ttl 120

# Limit each model to 6 GiB RAM

omlx set-engine-pool --max-memory 6G

```

The CLI commands in [[`omlx/cli.py`](https://github.com/jundot/omlx/blob/main/omlx/cli.py)](https://github.com/jundot/omlx/blob/main/omlx/cli.py) (lines 98-112) write to the same JSON files as the Python API.

For programmatic access, the REST API endpoint `/admin/models/:id/settings` in [[`omlx/admin/routes.py`](https://github.com/jundot/omlx/blob/main/omlx/admin/routes.py)](https://github.com/jundot/omlx/blob/main/omlx/admin/routes.py) (lines 4875-4902) accepts JSON bodies:

```json
{
  "ttl_seconds": 120,
  "max_model_memory": 6442450944
}

```

## Using Configuration Profiles

For reusable resource policies across multiple models, define profiles stored in [`model_profiles.json`](https://github.com/jundot/omlx/blob/main/model_profiles.json):

```json
{
  "name": "low-mem-fast-ttl",
  "display_name": "Low Memory + Fast TTL",
  "settings": {
    "max_model_memory": 8589934592,
    "ttl_seconds": 120
  }
}

```

Apply the profile using the manager:

```python
settings_mgr.save_profile(
    model_id="llama-2-13b",
    name="low-mem-fast-ttl",
    display_name="Low Memory + Fast TTL",
    description="8 GB limit, 2-min TTL",
    settings={"max_model_memory": 8 * 1024**3, "ttl_seconds": 120},
)
settings_mgr.apply_profile("llama-2-13b", "low-mem-fast-ttl")

```

## Summary

- **Per-model configuration** lives in the `ModelSettings` dataclass within [`omlx/model_settings.py`](https://github.com/jundot/omlx/blob/main/omlx/model_settings.py), providing independent `ttl_seconds` and memory constraints for each model.
- **TTL enforcement** occurs through `EnginePool._check_ttl`, automatically unloading models after idle periods unless `is_pinned` is `True`.
- **Memory limits** are enforced by the `EnginePool` and `ProcessMemoryEnforcer`, preventing models from exceeding their `max_model_memory` allocation in bytes.
- **Persistence** is handled by `ModelSettingsManager`, which atomically updates [`model_settings.json`](https://github.com/jundot/omlx/blob/main/model_settings.json) and supports reusable profiles.
- **Multiple interfaces** include the Python API, CLI commands (`omlx set-model-settings`), and REST endpoints (`/admin/models/:id/settings`).

## Frequently Asked Questions

### What happens when a model exceeds its memory limit?

When a model attempts to allocate beyond its `max_model_memory` cap, the `EnginePool` refuses new inference requests for that model until sufficient memory is freed. The `ProcessMemoryEnforcer` may trigger eviction of idle models based on their TTL settings to reclaim space, as implemented in [`omlx/process_memory_enforcer.py`](https://github.com/jundot/omlx/blob/main/omlx/process_memory_enforcer.py) lines 191-210.

### How does pinning affect TTL settings?

Setting `is_pinned = True` in `ModelSettings` completely disables TTL-based unloading for that model, regardless of the `ttl_seconds` value. Pinned models remain resident in RAM for the server's lifetime or until explicitly unloaded via the admin API.

### Can I set different TTL values for different models?

Yes. Since `ttl_seconds` is a per-model setting stored in [`model_settings.json`](https://github.com/jundot/omlx/blob/main/model_settings.json), each model can have its own idle timeout. One model can use a 300-second TTL while another uses 120 seconds, allowing granular control over memory retention based on usage patterns.

### Where are model settings stored on disk?

oMLX persists model configuration to [`model_settings.json`](https://github.com/jundot/omlx/blob/main/model_settings.json) in the base configuration directory (typically `/var/omlx/config`). The `ModelSettingsManager` class handles thread-safe atomic writes to this file, with profiles stored separately in [`model_profiles.json`](https://github.com/jundot/omlx/blob/main/model_profiles.json).