# How to Use Model Pinning in oMLX to Keep Models Resident in Memory

> Learn how to use model pinning in oMLX to keep frequently used models in memory. Set is_pinned=True to prevent eviction and optimize performance.

- Repository: [Jun Kim/omlx](https://github.com/jundot/omlx)
- Tags: how-to-guide
- Published: 2026-05-11

---

**Set `is_pinned=True` in the `ModelSettings` dataclass via `ModelSettingsManager` to prevent the oMLX runtime from evicting frequently used models from the in-memory cache.**

The oMLX framework provides a built-in **model pinning** mechanism that allows you to designate specific models as resident in memory, eliminating cold-start latency for high-traffic inference workloads. By configuring the `is_pinned` flag in the model settings, you ensure that the process memory enforcer skips these models during routine cache eviction cycles.

## Understanding Model Pinning in oMLX

Model pinning works as a soft guarantee: pinned models remain loaded for the lifetime of the process, bypassing the standard least-recently-used (LRU) eviction logic that would otherwise unload idle models to free VRAM or system RAM.

### The ModelSettings Dataclass

Per-model configuration resides in the `ModelSettings` dataclass defined in [[`omlx/model_settings.py`](https://github.com/jundot/omlx/blob/main/omlx/model_settings.py)](https://github.com/jundot/omlx/blob/main/omlx/model_settings.py#L31-L49). The critical field for this feature is:

```python
is_pinned: bool = False

```

When this boolean is set to `True`, the runtime treats the model as exempt from eviction policies.

### The ModelSettingsManager Persistence Layer

The persistence logic is handled by `ModelSettingsManager` (lines ≈ 220‑390 in the same file). This manager handles loading and saving settings to a JSON file ([`model_settings.json`](https://github.com/jundot/omlx/blob/main/model_settings.json)) and provides the `get_pinned_model_ids()` method at lines 83‑89, which queries all models with `is_pinned=True`.

When the engine pool faces memory pressure, it consults this list and excludes pinned model IDs from eviction candidates. The test suite in [[`tests/test_process_memory_enforcer.py`](https://github.com/jundot/omlx/blob/main/tests/test_process_memory_enforcer.py)](https://github.com/jundot/omlx/blob/main/tests/test_process_memory_enforcer.py#L94-L100) validates this behavior, confirming that eviction halts when all loaded models are pinned.

## How to Pin a Model Programmatically

To pin a model using the Python API, initialize `ModelSettingsManager` and toggle the `is_pinned` flag:

```python
from pathlib import Path
from omlx.model_settings import ModelSettings, ModelSettingsManager

# Initialize the manager (uses a writable directory for persistence)

mgr = ModelSettingsManager(Path(".omlx"))

# Retrieve current settings for the target model (creates defaults if absent)

settings = mgr.get_settings("gpt-4-turbo")

# Enable pinning

settings.is_pinned = True

# Persist to disk

mgr.set_settings("gpt-4-turbo", settings)

print("Pinned models:", mgr.get_pinned_model_ids())

```

After execution, the [`model_settings.json`](https://github.com/jundot/omlx/blob/main/model_settings.json) file stores the configuration, ensuring the model loads as pinned in subsequent process restarts.

## Managing Pinned Models

### Unpinning a Model

To remove the resident-memory guarantee, reverse the flag:

```python
settings = mgr.get_settings("gpt-4-turbo")
settings.is_pinned = False
mgr.set_settings("gpt-4-turbo", settings)

```

### Listing All Pinned Models

Audit currently pinned models without loading them:

```python
pinned = mgr.get_pinned_model_ids()
print("Currently pinned:", pinned)

```

This returns a list of model ID strings that the runtime will protect during memory enforcement.

## Pinning Models via the Admin API

If the oMLX server is running, you can toggle pinning via HTTP without restarting the process. The admin endpoint accepts JSON payloads that mirror the `ModelSettings` structure.

Pin a model using `curl`:

```bash
curl -X POST http://localhost:8000/admin/model/gpt-4-turbo/settings \
     -H "Content-Type: application/json" \
     -d '{"is_pinned": true}'

```

The server internally delegates to `ModelSettingsManager.set_settings`, writing the configuration to the same JSON backing store used by the Python API.

## How the Runtime Respects Pinning

The `ProcessMemoryEnforcer` (tested at lines 94‑100 in [`test_process_memory_enforcer.py`](https://github.com/jundot/omlx/blob/main/test_process_memory_enforcer.py)) implements the eviction logic. Before unloading a model, the enforcer calls `get_pinned_model_ids()` and filters the candidate set. If a model ID appears in the pinned list, the enforcer skips it regardless of idle time or memory pressure.

This integration ensures that **frequently used models**—such as foundational LLMs serving real-time traffic—remain hot in memory while cold or development models are evicted first.

## Summary

- **Model pinning** in oMLX prevents specific models from being unloaded during memory pressure events.
- The `is_pinned` boolean in the `ModelSettings` dataclass (lines 31‑49 in [`omlx/model_settings.py`](https://github.com/jundot/omlx/blob/main/omlx/model_settings.py)) controls this behavior.
- Use `ModelSettingsManager.get_settings()` and `set_settings()` to toggle pinning programmatically, or use the admin HTTP API for runtime updates.
- Pinned models are persisted to [`model_settings.json`](https://github.com/jundot/omlx/blob/main/model_settings.json) and automatically respected by the `ProcessMemoryEnforcer` during cache eviction cycles.
- Query current pinned status anytime via `get_pinned_model_ids()` to audit resident models.

## Frequently Asked Questions

### What happens if I pin more models than available memory allows?

Pinned models act as a soft guarantee, not a hard allocation. If total pinned model size exceeds available VRAM or RAM, the system may crash or invoke OOM killers. The oMLX runtime does not prevent you from pinning models beyond physical capacity, so monitor memory usage when pinning large base models.

### Does pinning persist across server restarts?

Yes. Because `ModelSettingsManager` writes the `is_pinned` flag to [`model_settings.json`](https://github.com/jundot/omlx/blob/main/model_settings.json) on disk, the configuration survives process restarts. When the server reloads, it reads this file and treats previously pinned models as resident from initialization.

### Can I pin models while the server is running without restarting?

Yes. Use the admin HTTP endpoint at `/admin/model/{model_id}/settings` with a JSON payload containing `{"is_pinned": true}`. The change takes effect immediately for future eviction checks; if the model is already loaded, it remains loaded, and if it is not loaded, the next request will load it as a pinned resident.

### How does pinning affect model loading time for the first request?

Pinning does not accelerate the initial load time of a cold model. The benefit appears on subsequent inference requests after the model has been loaded once. By preventing eviction, pinning eliminates the reload latency that would otherwise occur between idle periods, making it ideal for models with steady traffic patterns.