How to Use Model Pinning in oMLX to Keep Models Resident in Memory

Set is_pinned=True in the ModelSettings dataclass via ModelSettingsManager to prevent the oMLX runtime from evicting frequently used models from the in-memory cache.

The oMLX framework provides a built-in model pinning mechanism that allows you to designate specific models as resident in memory, eliminating cold-start latency for high-traffic inference workloads. By configuring the is_pinned flag in the model settings, you ensure that the process memory enforcer skips these models during routine cache eviction cycles.

Understanding Model Pinning in oMLX

Model pinning works as a soft guarantee: pinned models remain loaded for the lifetime of the process, bypassing the standard least-recently-used (LRU) eviction logic that would otherwise unload idle models to free VRAM or system RAM.

The ModelSettings Dataclass

Per-model configuration resides in the ModelSettings dataclass defined in [omlx/model_settings.py](https://github.com/jundot/omlx/blob/main/omlx/model_settings.py#L31-L49). The critical field for this feature is:

is_pinned: bool = False

When this boolean is set to True, the runtime treats the model as exempt from eviction policies.

The ModelSettingsManager Persistence Layer

The persistence logic is handled by ModelSettingsManager (lines ≈ 220‑390 in the same file). This manager handles loading and saving settings to a JSON file (model_settings.json) and provides the get_pinned_model_ids() method at lines 83‑89, which queries all models with is_pinned=True.

When the engine pool faces memory pressure, it consults this list and excludes pinned model IDs from eviction candidates. The test suite in [tests/test_process_memory_enforcer.py](https://github.com/jundot/omlx/blob/main/tests/test_process_memory_enforcer.py#L94-L100) validates this behavior, confirming that eviction halts when all loaded models are pinned.

How to Pin a Model Programmatically

To pin a model using the Python API, initialize ModelSettingsManager and toggle the is_pinned flag:

from pathlib import Path
from omlx.model_settings import ModelSettings, ModelSettingsManager

# Initialize the manager (uses a writable directory for persistence)

mgr = ModelSettingsManager(Path(".omlx"))

# Retrieve current settings for the target model (creates defaults if absent)

settings = mgr.get_settings("gpt-4-turbo")

# Enable pinning

settings.is_pinned = True

# Persist to disk

mgr.set_settings("gpt-4-turbo", settings)

print("Pinned models:", mgr.get_pinned_model_ids())

After execution, the model_settings.json file stores the configuration, ensuring the model loads as pinned in subsequent process restarts.

Managing Pinned Models

Unpinning a Model

To remove the resident-memory guarantee, reverse the flag:

settings = mgr.get_settings("gpt-4-turbo")
settings.is_pinned = False
mgr.set_settings("gpt-4-turbo", settings)

Listing All Pinned Models

Audit currently pinned models without loading them:

pinned = mgr.get_pinned_model_ids()
print("Currently pinned:", pinned)

This returns a list of model ID strings that the runtime will protect during memory enforcement.

Pinning Models via the Admin API

If the oMLX server is running, you can toggle pinning via HTTP without restarting the process. The admin endpoint accepts JSON payloads that mirror the ModelSettings structure.

Pin a model using curl:

curl -X POST http://localhost:8000/admin/model/gpt-4-turbo/settings \
     -H "Content-Type: application/json" \
     -d '{"is_pinned": true}'

The server internally delegates to ModelSettingsManager.set_settings, writing the configuration to the same JSON backing store used by the Python API.

How the Runtime Respects Pinning

The ProcessMemoryEnforcer (tested at lines 94‑100 in test_process_memory_enforcer.py) implements the eviction logic. Before unloading a model, the enforcer calls get_pinned_model_ids() and filters the candidate set. If a model ID appears in the pinned list, the enforcer skips it regardless of idle time or memory pressure.

This integration ensures that frequently used models—such as foundational LLMs serving real-time traffic—remain hot in memory while cold or development models are evicted first.

Summary

• Model pinning in oMLX prevents specific models from being unloaded during memory pressure events.
• The is_pinned boolean in the ModelSettings dataclass (lines 31‑49 in omlx/model_settings.py) controls this behavior.
• Use ModelSettingsManager.get_settings() and set_settings() to toggle pinning programmatically, or use the admin HTTP API for runtime updates.
• Pinned models are persisted to model_settings.json and automatically respected by the ProcessMemoryEnforcer during cache eviction cycles.
• Query current pinned status anytime via get_pinned_model_ids() to audit resident models.

Frequently Asked Questions

What happens if I pin more models than available memory allows?

Pinned models act as a soft guarantee, not a hard allocation. If total pinned model size exceeds available VRAM or RAM, the system may crash or invoke OOM killers. The oMLX runtime does not prevent you from pinning models beyond physical capacity, so monitor memory usage when pinning large base models.

Does pinning persist across server restarts?

Yes. Because ModelSettingsManager writes the is_pinned flag to model_settings.json on disk, the configuration survives process restarts. When the server reloads, it reads this file and treats previously pinned models as resident from initialization.

Can I pin models while the server is running without restarting?

Yes. Use the admin HTTP endpoint at /admin/model/{model_id}/settings with a JSON payload containing {"is_pinned": true}. The change takes effect immediately for future eviction checks; if the model is already loaded, it remains loaded, and if it is not loaded, the next request will load it as a pinned resident.

How does pinning affect model loading time for the first request?

Pinning does not accelerate the initial load time of a cold model. The benefit appears on subsequent inference requests after the model has been loaded once. By preventing eviction, pinning eliminates the reload latency that would otherwise occur between idle periods, making it ideal for models with steady traffic patterns.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}