# omlx | Jun Kim | Knowledge Base | Instagit

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

GitHub Stars: 13.5k

Repository: https://github.com/jundot/omlx

---

## Articles

### [oMLX Cache Stack Architecture: Two-Tier Caching for Vision Features](/jundot/omlx/what-is-omlx-cache-stack-architecture)

Explore the oMLX cache stack architecture. Discover its two-tier caching with in-memory LRU and SSD persistence to speed up vision feature tensor computation.

- Tags: architecture
- Published: 2026-05-11

### [How to Set Up Model Aliases for Custom API-Visible Names in oMLX](/jundot/omlx/how-to-set-up-model-aliases-omlx)

Learn how to set up model aliases in oMLX to use custom API-visible names. Decouple storage layout from client identifiers for better API management.

- Tags: how-to-guide
- Published: 2026-05-11

### [How to Integrate oMLX with OpenCode, OpenClaw, or Codex: Complete Setup Guide](/jundot/omlx/how-to-integrate-omlx-with-open-ai-tools)

Integrate oMLX with OpenCode OpenClaw or Codex effortlessly. This guide shows how to run the oMLX inference server and use Python classes to auto configure your tools for seamless integration.

- Tags: how-to-guide
- Published: 2026-05-11

### [How LRU Eviction Works for Multi-Model Serving in oMLX](/jundot/omlx/how-lru-eviction-works-multi-model-serving-omlx)

Learn how LRU eviction in oMLX efficiently manages GPU memory for multi-model serving, evicting least-recently-used models to make space for new requests.

- Tags: internals
- Published: 2026-05-11

### [How to Configure Hot Cache Size vs Cold Cache Size in oMLX](/jundot/omlx/how-to-configure-hot-vs-cold-cache-size-omlx)

Learn how to configure oMLX hot cache size and cold cache size using environment variables or CLI flags for optimal performance. Set RAM and disk allocations effectively.

- Tags: how-to-guide
- Published: 2026-05-11

### [How to Run Performance Benchmarks with Prefill and Generation Speed Tests in oMLX](/jundot/omlx/how-to-run-performance-benchmarks-omlx)

Learn how to run performance benchmarks in oMLX using its built-in utility. Measure TTFT, generation throughput, and continuous batching performance with ease.

- Tags: performance
- Published: 2026-05-11

### [How to Download Models from HuggingFace Directly in the oMLX Admin Dashboard](/jundot/omlx/how-to-download-models-huggingface-omlx-admin)

Easily download HuggingFace models directly within the oMLX admin dashboard. Our FastAPI integration simplifies model management for your projects.

- Tags: how-to-guide
- Published: 2026-05-11

### [How to Use the oMLX Anthropic Messages API Endpoint: A Complete Implementation Guide](/jundot/omlx/how-to-use-omlx-anthropic-messages-api-endpoint)

Implement the Anthropic Messages API endpoint with oMLX. Discover how to use this compatible API for seamless integration and JSON or SSE stream responses. Get the complete guide.

- Tags: how-to-guide
- Published: 2026-05-11

### [How to Use oMLX as an OpenAI API Compatible Replacement](/jundot/omlx/how-to-use-omlx-openai-api-compatible-replacement)

Use oMLX to easily replace the OpenAI API with local MLX models. Redirect your OpenAI client to local MLX models by changing the base URL.

- Tags: how-to-guide
- Published: 2026-05-11

### [How to Customize Chat Template Kwargs Per Model in oMLX: Complete Guide](/jundot/omlx/how-to-customize-chat-template-kwargs-omlx)

Learn to customize chat template kwargs per model in oMLX. Control Jinja parameters, merge settings, and enforce policies with forced_ct_kwargs for robust chat management in your Jundot/omlx projects.

- Tags: how-to-guide
- Published: 2026-05-11

### [How to Override Model Type Detection (LLM vs VLM) in oMLX](/jundot/omlx/how-to-override-model-type-detection-omlx)

Force oMLX model type detection (LLM vs VLM) with model_type_override. Update the engine pool instantly without server restarts and gain full control over your model configurations.

- Tags: how-to-guide
- Published: 2026-05-11

### [How oMLX Enforces Process Memory Limits to Prevent OOM Crashes](/jundot/omlx/how-omlx-enforces-process-memory-limits-prevent-oom)

Learn how oMLX enforces process memory limits to prevent OOM crashes. Discover its continuous monitoring and eviction strategies for stable MLX runtime.

- Tags: internals
- Published: 2026-05-11

### [How to Adjust Max Concurrent Requests in oMLX's Scheduler: 3 Configuration Methods](/jundot/omlx/how-to-adjust-max-concurrent-requests-omlx-scheduler)

Learn to adjust max concurrent requests in oMLX's scheduler. Discover three methods: CLI flag, environment variable, and settings file to effectively configure your oMLX instance.

- Tags: how-to-guide
- Published: 2026-05-11

### [How to Set Up Paged SSD Cache for KV Blocks in oMLX](/jundot/omlx/how-to-set-up-paged-ssd-cache-kv-blocks-omlx)

Learn to set up paged SSD cache for KV blocks in oMLX. Configure cache directory, leverage LRU eviction, and use asynchronous background writes for efficient disk storage.

- Tags: how-to-guide
- Published: 2026-05-11

### [How to Configure API Key Authentication for oMLX Server: Complete Setup Guide](/jundot/omlx/how-to-configure-api-key-authentication-omlx-server)

Learn how to configure API key authentication for oMLX server. Secure your server using CLI flags environment variables or the Admin UI with this simple setup guide.

- Tags: how-to-guide
- Published: 2026-05-11

### [How to Set Up MCP (Model Context Protocol) with an oMLX Server](/jundot/omlx/how-to-set-up-mcp-omlx-server)

Learn how to set up MCP with an oMLX server by creating a config file and launching the server. Expose model inference easily.

- Tags: how-to-guide
- Published: 2026-05-11

### [How oMLX Implements Tool Calling with Structured Output for Different Model Families](/jundot/omlx/how-omlx-implements-tool-calling-structured-output)

Discover how oMLX implements flexible tool calling with structured JSON schema output for OpenAI Gemma and XML models. Centralized logic adapts to model families.

- Tags: deep-dive
- Published: 2026-05-11

### [How to Configure Claude Code Optimization with Context Scaling in oMLX](/jundot/omlx/how-to-configure-claude-code-optimization-omlx)

Configure Claude Code optimization with context scaling in oMLX. Learn how to auto-compact prompts against a larger virtual context window for better performance.

- Tags: how-to-guide
- Published: 2026-05-11

### [How oMLX Handles Multi-Image Chat with Vision-Language Models (VLMs)](/jundot/omlx/how-omlx-handles-vlm-multi-image-chat)

Discover how oMLX powers multi-image chat with VLMs using VLMBatchedEngine. Learn about efficient feature caching and embedding injection for seamless vision-language interaction.

- Tags: deep-dive
- Published: 2026-05-11

### [How to Set Up Per-Model TTL and Memory Limits in oMLX](/jundot/omlx/how-to-set-up-per-model-ttl-memory-limits-omlx)

Learn to set per-model TTL and memory limits in oMLX using ModelSettings. Ensure efficient resource management with TTL checks and memory monitoring for your models.

- Tags: how-to-guide
- Published: 2026-05-11

### [How to Use Model Pinning in oMLX to Keep Models Resident in Memory](/jundot/omlx/how-to-use-model-pinning-omlx)

Learn how to use model pinning in oMLX to keep frequently used models in memory. Set is_pinned=True to prevent eviction and optimize performance.

- Tags: how-to-guide
- Published: 2026-05-11

### [How OMLX's Tiered KV Cache (Hot + Cold SSD) Works for Token Caching](/jundot/omlx/how-does-omlx-tiered-kv-cache-work)

Discover how OMLX's tiered KV cache (hot RAM + cold SSD) optimizes token caching, allowing sequences to exceed memory limits for efficient language model performance.

- Tags: internals
- Published: 2026-05-11