How to Use oMLX as an OpenAI API Compatible Replacement

oMLX runs a FastAPI server that mirrors OpenAI's REST API endpoints and JSON schemas, allowing you to redirect any OpenAI client to local MLX models by changing the base URL to your local instance.

oMLX provides a complete OpenAI API compatible replacement for running MLX models locally on Apple Silicon. The project implements the exact route structure, request bodies, and response formats used by the official OpenAI API, making it interchangeable with api.openai.com in existing applications. All compatibility layers are built directly into the source code, with FastAPI routes defined in omlx/server.py and Pydantic models declared in omlx/api/openai_models.py.

OpenAI-Compatible Architecture

The server exposes standard OpenAI endpoints through a modular FastAPI implementation. Route handlers in omlx/server.py map HTTP requests to MLX inference engines through a clean adapter pattern.

Key routes include:

• POST /v1/completions handled by create_completion
• POST /v1/chat/completions handled by create_chat_completion
• POST /v1/embeddings handled by create_embedding
• POST /v1/rerank handled by create_rerank
• GET /v1/models handled by list_models

Request validation and serialization use Pydantic models that match the OpenAI specification character-for-character. For example, ChatCompletionRequest (lines 25-68), CompletionRequest (lines 84-110), and ChatCompletionResponse (lines 122-132) in omlx/api/openai_models.py ensure binary-compatible JSON contracts.

The architecture separates concerns across four core layers:

1. Server entry point (omlx/server.py): Creates the FastAPI app, registers middleware, and wires routes via init_server (lines 71-124).
2. Engine pool (omlx/engine_pool.py): Lazily loads MLX models on demand and manages per-model LRU caches through the get_engine interface.
3. Protocol adapters (omlx/api/openai_adapter.py, omlx/api/anthropic_adapter.py): Translate between OpenAI/Anthropic JSON payloads and internal inference representations.
4. Capability modules: omlx/api/tool_calling.py handles function calling, while omlx/api/structured_output.py enforces JSON-schema constraints.

Streaming responses use the _with_sse_keepalive helper (lines 1410-1480) to inject periodic keep-alive frames, ensuring long-running generations stay alive for SSE clients. Error handling converts exceptions into OpenAI-compatible error objects via http_exception_handler and _openai_error_body (lines 48-57).

Starting the oMLX Server

Deploy the server locally using the CLI entry point. The omlx serve command triggers init_server to instantiate the FastAPI application and EnginePool.

# Install from the Python package

pip install omlx

# Launch the server with your MLX model directory

omlx serve --model-dir /path/to/mlx/models --max-model-memory 32GB

The initialization process:

• Parses --model-dir to discover available MLX models.
• Constructs an EnginePool for lazy model loading and memory management.
• Configures optional API-key authentication, CORS policies, and MCP tool management.

By default, the server binds to http://127.0.0.1:8000. Once running, it accepts the same HTTP requests as OpenAI's production API.

Making Direct API Requests

Because oMLX implements identical routes and schemas, standard HTTP clients work without modification.

Text Completions

Send a POST request to /v1/completions using the same payload structure as OpenAI:

import requests, json

url = "http://127.0.0.1:8000/v1/completions"
payload = {
    "model": "Llama-3.2-3B",
    "prompt": "Write a haiku about autumn.",
    "max_tokens": 50,
    "temperature": 0.7
}
resp = requests.post(url, json=payload)
print(resp.json())

The response follows the CompletionResponse schema defined in omlx/api/openai_models.py, returning choices, usage, and finish_reason fields in the standard format.

Streaming Chat Completions

Enable Server-Sent Events (SSE) by setting stream: true in your chat request:

curl -N -X POST http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "Llama-3.2-3B",
        "messages": [{"role":"user","content":"Explain quantum entanglement in two sentences."}],
        "stream": true
      }'

Each SSE chunk conforms to ChatCompletionChunk (lines 53-62 of openai_models.py). The server injects keep-alive frames automatically to prevent timeouts during long generations.

Embeddings

Generate vector embeddings via the /v1/embeddings endpoint:

import requests

url = "http://127.0.0.1:8000/v1/embeddings"
payload = {
    "model": "Llama-3.2-3B",
    "input": "The quick brown fox jumps over the lazy dog."
}
resp = requests.post(url, json=payload)
print(resp.json())

The response shape matches OpenAI's EmbeddingResponse (lines 24-32 of omlx/api/openai_models.py).

Tool Calling and Function Calling

oMLX supports OpenAI-style function definitions through the implementation in omlx/api/tool_calling.py. Submit tool definitions in the tools array:

payload = {
    "model": "Llama-3.2-3B",
    "messages": [
        {"role": "user", "content": "What is the weather in Paris?"}
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Returns weather for a city",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"]
                }
            }
        }
    ],
    "tool_choice": "auto"
}

The server returns a ChatCompletionResponse containing message.tool_calls with ToolCall objects (defined at lines 55-60 of omlx/api/openai_models.py).

Using the OpenAI Python SDK

Point the official openai Python library at your local oMLX instance by changing the base_url:

import openai

client = openai.OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="anyvalue"  # Required but unused unless configured

)

resp = client.chat.completions.create(
    model="Llama-3.2-3B",
    messages=[{"role": "user", "content": "Tell me a joke"}],
    temperature=0.8
)
print(resp.choices[0].message.content)

No other code changes are required. The SDK generates requests targeting the routes defined in omlx/server.py, and the Pydantic models in omlx/api/openai_models.py validate the responses.

Summary of Core Files

Understanding the codebase structure helps with debugging and extending functionality:

• omlx/server.py: FastAPI entry point, route definitions, middleware, and error handling via http_exception_handler.
• omlx/api/openai_models.py: Pydantic schemas for all OpenAI-compatible request/response bodies including ChatCompletionRequest and EmbeddingResponse.
• omlx/engine_pool.py: Lazy model discovery, LRU memory management, and engine retrieval via get_engine.
• omlx/api/openai_adapter.py: Translation layer between OpenAI JSON payloads and internal message formats.
• omlx/api/tool_calling.py: Function definition parsing and JSON-schema enforcement for tool calls.
• omlx/api/structured_output.py: Structured JSON output validation and constraint enforcement.
• omlx/utils/image.py: Vision support utilities for extracting image URLs from OpenAI-style messages.

Summary

• oMLX implements the full OpenAI API specification using FastAPI, with routes for completions, chat, embeddings, and models defined in omlx/server.py.
• Deploy locally using omlx serve --model-dir <path> which invokes init_server (lines 71-124) to configure the application.
• Use any HTTP client or SDK by pointing requests to http://localhost:8000/v1 instead of OpenAI's domain; schemas are defined in omlx/api/openai_models.py.
• Support for advanced features includes streaming SSE via _with_sse_keepalive, tool calling via omlx/api/tool_calling.py, and structured output via omlx/api/structured_output.py.
• No client-side code changes required for existing OpenAI integrations due to binary-compatible JSON contracts and error formats.

Frequently Asked Questions

Do I need to modify my existing OpenAI client code to use oMLX?

No. Because oMLX implements identical route paths and JSON schemas as the official OpenAI API, you only need to change the base_url to your local server address (e.g., http://127.0.0.1:8000/v1). The Pydantic models in omlx/api/openai_models.py ensure that request and response shapes match the official specification exactly.

How does oMLX handle model loading and memory management?

The EnginePool class in omlx/engine_pool.py manages model lifecycle using lazy loading and LRU caching. When a request specifies a model ID, get_engine checks the pool first; if not cached, it loads the MLX model from the directory specified in --model-dir. The --max-model-memory flag constrains the total resident model size.

Can I use Anthropic's API format with oMLX?

Yes. In addition to OpenAI compatibility, omlx/server.py exposes /v1/messages for Anthropic-style requests, handled by anthropic_messages. The adapter logic in omlx/api/anthropic_adapter.py translates these payloads into the internal representation before inference, then converts responses back to Anthropic's JSON schema.

Does oMLX support streaming responses and function calling simultaneously?

Yes. The server handles streaming chat completions with tool calls through the create_chat_completion handler in omlx/server.py. Streaming uses SSE with the _with_sse_keepalive helper (lines 1410-1480) to maintain connections, while omlx/api/tool_calling.py parses function arguments from incremental tokens. Both features operate together transparently to the client.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}