# How to Use oMLX as an OpenAI API Compatible Replacement

> Use oMLX to easily replace the OpenAI API with local MLX models. Redirect your OpenAI client to local MLX models by changing the base URL.

- Repository: [Jun Kim/omlx](https://github.com/jundot/omlx)
- Tags: how-to-guide
- Published: 2026-05-11

---

**oMLX runs a FastAPI server that mirrors OpenAI's REST API endpoints and JSON schemas, allowing you to redirect any OpenAI client to local MLX models by changing the base URL to your local instance.**

oMLX provides a complete OpenAI API compatible replacement for running MLX models locally on Apple Silicon. The project implements the exact route structure, request bodies, and response formats used by the official OpenAI API, making it interchangeable with `api.openai.com` in existing applications. All compatibility layers are built directly into the source code, with FastAPI routes defined in [`omlx/server.py`](https://github.com/jundot/omlx/blob/main/omlx/server.py) and Pydantic models declared in [`omlx/api/openai_models.py`](https://github.com/jundot/omlx/blob/main/omlx/api/openai_models.py).

## OpenAI-Compatible Architecture

The server exposes standard OpenAI endpoints through a modular FastAPI implementation. Route handlers in [`omlx/server.py`](https://github.com/jundot/omlx/blob/main/omlx/server.py) map HTTP requests to MLX inference engines through a clean adapter pattern.

Key routes include:

- `POST /v1/completions` handled by `create_completion`
- `POST /v1/chat/completions` handled by `create_chat_completion`
- `POST /v1/embeddings` handled by `create_embedding`
- `POST /v1/rerank` handled by `create_rerank`
- `GET /v1/models` handled by `list_models`

Request validation and serialization use Pydantic models that match the OpenAI specification character-for-character. For example, `ChatCompletionRequest` (lines 25-68), `CompletionRequest` (lines 84-110), and `ChatCompletionResponse` (lines 122-132) in [`omlx/api/openai_models.py`](https://github.com/jundot/omlx/blob/main/omlx/api/openai_models.py) ensure binary-compatible JSON contracts.

The architecture separates concerns across four core layers:

1. **Server entry point** ([`omlx/server.py`](https://github.com/jundot/omlx/blob/main/omlx/server.py)): Creates the FastAPI app, registers middleware, and wires routes via `init_server` (lines 71-124).
2. **Engine pool** ([`omlx/engine_pool.py`](https://github.com/jundot/omlx/blob/main/omlx/engine_pool.py)): Lazily loads MLX models on demand and manages per-model LRU caches through the `get_engine` interface.
3. **Protocol adapters** ([`omlx/api/openai_adapter.py`](https://github.com/jundot/omlx/blob/main/omlx/api/openai_adapter.py), [`omlx/api/anthropic_adapter.py`](https://github.com/jundot/omlx/blob/main/omlx/api/anthropic_adapter.py)): Translate between OpenAI/Anthropic JSON payloads and internal inference representations.
4. **Capability modules**: [`omlx/api/tool_calling.py`](https://github.com/jundot/omlx/blob/main/omlx/api/tool_calling.py) handles function calling, while [`omlx/api/structured_output.py`](https://github.com/jundot/omlx/blob/main/omlx/api/structured_output.py) enforces JSON-schema constraints.

Streaming responses use the `_with_sse_keepalive` helper (lines 1410-1480) to inject periodic keep-alive frames, ensuring long-running generations stay alive for SSE clients. Error handling converts exceptions into OpenAI-compatible error objects via `http_exception_handler` and `_openai_error_body` (lines 48-57).

## Starting the oMLX Server

Deploy the server locally using the CLI entry point. The `omlx serve` command triggers `init_server` to instantiate the FastAPI application and `EnginePool`.

```bash

# Install from the Python package

pip install omlx

# Launch the server with your MLX model directory

omlx serve --model-dir /path/to/mlx/models --max-model-memory 32GB

```

The initialization process:

- Parses `--model-dir` to discover available MLX models.
- Constructs an `EnginePool` for lazy model loading and memory management.
- Configures optional API-key authentication, CORS policies, and MCP tool management.

By default, the server binds to `http://127.0.0.1:8000`. Once running, it accepts the same HTTP requests as OpenAI's production API.

## Making Direct API Requests

Because oMLX implements identical routes and schemas, standard HTTP clients work without modification.

### Text Completions

Send a `POST` request to `/v1/completions` using the same payload structure as OpenAI:

```python
import requests, json

url = "http://127.0.0.1:8000/v1/completions"
payload = {
    "model": "Llama-3.2-3B",
    "prompt": "Write a haiku about autumn.",
    "max_tokens": 50,
    "temperature": 0.7
}
resp = requests.post(url, json=payload)
print(resp.json())

```

The response follows the `CompletionResponse` schema defined in [`omlx/api/openai_models.py`](https://github.com/jundot/omlx/blob/main/omlx/api/openai_models.py), returning `choices`, `usage`, and `finish_reason` fields in the standard format.

### Streaming Chat Completions

Enable Server-Sent Events (SSE) by setting `stream: true` in your chat request:

```bash
curl -N -X POST http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "Llama-3.2-3B",
        "messages": [{"role":"user","content":"Explain quantum entanglement in two sentences."}],
        "stream": true
      }'

```

Each SSE chunk conforms to `ChatCompletionChunk` (lines 53-62 of [`openai_models.py`](https://github.com/jundot/omlx/blob/main/openai_models.py)). The server injects keep-alive frames automatically to prevent timeouts during long generations.

### Embeddings

Generate vector embeddings via the `/v1/embeddings` endpoint:

```python
import requests

url = "http://127.0.0.1:8000/v1/embeddings"
payload = {
    "model": "Llama-3.2-3B",
    "input": "The quick brown fox jumps over the lazy dog."
}
resp = requests.post(url, json=payload)
print(resp.json())

```

The response shape matches OpenAI's `EmbeddingResponse` (lines 24-32 of [`omlx/api/openai_models.py`](https://github.com/jundot/omlx/blob/main/omlx/api/openai_models.py)).

### Tool Calling and Function Calling

oMLX supports OpenAI-style function definitions through the implementation in [`omlx/api/tool_calling.py`](https://github.com/jundot/omlx/blob/main/omlx/api/tool_calling.py). Submit tool definitions in the `tools` array:

```python
payload = {
    "model": "Llama-3.2-3B",
    "messages": [
        {"role": "user", "content": "What is the weather in Paris?"}
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Returns weather for a city",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"]
                }
            }
        }
    ],
    "tool_choice": "auto"
}

```

The server returns a `ChatCompletionResponse` containing `message.tool_calls` with `ToolCall` objects (defined at lines 55-60 of [`omlx/api/openai_models.py`](https://github.com/jundot/omlx/blob/main/omlx/api/openai_models.py)).

## Using the OpenAI Python SDK

Point the official `openai` Python library at your local oMLX instance by changing the `base_url`:

```python
import openai

client = openai.OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="anyvalue"  # Required but unused unless configured

)

resp = client.chat.completions.create(
    model="Llama-3.2-3B",
    messages=[{"role": "user", "content": "Tell me a joke"}],
    temperature=0.8
)
print(resp.choices[0].message.content)

```

No other code changes are required. The SDK generates requests targeting the routes defined in [`omlx/server.py`](https://github.com/jundot/omlx/blob/main/omlx/server.py), and the Pydantic models in [`omlx/api/openai_models.py`](https://github.com/jundot/omlx/blob/main/omlx/api/openai_models.py) validate the responses.

## Summary of Core Files

Understanding the codebase structure helps with debugging and extending functionality:

- **[`omlx/server.py`](https://github.com/jundot/omlx/blob/main/omlx/server.py)**: FastAPI entry point, route definitions, middleware, and error handling via `http_exception_handler`.
- **[`omlx/api/openai_models.py`](https://github.com/jundot/omlx/blob/main/omlx/api/openai_models.py)**: Pydantic schemas for all OpenAI-compatible request/response bodies including `ChatCompletionRequest` and `EmbeddingResponse`.
- **[`omlx/engine_pool.py`](https://github.com/jundot/omlx/blob/main/omlx/engine_pool.py)**: Lazy model discovery, LRU memory management, and engine retrieval via `get_engine`.
- **[`omlx/api/openai_adapter.py`](https://github.com/jundot/omlx/blob/main/omlx/api/openai_adapter.py)**: Translation layer between OpenAI JSON payloads and internal message formats.
- **[`omlx/api/tool_calling.py`](https://github.com/jundot/omlx/blob/main/omlx/api/tool_calling.py)**: Function definition parsing and JSON-schema enforcement for tool calls.
- **[`omlx/api/structured_output.py`](https://github.com/jundot/omlx/blob/main/omlx/api/structured_output.py)**: Structured JSON output validation and constraint enforcement.
- **[`omlx/utils/image.py`](https://github.com/jundot/omlx/blob/main/omlx/utils/image.py)**: Vision support utilities for extracting image URLs from OpenAI-style messages.

## Summary

- **oMLX implements the full OpenAI API specification** using FastAPI, with routes for completions, chat, embeddings, and models defined in [`omlx/server.py`](https://github.com/jundot/omlx/blob/main/omlx/server.py).
- **Deploy locally** using `omlx serve --model-dir <path>` which invokes `init_server` (lines 71-124) to configure the application.
- **Use any HTTP client or SDK** by pointing requests to `http://localhost:8000/v1` instead of OpenAI's domain; schemas are defined in [`omlx/api/openai_models.py`](https://github.com/jundot/omlx/blob/main/omlx/api/openai_models.py).
- **Support for advanced features** includes streaming SSE via `_with_sse_keepalive`, tool calling via [`omlx/api/tool_calling.py`](https://github.com/jundot/omlx/blob/main/omlx/api/tool_calling.py), and structured output via [`omlx/api/structured_output.py`](https://github.com/jundot/omlx/blob/main/omlx/api/structured_output.py).
- **No client-side code changes required** for existing OpenAI integrations due to binary-compatible JSON contracts and error formats.

## Frequently Asked Questions

### Do I need to modify my existing OpenAI client code to use oMLX?

No. Because oMLX implements identical route paths and JSON schemas as the official OpenAI API, you only need to change the `base_url` to your local server address (e.g., `http://127.0.0.1:8000/v1`). The Pydantic models in [`omlx/api/openai_models.py`](https://github.com/jundot/omlx/blob/main/omlx/api/openai_models.py) ensure that request and response shapes match the official specification exactly.

### How does oMLX handle model loading and memory management?

The `EnginePool` class in [`omlx/engine_pool.py`](https://github.com/jundot/omlx/blob/main/omlx/engine_pool.py) manages model lifecycle using lazy loading and LRU caching. When a request specifies a model ID, `get_engine` checks the pool first; if not cached, it loads the MLX model from the directory specified in `--model-dir`. The `--max-model-memory` flag constrains the total resident model size.

### Can I use Anthropic's API format with oMLX?

Yes. In addition to OpenAI compatibility, [`omlx/server.py`](https://github.com/jundot/omlx/blob/main/omlx/server.py) exposes `/v1/messages` for Anthropic-style requests, handled by `anthropic_messages`. The adapter logic in [`omlx/api/anthropic_adapter.py`](https://github.com/jundot/omlx/blob/main/omlx/api/anthropic_adapter.py) translates these payloads into the internal representation before inference, then converts responses back to Anthropic's JSON schema.

### Does oMLX support streaming responses and function calling simultaneously?

Yes. The server handles streaming chat completions with tool calls through the `create_chat_completion` handler in [`omlx/server.py`](https://github.com/jundot/omlx/blob/main/omlx/server.py). Streaming uses SSE with the `_with_sse_keepalive` helper (lines 1410-1480) to maintain connections, while [`omlx/api/tool_calling.py`](https://github.com/jundot/omlx/blob/main/omlx/api/tool_calling.py) parses function arguments from incremental tokens. Both features operate together transparently to the client.