How to Set Up MCP (Model Context Protocol) with an oMLX Server

Setting up MCP with oMLX requires creating a JSON or YAML configuration file, pointing the server to it via the OMLX_MCP_CONFIG environment variable or --mcp-config CLI flag, and launching the server with omlx serve to expose model inference capabilities over STDIO or SSE transports.

The Model Context Protocol (MCP) is the standardized mechanism that the jundot/omlx repository uses to bridge external clients with model inference, tool calling, and streaming completions. By implementing the transport layer defined in omlx/mcp/, the server can route OpenAI-compatible API requests through configurable backends. This guide walks through the exact configuration steps based on the current source implementation.

Create the MCP Configuration File

The server expects a configuration file that describes the model, transport protocol, and optional tool definitions. This file drives the initialization logic in omlx/mcp/config.py, where the MCPConfig dataclass validates all inputs.

A minimal configuration (mcp.json) requires the following fields:

{
  "model_name": "llama-3.1-8b-instruct",
  "max_context_len": 8192,
  "transport": "stdio",
  "server_name": "default",
  "tools": [
    {
      "server_name": "default",
      "name": "search",
      "description": "Perform a web search and return the top result."
    }
  ],
  "default_temperature": 0.7,
  "default_top_p": 0.95
}

Key fields defined in omlx/mcp/types.py include:

• model_name – Must match an entry registered in omlx/model_registry.py.
• transport – Accepts "stdio" for subprocess communication via newline-delimited JSON, or "sse" for Server-Sent Events over HTTP.
• tools – Optional array of MCPTool definitions that expose callable functions to the client.

Save this file to ~/.config/omlx/mcp.json (or use the .yaml extension if preferred). The MCPConfig.from_path() method in omlx/mcp/config.py handles parsing and validation for both formats.

Point the Server to Your Configuration

The server resolves the configuration location using a strict precedence order implemented in the loader:

1. Path supplied via --mcp-config <path> on the CLI.
2. Path specified in the OMLX_MCP_CONFIG environment variable.
3. Default locations: ~/.config/omlx/mcp.json or ~/.config/omlx/mcp.yaml.

Set the environment variable before launching:

export OMLX_MCP_CONFIG=$HOME/.config/omlx/mcp.json

Alternatively, pass the path directly when starting the server:

omlx serve --mcp-config /custom/path/mcp.yaml

If the file is missing or malformed, MCPConfig.from_path() raises a validation error referencing the specific schema violation found in omlx/mcp/types.py.

Launch the oMLX Server

With the configuration in place, start the server:

omlx serve

During initialization, the server executes the following sequence defined in omlx/mcp/manager.py and omlx/mcp/executor.py:

• Invokes MCPConfig.from_dict() to instantiate the configuration object.
• Creates an MCPServerStatus instance with state=CONNECTING and the selected MCPTransport enum.
• Spawns the appropriate executor: MCPExecutorSTDIO for subprocess management or MCPExecutorSSE for HTTP event streaming.

For STDIO transport, the server launches a subprocess (e.g., python -m omlx.mlx) and communicates via JSON lines over stdin/stdout. For SSE transport, the server exposes an endpoint at /v1/mcp/events that clients subscribe to for streaming partial completions.

Connect a Client and Issue Requests

Once running, use the MCPClient class from omlx/mcp/client.py to interact with the server. This façade wraps the generic request building logic in omlx/api/openai_models.py and translates standard OpenAI-style calls into the MCP transport layer.

from omlx.mcp.client import MCPClient

# Automatically reads OMLX_MCP_CONFIG if set

client = MCPClient()

response = client.chat_completion(
    model="llama-3.1-8b-instruct",
    messages=[{"role": "user", "content": "Write a haiku about AI."}],
    temperature=0.7,
)

print(response["choices"][0]["message"]["content"])

For tool-calling, include the tool definition in your request. The client maps MCPTool definitions to the OpenAI-compatible format expected by the executor:

response = client.chat_completion(
    model="llama-3.1-8b-instruct",
    messages=[
        {"role": "user", "content": "What is 7 * 6? Use the calc tool."}
    ],
    tool_calls=[{"name": "calc", "arguments": {"expr": "7*6"}}],
)

To consume SSE streams directly without the client wrapper:

import requests

url = "http://localhost:8000/v1/mcp/events?model=llama-3.1-8b-instruct"
with requests.get(url, stream=True) as r:
    for line in r.iter_lines():
        if line:
            event = line.decode('utf-8')
            print(event)

Summary

• MCP standardizes how oMLX exposes inference capabilities to external clients through a configuration-driven architecture.
• Create a JSON or YAML file defining model_name, transport (stdio or sse), and optional tools.
• Point the server to this file using OMLX_MCP_CONFIG, --mcp-config, or place it at ~/.config/omlx/mcp.json.
• Launch with omlx serve to instantiate MCPExecutorSTDIO or MCPExecutorSSE as defined in omlx/mcp/executor.py.
• Use MCPClient from omlx/mcp/client.py for high-level API calls, or connect directly to SSE endpoints for streaming.

Frequently Asked Questions

What is the difference between STDIO and SSE transport in oMLX MCP?

STDIO spawns the model as a subprocess and communicates via newline-delimited JSON over stdin/stdout, making it ideal for local, process-isolated deployments. SSE (Server-Sent Events) exposes an HTTP endpoint that streams partial completions to connected clients, better suited for network-distributed or browser-based consumers. The executor selection is controlled by the transport field in your MCP configuration file, processed by MCPExecutorSTDIO and MCPExecutorSSE classes in omlx/mcp/executor.py.

Can I use YAML instead of JSON for the MCP configuration?

Yes. The MCPConfig.from_path() method in omlx/mcp/config.py automatically detects the file extension and parses accordingly. Save your configuration as ~/.config/omlx/mcp.yaml and the loader will validate it against the same MCPConfig schema defined in omlx/mcp/types.py without requiring any syntax changes to the key names or structure.

Why does the server fail to start with an "Invalid model_name" error?

The model_name value in your MCP configuration must exactly match a model registered in the internal model registry (typically omlx/model_registry.py). If you specify a custom or local model not yet registered, the validation logic in MCPConfig.from_dict() will reject the configuration. Verify available models by checking the registry or consult the repository documentation for the correct identifier strings.

How do I enable tool-calling for my oMLX server?

Define the tools array in your MCP configuration file with objects containing server_name, name, and description fields that map to the MCPTool dataclass in omlx/mcp/types.py. When using MCPClient, pass the tool_calls parameter in your request; the client will format these according to the OpenAI tool-calling specification and route them through the MCP layer to the executor backend.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}