How to Override Model Type Detection (LLM vs VLM) in oMLX

You can force oMLX to treat a model as a specific type by setting the model_type_override field in the model settings, which immediately updates the engine pool without requiring a server restart.

oMLX is an open-source inference server that automatically classifies models as LLMs, VLMs, embeddings, or other types by inspecting config.json and architecture strings in omlx/model_discovery.py. When this auto-detection misidentifies a vision-language model as a plain text model, you can manually override the classification using the model_type_override setting stored in omlx/model_settings.py.

How Automatic Model Type Detection Works

In omlx/model_discovery.py, the detect_model_type() function analyzes a model's configuration files and architecture metadata to determine whether it should function as an LLM, VLM, embedding model, or other specialized types. This detection relies on heuristics that examine vision or audio cues in the model configuration, which can occasionally misclassify multimodal models as standard language models.

Overriding Model Type Detection

When auto-detection fails, you can force the correct classification by updating the model type override in the per-model settings. The override is applied by omlx/engine_pool.py to the engine pool immediately upon saving.

Via the Admin API (REST)

The simplest method is using the OpenAI-compatible admin endpoint exposed in omlx/admin/routes.py. Send a PATCH request to /admin/models/{model_id} with the desired model_type_override value:

curl -X PATCH "http://localhost:8000/admin/models/my-vlm-model" \
  -H "Content-Type: application/json" \
  -d '{
        "model_type_override": "vlm"
      }'

The server applies the override instantly, logging:

Applied model_type override for my-vlm-model: type=vlm, engine=vlm

Via Python (Programmatic)

For programmatic control, use the ModelSettingsManager and EnginePool classes from omlx/model_settings.py and omlx/engine_pool.py:

from pathlib import Path
from omlx.model_settings import ModelSettings, ModelSettingsManager
from omlx.engine_pool import EnginePool

# 1️⃣ Load the settings manager (default location ~/.omlx)

settings_mgr = ModelSettingsManager(Path.home() / ".omlx")

# 2️⃣ Create a new settings object with the desired override

new_settings = ModelSettings(model_type_override="vlm")

# 3️⃣ Persist the settings for a given model ID

settings_mgr.set_settings("my-vlm-model", new_settings)

# 4️⃣ Apply overrides to the running engine pool

engine_pool = EnginePool(max_model_memory=None)
engine_pool.apply_settings_overrides(settings_mgr)

Clearing an Existing Override

To re-enable auto-detection, set the override to null via the API:

curl -X PATCH "http://localhost:8000/admin/models/my-vlm-model" \
  -H "Content-Type: application/json" \
  -d '{"model_type_override": null}'

Or use Python:

settings_mgr.set_settings("my-vlm-model", ModelSettings(model_type_override=None))
engine_pool.apply_settings_overrides(settings_mgr)

Supported Model Type Values

The model_type_override field accepts any of the following string literals defined in the oMLX source:

• "llm" – Standard language models
• "vlm" – Vision-language models
• "embedding" – Text embedding models
• "reranker" – Reranking models
• "audio_stt" – Audio speech-to-text models
• "audio_tts" – Audio text-to-speech models
• "audio_sts" – Audio speech-to-speech models

How the Override Works Internally

The override mechanism follows a specific code path through three core files:

1. Detection – detect_model_type() in omlx/model_discovery.py returns the auto-detected type based on model architecture.
2. Storage – ModelSettings.model_type_override in omlx/model_settings.py stores the user-defined override value in ~/.omlx/model_settings.json.
3. Application – EnginePool.apply_settings_overrides() in omlx/engine_pool.py reads the override and updates EngineEntry.model_type and EngineEntry.engine_type, potentially switching from a batched engine (for LLMs) to a vlm engine (for vision models).

When a non-None value is present, the engine pool replaces the auto-detected type immediately without restarting the server.

Summary

• Automatic detection in omlx/model_discovery.py may misclassify multimodal models as standard LLMs.
• Force the correct type by setting model_type_override to supported values like "vlm" or "llm".
• Persist settings in ~/.omlx/model_settings.json using the Admin API or Python's ModelSettingsManager.
• Apply immediately through EnginePool.apply_settings_overrides() without server restarts.
• Clear overrides by setting the field to null to re-enable automatic detection.

Frequently Asked Questions

What happens if I set an invalid model type override?

oMLX validates the input against supported literals. If you provide an invalid type, the API returns a validation error and the engine pool ignores the request, retaining the previous auto-detected or manually set type.

Does overriding the model type require restarting the oMLX server?

No. The EnginePool.apply_settings_overrides() method applies changes to the running engine pool immediately. The logs will confirm the override has been applied, and subsequent inference requests will use the correct engine type (e.g., switching from batched to vlm).

Where are model type overrides stored permanently?

Overrides persist in the JSON configuration file at ~/.omlx/model_settings.json (or your configured settings directory), managed by the ModelSettingsManager class in omlx/model_settings.py. This ensures settings survive server restarts.

Can I override the model type for any model architecture?

Yes, you can set any supported type literal for any loaded model, though forcing an incompatible type (such as treating a pure LLM as a VLM) may cause runtime errors when the engine expects vision-specific inputs that the model cannot process.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}