# How to Override Model Type Detection (LLM vs VLM) in oMLX

> Force oMLX model type detection (LLM vs VLM) with model_type_override. Update the engine pool instantly without server restarts and gain full control over your model configurations.

- Repository: [Jun Kim/omlx](https://github.com/jundot/omlx)
- Tags: how-to-guide
- Published: 2026-05-11

---

**You can force oMLX to treat a model as a specific type by setting the `model_type_override` field in the model settings, which immediately updates the engine pool without requiring a server restart.**

oMLX is an open-source inference server that automatically classifies models as LLMs, VLMs, embeddings, or other types by inspecting [`config.json`](https://github.com/jundot/omlx/blob/main/config.json) and architecture strings in [`omlx/model_discovery.py`](https://github.com/jundot/omlx/blob/main/omlx/model_discovery.py). When this auto-detection misidentifies a vision-language model as a plain text model, you can manually override the classification using the `model_type_override` setting stored in [`omlx/model_settings.py`](https://github.com/jundot/omlx/blob/main/omlx/model_settings.py).

## How Automatic Model Type Detection Works

In [`omlx/model_discovery.py`](https://github.com/jundot/omlx/blob/main/omlx/model_discovery.py), the `detect_model_type()` function analyzes a model's configuration files and architecture metadata to determine whether it should function as an **LLM**, **VLM**, **embedding** model, or other specialized types. This detection relies on heuristics that examine vision or audio cues in the model configuration, which can occasionally misclassify multimodal models as standard language models.

## Overriding Model Type Detection

When auto-detection fails, you can force the correct classification by updating the **model type override** in the per-model settings. The override is applied by [`omlx/engine_pool.py`](https://github.com/jundot/omlx/blob/main/omlx/engine_pool.py) to the engine pool immediately upon saving.

### Via the Admin API (REST)

The simplest method is using the OpenAI-compatible admin endpoint exposed in [`omlx/admin/routes.py`](https://github.com/jundot/omlx/blob/main/omlx/admin/routes.py). Send a **PATCH** request to `/admin/models/{model_id}` with the desired `model_type_override` value:

```bash
curl -X PATCH "http://localhost:8000/admin/models/my-vlm-model" \
  -H "Content-Type: application/json" \
  -d '{
        "model_type_override": "vlm"
      }'

```

The server applies the override instantly, logging:

```

Applied model_type override for my-vlm-model: type=vlm, engine=vlm

```

### Via Python (Programmatic)

For programmatic control, use the `ModelSettingsManager` and `EnginePool` classes from [`omlx/model_settings.py`](https://github.com/jundot/omlx/blob/main/omlx/model_settings.py) and [`omlx/engine_pool.py`](https://github.com/jundot/omlx/blob/main/omlx/engine_pool.py):

```python
from pathlib import Path
from omlx.model_settings import ModelSettings, ModelSettingsManager
from omlx.engine_pool import EnginePool

# 1️⃣ Load the settings manager (default location ~/.omlx)

settings_mgr = ModelSettingsManager(Path.home() / ".omlx")

# 2️⃣ Create a new settings object with the desired override

new_settings = ModelSettings(model_type_override="vlm")

# 3️⃣ Persist the settings for a given model ID

settings_mgr.set_settings("my-vlm-model", new_settings)

# 4️⃣ Apply overrides to the running engine pool

engine_pool = EnginePool(max_model_memory=None)
engine_pool.apply_settings_overrides(settings_mgr)

```

### Clearing an Existing Override

To re-enable auto-detection, set the override to `null` via the API:

```bash
curl -X PATCH "http://localhost:8000/admin/models/my-vlm-model" \
  -H "Content-Type: application/json" \
  -d '{"model_type_override": null}'

```

Or use Python:

```python
settings_mgr.set_settings("my-vlm-model", ModelSettings(model_type_override=None))
engine_pool.apply_settings_overrides(settings_mgr)

```

## Supported Model Type Values

The `model_type_override` field accepts any of the following string literals defined in the oMLX source:

- `"llm"` – Standard language models
- `"vlm"` – Vision-language models
- `"embedding"` – Text embedding models
- `"reranker"` – Reranking models
- `"audio_stt"` – Audio speech-to-text models
- `"audio_tts"` – Audio text-to-speech models
- `"audio_sts"` – Audio speech-to-speech models

## How the Override Works Internally

The override mechanism follows a specific code path through three core files:

1. **Detection** – `detect_model_type()` in [`omlx/model_discovery.py`](https://github.com/jundot/omlx/blob/main/omlx/model_discovery.py) returns the auto-detected type based on model architecture.
2. **Storage** – `ModelSettings.model_type_override` in [`omlx/model_settings.py`](https://github.com/jundot/omlx/blob/main/omlx/model_settings.py) stores the user-defined override value in `~/.omlx/model_settings.json`.
3. **Application** – `EnginePool.apply_settings_overrides()` in [`omlx/engine_pool.py`](https://github.com/jundot/omlx/blob/main/omlx/engine_pool.py) reads the override and updates `EngineEntry.model_type` and `EngineEntry.engine_type`, potentially switching from a `batched` engine (for LLMs) to a `vlm` engine (for vision models).

When a non-`None` value is present, the engine pool replaces the auto-detected type immediately without restarting the server.

## Summary

- **Automatic detection** in [`omlx/model_discovery.py`](https://github.com/jundot/omlx/blob/main/omlx/model_discovery.py) may misclassify multimodal models as standard LLMs.
- **Force the correct type** by setting `model_type_override` to supported values like `"vlm"` or `"llm"`.
- **Persist settings** in `~/.omlx/model_settings.json` using the Admin API or Python's `ModelSettingsManager`.
- **Apply immediately** through `EnginePool.apply_settings_overrides()` without server restarts.
- **Clear overrides** by setting the field to `null` to re-enable automatic detection.

## Frequently Asked Questions

### What happens if I set an invalid model type override?

oMLX validates the input against supported literals. If you provide an invalid type, the API returns a validation error and the engine pool ignores the request, retaining the previous auto-detected or manually set type.

### Does overriding the model type require restarting the oMLX server?

No. The `EnginePool.apply_settings_overrides()` method applies changes to the running engine pool immediately. The logs will confirm the override has been applied, and subsequent inference requests will use the correct engine type (e.g., switching from `batched` to `vlm`).

### Where are model type overrides stored permanently?

Overrides persist in the JSON configuration file at `~/.omlx/model_settings.json` (or your configured settings directory), managed by the `ModelSettingsManager` class in [`omlx/model_settings.py`](https://github.com/jundot/omlx/blob/main/omlx/model_settings.py). This ensures settings survive server restarts.

### Can I override the model type for any model architecture?

Yes, you can set any supported type literal for any loaded model, though forcing an incompatible type (such as treating a pure LLM as a VLM) may cause runtime errors when the engine expects vision-specific inputs that the model cannot process.