How to Customize Chat Template Kwargs Per Model in oMLX: Complete Guide

oMLX allows you to define model-specific Jinja template parameters through chat_template_kwargs in ModelSettings, merge them with request-level overrides, and protect sensitive keys using forced_ct_kwargs to enforce server-side policies.

oMLX provides a flexible mechanism for customizing how prompts are formatted before being sent to the model. By configuring chat template kwargs per model, operators can set default behaviors—such as enabling reasoning modes or setting custom tokens—while still allowing selective client overrides.

Understanding Chat Template Kwargs Storage

In oMLX, per-model chat template configurations are stored in the ModelSettings dataclass defined in omlx/model_settings.py. This class contains two critical fields for template customization:

• chat_template_kwargs: A dictionary of default key-value pairs passed to the Jinja template when rendering prompts
• forced_ct_kwargs: A list of keys that cannot be overridden by client requests, ensuring policy compliance

The ModelSettingsManager class in the same file persists these settings to a model_settings.json file, making configurations durable across server restarts.

Configuring Model-Level Chat Template Kwargs

Programmatic Setup

You can define template kwargs directly in Python before starting the server. This approach is ideal for initial configuration or automated deployments:

from pathlib import Path
from omlx.model_settings import ModelSettings, ModelSettingsManager

# Initialize the settings manager (typically done once at server startup)

mgr = ModelSettingsManager(base_path=Path("/var/omlx"))

# Create settings for a specific model

settings = ModelSettings(
    chat_template_kwargs={
        "enable_thinking": True,
        "reasoning_effort": "high",
        "custom_system_prompt": "You are a helpful assistant."
    },
    forced_ct_kwargs=["reasoning_effort"]  # Clients cannot override this key

)

mgr.set_settings("gemma4", settings)

When persisted, these settings are stored in model_settings.json and loaded automatically when the model is requested.

Via Admin API

For runtime adjustments, oMLX exposes an HTTP endpoint in omlx/admin/routes.py that allows you to update template kwargs without restarting the server:

curl -X PATCH "http://localhost:8000/admin/settings/gemma4" \
     -H "Content-Type: application/json" \
     -d '{
           "chat_template_kwargs": {
               "enable_thinking": false,
               "reasoning_effort": "medium"
           },
           "forced_ct_kwargs": ["reasoning_effort"]
         }'

The admin handler updates the in-memory ModelSettings object and persists the change to disk immediately.

Request-Level Overrides and Merge Logic

When a client sends a chat completion request, oMLX merges the model-level defaults with request-specific overrides. The merge logic, implemented in omlx/server.py, follows a strict hierarchy:

1. Base layer: Model-level chat_template_kwargs from ModelSettings
2. Policy layer: The enable_thinking boolean from model settings (if not None) overrides any kwargs value
3. Client layer: Request-level chat_template_kwargs are applied, except for keys listed in forced_ct_kwargs

# Simplified merge logic from omlx/server.py (lines 2082-2095)

ms = settings_manager.get_settings(model_id)
merged_ct_kwargs = {}
forced_keys = set(ms.forced_ct_kwargs or [])

if ms.chat_template_kwargs:
    merged_ct_kwargs.update(ms.chat_template_kwargs)  # Model defaults

if ms.enable_thinking is not None:
    merged_ct_kwargs["enable_thinking"] = ms.enable_thinking  # Policy override

if request.chat_template_kwargs:
    for k, v in request.chat_template_kwargs.items():
        if k not in forced_keys:
            merged_ct_kwargs[k] = v  # Client override (respecting forced keys)

The final merged_ct_kwargs dictionary is then passed to the engine in omlx/engine/batched.py (and similar files like vlm.py and dflash.py) when calling tokenizer.apply_chat_template.

Example Request with Overrides

POST /v1/chat/completions
{
  "model": "gemma4",
  "messages": [{ "role": "user", "content": "Explain quantum tunneling." }],
  "chat_template_kwargs": {
    "reasoning_effort": "low",
    "custom_note": "demo"
  }
}

Result: Because reasoning_effort is listed in forced_ct_kwargs, the request's "low" value is ignored, retaining the model's "high" setting. However, custom_note is successfully merged into the final kwargs.

Protecting Sensitive Template Parameters

The forced_ct_kwargs field serves as a security mechanism to prevent clients from modifying critical template behavior. When a key appears in this list, the server guarantees that only the model-level value from chat_template_kwargs (or the enable_thinking toggle) will be used, regardless of what the client requests.

This is particularly useful for:

• Enforcing specific reasoning effort levels across all interactions
• Locking system prompts or persona definitions
• Maintaining compliance with usage policies that require specific template formatting

Using Kwargs in Jinja Templates

To consume these parameters, your model's chat_template.jinja file should reference the variables passed through chat_template_kwargs:

{% if enable_thinking %}
<start_thinking>
{% endif %}

{{ messages[0].content }}

{% if enable_thinking %}
</end_thinking>
{{ reasoning_effort | default('low') }}
{% endif %}

{{ custom_note | default('') }}

When the engine renders this template, it receives the merged dictionary containing enable_thinking, reasoning_effort, and any client-provided overrides, allowing dynamic prompt construction based on per-model configuration.

Summary

• Store defaults in ModelSettings.chat_template_kwargs via omlx/model_settings.py for persistent per-model behavior
• Force immutability by adding sensitive keys to forced_ct_kwargs to prevent client overrides
• Merge hierarchy follows: model defaults → enable_thinking policy → filtered request kwargs (excluding forced keys)
• Access anywhere in your Jinja templates by referencing the kwargs as variables, enabling dynamic prompt formatting
• Update dynamically through the Admin API in omlx/admin/routes.py or programmatically via ModelSettingsManager

Frequently Asked Questions

How do I prevent clients from changing specific chat template parameters?

Add the parameter name to the forced_ct_kwargs list in your ModelSettings. Any key listed there will be ignored when processing client requests, ensuring the model-level value remains in effect. This is implemented in the merge logic within omlx/server.py where request kwargs are filtered against the forced keys set.

Can I set different chat template kwargs for different models on the same server?

Yes. The ModelSettingsManager stores a separate ModelSettings object for each model ID in model_settings.json. When you call mgr.set_settings("model_name", settings), the configuration is scoped specifically to that model identifier, allowing completely independent template configurations per model.

What happens if enable_thinking is set in both chat_template_kwargs and as a standalone field?

The standalone enable_thinking boolean in ModelSettings takes precedence. According to the merge logic in omlx/server.py, after loading the base chat_template_kwargs, the code explicitly checks if ms.enable_thinking is not None: and overwrites the dictionary value, ensuring the top-level toggle always controls the final parameter sent to the template.

Where are the merged kwargs actually used in the inference pipeline?

After merging in omlx/server.py, the final dictionary is passed to engine.count_chat_tokens() and the generation methods in omlx/engine/batched.py. The engine then passes these kwargs directly to tokenizer.apply_chat_template when formatting messages for the model, making them available as variables within the Jinja template context.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}