How to Use the mlx_vlm.chat_ui Command: Launching Vision-Language Models with Gradio

The mlx_vlm.chat_ui command launches a Gradio-based web interface that allows you to chat with Vision-Language Models (VLMs) locally using the MLX runtime, supporting multimodal inputs including text and images.

The mlx_vlm.chat_ui entry point in the Blaizzy/mlx-vlm repository provides a self-contained way to interact with vision-capable language models on Apple Silicon or Linux machines. This command handles model loading, cache management, and streaming generation through an intuitive browser interface.

Installation Requirements

Before running the command, install the package with UI extras to ensure Gradio and related dependencies are available:

pip install "mlx-vlm[ui]"

The [ui] extra installs the necessary components to build the interactive interface defined in mlx_vlm/chat_ui.py.

Launching the Chat Interface

Basic Usage

To start the default chat UI with the built-in model, run:

mlx_vlm.chat_ui

This executes the main() function in mlx_vlm/chat_ui.py, which parses arguments via parse_arguments() (lines 23-34), initializes a global ModelState object (lines 36-45), and loads the default model qnguyen3/nanoLLaVA. The interface automatically opens at http://127.0.0.1:7860 in your default browser.

Specifying a Custom Model

To load a specific vision-language model from Hugging Face or a local directory:

mlx_vlm.chat_ui --model liuhaotian/llava-v1.5-7b

The ModelState.load() method (invoked at lines 71-74) handles the model initialization. The UI displays ✓ Loaded next to the model name once loading completes.

Core Interface Components

Model Selection and Local Cache

The interface includes a dropdown populated by get_cached_vlm_models() (lines 82-136 of mlx_vlm/chat_ui.py), which scans your local Hugging Face cache for compatible vision models. The refresh button triggers refresh_model_list() to update this cache without restarting the application.

Generation Parameters

The UI provides sliders to configure inference behavior:

• Temperature: Controls randomness in generation
• Max Tokens: Limits response length
• Top-p: Nucleus sampling parameter
• Repetition Penalty: Reduces token repetition

These values are collected into gen_kwargs and passed to stream_generate() from mlx_vlm/generate.py during the chat loop (lines 98-112).

Multimodal Chat Capabilities

The chat component accepts both text and image inputs. When you upload an image, the extract_image_from_message() function processes the file, and the system constructs message JSON using get_message_json() from mlx_vlm/prompt_utils.py. The chat() function (lines 97-112) handles the streaming loop, yielding tokens in real-time via stream_generate().

Session Persistence Features

The interface persists user preferences across browser sessions:

• Dark Mode: Toggles between themes using JavaScript blocks defined in dark_mode_js and toggle_dark_js (lines 55-73)
• Model Selection: Saves the last used model to localStorage via save_model_js and load_model_js (lines 74-89)

Programmatic Usage

Running from Python Scripts

You can launch the UI programmatically without the console command:

from mlx_vlm.chat_ui import main

# Launch the Gradio interface directly

main()

This bypasses the CLI argument parsing and uses default configurations defined in the parse_arguments() function.

Implementing Custom Streaming

For applications requiring manual control over generation, replicate the UI's streaming logic:

from mlx_vlm.generate import stream_generate
from mlx_vlm.chat_ui import state, extract_image_from_message, get_message_json, get_chat_template

def custom_generate(prompt_text, image_path=None):
    image = [image_path] if image_path else None
    messages = [get_message_json(
        state.config["model_type"], 
        prompt_text,
        role="user", 
        skip_image_token=False,
        num_images=1 if image else 0
    )]
    
    prompt = get_chat_template(
        state.processor, 
        messages, 
        add_generation_prompt=True
    )

    for chunk in stream_generate(
        state.model, 
        state.processor,
        prompt, 
        image=image,
        max_tokens=1024, 
        temperature=0.1
    ):
        print(chunk.text, end='', flush=True)

This mirrors the streaming implementation in mlx_vlm/chat_ui.py lines 98-112, utilizing the same ModelState object and generation utilities.

Key Implementation Files

• mlx_vlm/chat_ui.py: Main entry point containing parse_arguments(), ModelState class, and Gradio layout (lines 1-221)
• mlx_vlm/generate.py: Core inference engine with stream_generate() function
• mlx_vlm/utils.py: Helper functions for configuration and image processor loading
• mlx_vlm/vision_cache.py: Caching mechanism for vision encoder features
• mlx_vlm/prompt_utils.py: Chat template construction and message formatting utilities
• pyproject.toml: Defines the console script mapping mlx_vlm.chat_ui to mlx_vlm.chat_ui:main (lines 32-34)

Summary

• The mlx_vlm.chat_ui command starts a Gradio web interface for vision-language model interaction through the MLX runtime.
• Model management occurs via the ModelState class, which handles loading, caching, and memory cleanup when switching models.
• Multimodal support allows simultaneous text and image inputs, processed through extract_image_from_message() and the streaming generator.
• Configuration persistence uses browser localStorage to remember theme preferences and selected models across sessions.
• Programmatic access is available by importing main() from mlx_vlm.chat_ui for integration into larger Python applications.

Frequently Asked Questions

How do I stop generation mid-stream in the chat UI?

The interface includes a Stop button that sets a generation flag via stop_generation events. When clicked, it triggers stop_generating() (lines 35-38 of mlx_vlm/chat_ui.py), which aborts the stream_generate() loop immediately without reloading the model or interface.

Can I use mlx_vlm.chat_ui with models not in the Hugging Face cache?

Yes. The model dropdown accepts manual input of any Hugging Face repository ID or local path. When you enter a custom model name, the load_model_by_name() function (lines 40-58) clears the current model from memory and loads the new one, regardless of whether it appears in the cached model list.

What is the default model used when no --model flag is specified?

The default model is qnguyen3/nanoLLaVA, specified in the argument parser at lines 23-34 of mlx_vlm/chat_ui.py. This lightweight model loads automatically when you run mlx_vlm.chat_ui without arguments, providing immediate functionality while allowing you to switch to larger models through the interface.

Why does the UI scan for cached models on startup?

The get_cached_vlm_models() function (lines 82-136) scans the local Hugging Face cache directory to populate the model dropdown with available vision-capable models. This eliminates the need to manually type repository IDs for previously downloaded models, though you can still enter custom paths or IDs in the editable dropdown field.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}