# How to Use the mlx_vlm.chat_ui Command: Launching Vision-Language Models with Gradio

> Launch Vision-Language Models locally with mlx_vlm.chat_ui. Explore this Gradio interface for multimodal text and image chatting using the MLX runtime.

- Repository: [Prince Canuma/mlx-vlm](https://github.com/Blaizzy/mlx-vlm)
- Tags: how-to-guide
- Published: 2026-04-05

---

**The `mlx_vlm.chat_ui` command launches a Gradio-based web interface that allows you to chat with Vision-Language Models (VLMs) locally using the MLX runtime, supporting multimodal inputs including text and images.**

The `mlx_vlm.chat_ui` entry point in the [Blaizzy/mlx-vlm](https://github.com/Blaizzy/mlx-vlm) repository provides a self-contained way to interact with vision-capable language models on Apple Silicon or Linux machines. This command handles model loading, cache management, and streaming generation through an intuitive browser interface.

## Installation Requirements

Before running the command, install the package with UI extras to ensure Gradio and related dependencies are available:

```bash
pip install "mlx-vlm[ui]"

```

The `[ui]` extra installs the necessary components to build the interactive interface defined in [`mlx_vlm/chat_ui.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/chat_ui.py).

## Launching the Chat Interface

### Basic Usage

To start the default chat UI with the built-in model, run:

```bash
mlx_vlm.chat_ui

```

This executes the `main()` function in [`mlx_vlm/chat_ui.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/chat_ui.py), which parses arguments via `parse_arguments()` (lines 23-34), initializes a global `ModelState` object (lines 36-45), and loads the default model `qnguyen3/nanoLLaVA`. The interface automatically opens at `http://127.0.0.1:7860` in your default browser.

### Specifying a Custom Model

To load a specific vision-language model from Hugging Face or a local directory:

```bash
mlx_vlm.chat_ui --model liuhaotian/llava-v1.5-7b

```

The `ModelState.load()` method (invoked at lines 71-74) handles the model initialization. The UI displays **✓ Loaded** next to the model name once loading completes.

## Core Interface Components

### Model Selection and Local Cache

The interface includes a dropdown populated by `get_cached_vlm_models()` (lines 82-136 of [`mlx_vlm/chat_ui.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/chat_ui.py)), which scans your local Hugging Face cache for compatible vision models. The refresh button triggers `refresh_model_list()` to update this cache without restarting the application.

### Generation Parameters

The UI provides sliders to configure inference behavior:

- **Temperature**: Controls randomness in generation
- **Max Tokens**: Limits response length
- **Top-p**: Nucleus sampling parameter
- **Repetition Penalty**: Reduces token repetition

These values are collected into `gen_kwargs` and passed to `stream_generate()` from [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) during the chat loop (lines 98-112).

### Multimodal Chat Capabilities

The chat component accepts both text and image inputs. When you upload an image, the `extract_image_from_message()` function processes the file, and the system constructs message JSON using `get_message_json()` from [`mlx_vlm/prompt_utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/prompt_utils.py). The `chat()` function (lines 97-112) handles the streaming loop, yielding tokens in real-time via `stream_generate()`.

### Session Persistence Features

The interface persists user preferences across browser sessions:

- **Dark Mode**: Toggles between themes using JavaScript blocks defined in `dark_mode_js` and `toggle_dark_js` (lines 55-73)
- **Model Selection**: Saves the last used model to `localStorage` via `save_model_js` and `load_model_js` (lines 74-89)

## Programmatic Usage

### Running from Python Scripts

You can launch the UI programmatically without the console command:

```python
from mlx_vlm.chat_ui import main

# Launch the Gradio interface directly

main()

```

This bypasses the CLI argument parsing and uses default configurations defined in the `parse_arguments()` function.

### Implementing Custom Streaming

For applications requiring manual control over generation, replicate the UI's streaming logic:

```python
from mlx_vlm.generate import stream_generate
from mlx_vlm.chat_ui import state, extract_image_from_message, get_message_json, get_chat_template

def custom_generate(prompt_text, image_path=None):
    image = [image_path] if image_path else None
    messages = [get_message_json(
        state.config["model_type"], 
        prompt_text,
        role="user", 
        skip_image_token=False,
        num_images=1 if image else 0
    )]
    
    prompt = get_chat_template(
        state.processor, 
        messages, 
        add_generation_prompt=True
    )

    for chunk in stream_generate(
        state.model, 
        state.processor,
        prompt, 
        image=image,
        max_tokens=1024, 
        temperature=0.1
    ):
        print(chunk.text, end='', flush=True)

```

This mirrors the streaming implementation in [`mlx_vlm/chat_ui.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/chat_ui.py) lines 98-112, utilizing the same `ModelState` object and generation utilities.

## Key Implementation Files

- **[`mlx_vlm/chat_ui.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/chat_ui.py)**: Main entry point containing `parse_arguments()`, `ModelState` class, and Gradio layout (lines 1-221)
- **[`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py)**: Core inference engine with `stream_generate()` function
- **[`mlx_vlm/utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/utils.py)**: Helper functions for configuration and image processor loading
- **[`mlx_vlm/vision_cache.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/vision_cache.py)**: Caching mechanism for vision encoder features
- **[`mlx_vlm/prompt_utils.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/prompt_utils.py)**: Chat template construction and message formatting utilities
- **[`pyproject.toml`](https://github.com/Blaizzy/mlx-vlm/blob/main/pyproject.toml)**: Defines the console script mapping `mlx_vlm.chat_ui` to `mlx_vlm.chat_ui:main` (lines 32-34)

## Summary

- The **`mlx_vlm.chat_ui` command** starts a Gradio web interface for vision-language model interaction through the MLX runtime.
- **Model management** occurs via the `ModelState` class, which handles loading, caching, and memory cleanup when switching models.
- **Multimodal support** allows simultaneous text and image inputs, processed through `extract_image_from_message()` and the streaming generator.
- **Configuration persistence** uses browser `localStorage` to remember theme preferences and selected models across sessions.
- **Programmatic access** is available by importing `main()` from `mlx_vlm.chat_ui` for integration into larger Python applications.

## Frequently Asked Questions

### How do I stop generation mid-stream in the chat UI?

The interface includes a **Stop** button that sets a generation flag via `stop_generation` events. When clicked, it triggers `stop_generating()` (lines 35-38 of [`mlx_vlm/chat_ui.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/chat_ui.py)), which aborts the `stream_generate()` loop immediately without reloading the model or interface.

### Can I use mlx_vlm.chat_ui with models not in the Hugging Face cache?

Yes. The model dropdown accepts manual input of any Hugging Face repository ID or local path. When you enter a custom model name, the `load_model_by_name()` function (lines 40-58) clears the current model from memory and loads the new one, regardless of whether it appears in the cached model list.

### What is the default model used when no --model flag is specified?

The default model is **`qnguyen3/nanoLLaVA`**, specified in the argument parser at lines 23-34 of [`mlx_vlm/chat_ui.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/chat_ui.py). This lightweight model loads automatically when you run `mlx_vlm.chat_ui` without arguments, providing immediate functionality while allowing you to switch to larger models through the interface.

### Why does the UI scan for cached models on startup?

The `get_cached_vlm_models()` function (lines 82-136) scans the local Hugging Face cache directory to populate the model dropdown with available vision-capable models. This eliminates the need to manually type repository IDs for previously downloaded models, though you can still enter custom paths or IDs in the editable dropdown field.