Handling Multimodal Models with Image Input in llama.cpp: Complete Setup Guide

To process images with multimodal models in llama.cpp, you must load a vision-enabled GGUF model with its corresponding projector file (.mmproj), include the <__media__> placeholder token in your prompt, and submit base64-encoded image data via the OpenAI-compatible API or CLI.

llama.cpp supports vision-enabled large language models—such as Gemma-3-VL, Qwen-VL, and InternVL—through the multimodal library libmtmd. This system translates raw image data into the model’s latent space using an additional projector file, enabling you to analyze images alongside text prompts.

Prerequisites for Multimodal Image Input

Vision-Capable Model and Projector Files

Vision models require a multimodal GGUF file and a separate projector (.mmproj) that maps image embeddings to the text model's latent space. According to the source code in tools/server/README.md, you can either download a pre-quantized multimodal GGUF or supply a projector via the -mm flag. The high-level requirements are documented in docs/multimodal.md.

Build Requirements with libmtmd

The multimodal functionality resides in libmtmd, located under tools/mtmd/. This library is enabled by default when building the server with CMake. The core structure server_tokens in tools/server/server-common.h (lines 26-28) tracks multimodal requests through the has_mtmd boolean flag, which is set to true when image data is present.

Server Configuration Flags

When launching llama-server, specify the projector and image processing constraints:

• -mm <mmproj.gguf> or --mmproj-auto to load the vision projector
• --mmproj-offload to control GPU offloading of the vision encoder
• --image-min-tokens and --image-max-tokens to control token allocation per image

How Image Processing Works in llama.cpp

The MTMD Pipeline

When a request contains multimodal_data, the server creates a server_tokens instance with has_mtmd = true as defined in tools/server/server-common.h. Each base64-encoded image is decoded and passed to mtmd::process_image, which generates an input_chunk_ptr containing the processed visual embeddings.

Token Placeholder Mechanism

The prompt must include the default marker <__media__> (retrievable via mtmd_default_marker() from tools/mtmd/mtmd.h at line 87). During tokenization, this marker is replaced with LLAMA_TOKEN_NULL, while the map_idx_to_media structure in server-common.h (lines 30-38) links the token index to the corresponding image chunk.

KV Cache Considerations

Because image chunks occupy multiple positions in the key-value cache, the server disables speculative decoding for multimodal requests. This check appears in tools/server/server-context.cpp at lines 1392-1400, where the runtime rejects unsupported features for vision requests.

Implementing Image Input via the Server API

Starting the Server with Multimodal Support

Build and launch the server with a vision model:

cmake -B build
cmake --build build --target llama-server

./build/bin/llama-server \
    -hf ggml-org/gemma-3-4b-it-GGUF \
    --mmproj-auto \
    --image-min-tokens 128 \
    --image-max-tokens 1024

Verifying Model Capabilities

Before sending requests, query the model capabilities endpoint to confirm multimodal support:

curl http://localhost:8080/v1/models | jq '.data[0].capabilities'

The response must include "multimodal" in the capabilities array, as documented in the server endpoint specifications in tools/server/README.md.

Sending Base64-Encoded Images

Encode your image and construct an OpenAI-compatible JSON payload:

IMG_B64=$(base64 -w 0 sample.png)

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"gemma-3-4b-it\",
    \"messages\": [{
      \"role\": \"user\",
      \"content\": [
        {\"type\": \"text\", \"text\": \"Describe this: <__media__>\"},
        {\"type\": \"image_url\", \"image_url\": {\"url\": \"data:image/png;base64,$IMG_B64\"}}
      ]
    }],
    \"max_tokens\": 200
  }"

Command-Line Alternative with llama-mtmd-cli

For batch processing without an HTTP server, use the dedicated CLI wrapper implemented in tools/mtmd/mtmd-cli.cpp:

./build/bin/llama-mtmd-cli \
    -hf ggml-org/gemma-3-4b-it-GGUF \
    --image sample.png \
    --prompt "Explain what you see: <__media__>"

The CLI automatically handles base64 encoding and placeholder insertion.

Code Examples

C++ Implementation Using server_tokens

Developers integrating directly with the library can construct multimodal requests manually:

#include "mtmd.h"
#include "server-common.h"

// Process base64 image into MTMD chunk
std::string img_b64 = read_file_as_base64("sample.png");
auto chunk = mtmd::process_base64(img_b64);

// Build token list with placeholder
server_tokens toks;
toks = server_tokens(mtmd::input_chunks{chunk}, true); // has_mtmd = true
toks.push_back(llama_token_llama_eos);

This mirrors the logic in server-common.h lines 24-45 and the request parsing in server-context.cpp.

Python Client Example

For Python applications, use standard HTTP requests:

import base64, json, requests

with open("cat.png", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode()

payload = {
    "model": "gemma-3-4b-it",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "text", "text": "What is shown? <__media__>"},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img_b64}"}}
        ]
    }],
    "max_tokens": 150
}

r = requests.post("http://localhost:8080/v1/chat/completions",
                  headers={"Content-Type": "application/json"},
                  data=json.dumps(payload))
print(r.json()["choices"][0]["message"]["content"])

Summary

• Vision models require a projector: Load the .mmproj file alongside your GGUF using -mm or --mmproj-auto as documented in tools/server/README.md.
• Include the placeholder: The <__media__> marker (defined in tools/mtmd/mtmd.h) must appear in your prompt for each image.
• Encode images as base64: Send image data via the image_url field or multimodal_data array using standard base64 encoding.
• Verify capabilities: Check the /v1/models endpoint for the "multimodal" capability before submitting requests.
• Disable speculative decoding: The server automatically disables this feature for multimodal requests in tools/server/server-context.cpp to accommodate image token positions.

Frequently Asked Questions

What is the <__media__> token and why is it required?

The <__media__> string is the default marker token that tells libmtmd where to insert image embeddings in the token stream. According to tools/mtmd/mtmd.h at line 87, you can retrieve this marker via mtmd_default_marker(). The server replaces this placeholder with LLAMA_TOKEN_NULL and uses map_idx_to_media to link the position to actual image data.

Can I use speculative decoding with multimodal models?

No. As implemented in tools/server/server-context.cpp at lines 1392-1400, the server explicitly disables speculative decoding for requests where has_mtmd is true, because image chunks occupy multiple KV cache positions that conflict with speculative token prediction.

Where does the image processing happen in the codebase?

Image processing occurs in the libmtmd library under tools/mtmd/. The mtmd::process_image function converts raw pixels into model-compatible embeddings, while tools/server/server-common.h manages the token placement and the has_mtmd flag that tracks multimodal state.

How do I know if my GGUF model supports vision tasks?

Query the /v1/models or /models endpoint and inspect the capabilities array. Vision-capable models return "multimodal" in their capabilities list. You must also ensure you have loaded the corresponding .mmproj file, as the GGUF alone does not contain the vision projector weights.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}