LiteRT-LM Multimodal Input Types: Text, Images, and Audio Support

LiteRT-LM supports text, images, and audio as multimodal input types, routing each modality through dedicated vision and audio backends while rejecting unsupported media types.

LiteRT-LM, developed in the google-ai-edge/LiteRT-LM repository, is an inference engine designed to handle multiple data modalities within a single request. Understanding the supported multimodal input types and their specific formatting requirements is essential for building applications that combine vision, audio, and language understanding.

Supported Multimodal Input Formats

The engine recognizes three distinct content types, each identified through specific JSON schemas or inline parsing tags.

Text Input

Text is specified as a JSON object with the field "type": "text" containing the prompt string. According to the source code in python/litert_lm/interfaces.py (lines 70-76), the text content follows the standard message format:

{"type": "text", "text": "Describe the scene in detail."}

Image Input

Images can be provided in two formats. You can use a structured JSON object with "type": "image" and a "path" field, or an inline tag embedded directly in the prompt string. The builder logic in schema/py/litertlm_builder.py (lines 108-112) and the C++ runtime in runtime/engine/litert_lm_lib.cc (lines 376-421) handle both approaches:

JSON format:

{"type": "image", "path": "/path/to/image.jpg"}

Inline tag format:

[image:/path/to/image.jpg]

Audio Input

Similar to images, audio supports both JSON and inline tagging. The schema definitions in schema/py/litertlm_builder.py (lines 104-108) define the audio input structure, while the runtime parser processes the tags:

JSON format:

{"type": "audio", "path": "/path/to/audio.wav"}

Inline tag format:

[audio:/path/to/audio.wav]

Backend Routing and Hardware Acceleration

Each multimodal input type requires a corresponding backend configuration in the AbstractEngine class defined in python/litert_lm/interfaces.py. The engine validates that appropriate hardware accelerators are available before processing.

• Text: Processed through the standard LLM Backend (CPU/GPU/NPU)
• Images: Require vision_backend configuration
• Audio: Require audio_backend configuration

If you attempt to process an image without setting vision_backend or audio without audio_backend, the engine raises an error during initialization or inference.

Implementation Details

The multimodal pipeline relies on several key source files:

• schema/py/litertlm_builder.py: Defines enum values for AUDIO_* and VISION_* model components that drive backend selection
• runtime/engine/litert_lm_lib.cc: Contains the parser that extracts [image:...] and [audio:...] tags from prompt strings and builds the internal content_list
• runtime/util/model_type_utils.cc: Stores token-string constants including <start_of_image>, <end_of_image>, <start_of_audio>, and <end_of_audio> that the model expects for delimiting modalities

Working with Multimodal Inputs

Using the Python API with Structured Content

The following example demonstrates processing text, image, and audio simultaneously using the JSON content list format:

import litert_lm

model_path = "/path/to/model.lm"
image_path = "/path/to/photo.jpg"
audio_path = "/path/to/speech.wav"

with litert_lm.Engine(
    model_path,
    vision_backend=litert_lm.Backend.CPU,
    audio_backend=litert_lm.Backend.CPU,
) as engine:
    user_message = {
        "role": "user",
        "content": [
            {"type": "text", "text": "What is shown in this picture?"},
            {"type": "image", "path": image_path},
            {"type": "audio", "path": audio_path},
            {"type": "text", "text": "Also, transcribe the audio."},
        ],
    }

    with engine.create_conversation() as convo:
        response = convo.send_message(user_message)
        print("Assistant:", response["content"][0]["text"])

This pattern mirrors the reference implementation in python/litert_lm/examples/multimodal_main.py.

Processing Inline Tags

For simpler integration, you can embed media references directly in prompt strings:

prompt = (
    "Describe this image: [image:/tmp/cat.jpg] "
    "and transcribe this audio: [audio:/tmp/meow.wav]"
)

with litert_lm.Engine(
    model_path, 
    vision_backend=litert_lm.Backend.CPU,
    audio_backend=litert_lm.Backend.CPU
) as engine:
    with engine.create_conversation() as convo:
        response = convo.send_message(
            {"role": "user", "content": [{"type": "text", "text": prompt}]}
        )
        print(response["content"][0]["text"])

The C++ parser in runtime/engine/litert_lm_lib.cc extracts these tags and converts them to the same internal representation used by the JSON API.

Current Limitations

While LiteRT-LM multimodal input types cover text, images, and audio, the engine does not currently support video inputs. Although video tags appear in comments and test files within the repository, they are not parsed as valid media types in the current release.

Summary

• LiteRT-LM supports three multimodal input types: text, images, and audio
• Input formats: Structured JSON objects or inline tags ([image:path] and [audio:path])
• Backend requirements: Images require vision_backend, audio requires audio_backend, text uses the standard LLM backend
• Key implementation files: interfaces.py defines the API, litertlm_builder.py handles schema enums, litert_lm_lib.cc parses content
• No native video support is currently implemented in the engine

Frequently Asked Questions

What file formats does LiteRT-LM accept for images and audio?

The source code references file paths in the JSON path fields and inline tags, suggesting standard filesystem paths. While the specific codec support depends on the underlying vision and audio encoder implementations, typical formats like JPEG/PNG for images and WAV for audio are expected based on the examples in multimodal_main.py and the builder schema.

Can I use video as an input type in LiteRT-LM?

No, video is not a supported multimodal input type in LiteRT-LM. While the codebase contains references to video tags in comments and test files, the parser in runtime/engine/litert_lm_lib.cc does not recognize video as a valid media type, and no video backend exists in the current AbstractEngine implementation.

How does LiteRT-LM route different modalities to specific hardware?

According to python/litert_lm/interfaces.py, the engine exposes vision_backend and audio_backend parameters separate from the standard LLM Backend. When you provide an image or audio input, the runtime checks for the corresponding backend configuration and routes the data through the specified hardware (CPU, GPU, or NPU) for encoding before passing tokens to the language model.

What happens if I don't specify a vision or audio backend?

The engine validates backend availability before processing multimodal input types. If you include an image in your request but fail to configure vision_backend, or include audio without audio_backend, LiteRT-LM will raise an error indicating that the required backend is not set for the respective media type.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}