# LiteRT-LM Multimodal Input Types: Text, Images, and Audio Support > Explore LiteRT-LM's multimodal input capabilities. Learn how it processes text, images, and audio for advanced AI applications. - Repository: [google-ai-edge/LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) - Tags: getting-started - Published: 2026-04-06 --- **LiteRT-LM supports text, images, and audio as multimodal input types**, routing each modality through dedicated vision and audio backends while rejecting unsupported media types. LiteRT-LM, developed in the `google-ai-edge/LiteRT-LM` repository, is an inference engine designed to handle multiple data modalities within a single request. Understanding the supported **multimodal input types** and their specific formatting requirements is essential for building applications that combine vision, audio, and language understanding. ## Supported Multimodal Input Formats The engine recognizes three distinct content types, each identified through specific JSON schemas or inline parsing tags. ### Text Input Text is specified as a JSON object with the field `"type": "text"` containing the prompt string. According to the source code in [`python/litert_lm/interfaces.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/python/litert_lm/interfaces.py) (lines 70-76), the text content follows the standard message format: ```json {"type": "text", "text": "Describe the scene in detail."} ``` ### Image Input Images can be provided in two formats. You can use a structured JSON object with `"type": "image"` and a `"path"` field, or an inline tag embedded directly in the prompt string. The builder logic in [`schema/py/litertlm_builder.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/schema/py/litertlm_builder.py) (lines 108-112) and the C++ runtime in `runtime/engine/litert_lm_lib.cc` (lines 376-421) handle both approaches: **JSON format:** ```json {"type": "image", "path": "/path/to/image.jpg"} ``` **Inline tag format:** ``` [image:/path/to/image.jpg] ``` ### Audio Input Similar to images, audio supports both JSON and inline tagging. The schema definitions in [`schema/py/litertlm_builder.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/schema/py/litertlm_builder.py) (lines 104-108) define the audio input structure, while the runtime parser processes the tags: **JSON format:** ```json {"type": "audio", "path": "/path/to/audio.wav"} ``` **Inline tag format:** ``` [audio:/path/to/audio.wav] ``` ## Backend Routing and Hardware Acceleration Each **multimodal input type** requires a corresponding backend configuration in the `AbstractEngine` class defined in [`python/litert_lm/interfaces.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/python/litert_lm/interfaces.py). The engine validates that appropriate hardware accelerators are available before processing. - **Text**: Processed through the standard LLM `Backend` (CPU/GPU/NPU) - **Images**: Require `vision_backend` configuration - **Audio**: Require `audio_backend` configuration If you attempt to process an image without setting `vision_backend` or audio without `audio_backend`, the engine raises an error during initialization or inference. ## Implementation Details The multimodal pipeline relies on several key source files: - **[`schema/py/litertlm_builder.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/schema/py/litertlm_builder.py)**: Defines enum values for `AUDIO_*` and `VISION_*` model components that drive backend selection - **`runtime/engine/litert_lm_lib.cc`**: Contains the parser that extracts `[image:...]` and `[audio:...]` tags from prompt strings and builds the internal `content_list` - **`runtime/util/model_type_utils.cc`**: Stores token-string constants including ``, ``, ``, and `` that the model expects for delimiting modalities ## Working with Multimodal Inputs ### Using the Python API with Structured Content The following example demonstrates processing text, image, and audio simultaneously using the JSON content list format: ```python import litert_lm model_path = "/path/to/model.lm" image_path = "/path/to/photo.jpg" audio_path = "/path/to/speech.wav" with litert_lm.Engine( model_path, vision_backend=litert_lm.Backend.CPU, audio_backend=litert_lm.Backend.CPU, ) as engine: user_message = { "role": "user", "content": [ {"type": "text", "text": "What is shown in this picture?"}, {"type": "image", "path": image_path}, {"type": "audio", "path": audio_path}, {"type": "text", "text": "Also, transcribe the audio."}, ], } with engine.create_conversation() as convo: response = convo.send_message(user_message) print("Assistant:", response["content"][0]["text"]) ``` This pattern mirrors the reference implementation in [`python/litert_lm/examples/multimodal_main.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/python/litert_lm/examples/multimodal_main.py). ### Processing Inline Tags For simpler integration, you can embed media references directly in prompt strings: ```python prompt = ( "Describe this image: [image:/tmp/cat.jpg] " "and transcribe this audio: [audio:/tmp/meow.wav]" ) with litert_lm.Engine( model_path, vision_backend=litert_lm.Backend.CPU, audio_backend=litert_lm.Backend.CPU ) as engine: with engine.create_conversation() as convo: response = convo.send_message( {"role": "user", "content": [{"type": "text", "text": prompt}]} ) print(response["content"][0]["text"]) ``` The C++ parser in `runtime/engine/litert_lm_lib.cc` extracts these tags and converts them to the same internal representation used by the JSON API. ## Current Limitations While **LiteRT-LM multimodal input types** cover text, images, and audio, the engine does not currently support video inputs. Although video tags appear in comments and test files within the repository, they are not parsed as valid media types in the current release. ## Summary - **LiteRT-LM supports three multimodal input types**: text, images, and audio - **Input formats**: Structured JSON objects or inline tags (`[image:path]` and `[audio:path]`) - **Backend requirements**: Images require `vision_backend`, audio requires `audio_backend`, text uses the standard LLM backend - **Key implementation files**: [`interfaces.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/interfaces.py) defines the API, [`litertlm_builder.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/litertlm_builder.py) handles schema enums, `litert_lm_lib.cc` parses content - **No native video support** is currently implemented in the engine ## Frequently Asked Questions ### What file formats does LiteRT-LM accept for images and audio? The source code references file paths in the JSON `path` fields and inline tags, suggesting standard filesystem paths. While the specific codec support depends on the underlying vision and audio encoder implementations, typical formats like JPEG/PNG for images and WAV for audio are expected based on the examples in [`multimodal_main.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/multimodal_main.py) and the builder schema. ### Can I use video as an input type in LiteRT-LM? No, video is not a supported **multimodal input type** in LiteRT-LM. While the codebase contains references to video tags in comments and test files, the parser in `runtime/engine/litert_lm_lib.cc` does not recognize video as a valid media type, and no video backend exists in the current `AbstractEngine` implementation. ### How does LiteRT-LM route different modalities to specific hardware? According to [`python/litert_lm/interfaces.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/python/litert_lm/interfaces.py), the engine exposes `vision_backend` and `audio_backend` parameters separate from the standard LLM `Backend`. When you provide an image or audio input, the runtime checks for the corresponding backend configuration and routes the data through the specified hardware (CPU, GPU, or NPU) for encoding before passing tokens to the language model. ### What happens if I don't specify a vision or audio backend? The engine validates backend availability before processing **multimodal input types**. If you include an image in your request but fail to configure `vision_backend`, or include audio without `audio_backend`, LiteRT-LM will raise an error indicating that the required backend is not set for the respective media type.