How to Configure Backend Options for LiteRT-LM: CPU, GPU, Vision, and Audio Backends

LiteRT-LM configures hardware acceleration through the Backend enum, letting you set the main inference engine to CPU or GPU via CLI flags or Python arguments, while vision and audio encoders are controlled separately through the Python API only.

LiteRT-LM is an inference framework for multimodal language models that requires explicit hardware configuration to optimize performance. Whether you are running inference on edge devices or packaging models for specific deployment targets, understanding how to configure backend options for LiteRT-LM ensures you leverage the right compute resources for text, vision, and audio processing.

The Backend Enum Definition

All hardware backend specifications in LiteRT-LM rely on the Backend enum defined in python/litert_lm/interfaces.py. This enumeration maps hardware targets to integer values used by the underlying C++ engine.

class Backend(enum.Enum):
    """Hardware backends for LiteRT‑LM."""
    UNSPECIFIED = 0
    CPU = 3
    GPU = 4

The UNSPECIFIED value tells the engine to auto-select the backend, while CPU and GPU force execution to the respective hardware. When configuring backend options for LiteRT-LM, you reference these enum members in Python or their string equivalents in the CLI.

Configuring the Inference Backend

The inference backend determines whether the core language model runs on CPU or GPU. You can configure this via command-line interface or Python API.

CLI Configuration

The CLI provides the --backend (or -b) flag, defined in python/litert_lm_cli/main.py, which accepts the strings "cpu" or "gpu". The flag is available for both run and benchmark commands.

litert-lm run ./my_model.litertlm --backend=gpu

Behind the scenes, the model.py module converts this string to the enum using the _parse_backend function:

def _parse_backend(backend: str) -> litert_lm.Backend:
    backend_lower = backend.lower()
    if backend_lower == "gpu":
        return litert_lm.Backend.GPU
    return litert_lm.Backend.CPU

This function is located in python/litert_lm_cli/model.py and defaults to Backend.CPU if no match is found.

Python API Configuration

When initializing the inference engine programmatically, pass the backend argument directly to the Engine constructor:

import litert_lm

engine = litert_lm.Engine(
    model_path="my_model.litertlm",
    backend=litert_lm.Backend.GPU
)

If omitted, the Python API defaults to CPU execution.

Setting Vision and Audio Encoder Backends

Unlike the main inference backend, vision and audio encoder backends are only configurable through the Python API. The AbstractEngine class in python/litert_lm/interfaces.py exposes these as optional dataclass fields:

vision_backend: Backend | None = None
audio_backend: Backend | None = None

This design allows you to offload specific encoders to different hardware than the main model. For example, you might run the text model on CPU while processing vision tensors on GPU to balance memory and latency.

To configure these backends, pass the enum values when creating the Engine:

engine = litert_lm.Engine(
    model_path="multimodal_model.litertlm",
    backend=litert_lm.Backend.CPU,          # Inference on CPU

    vision_backend=litert_lm.Backend.GPU,     # Image encoder on GPU

    audio_backend=litert_lm.Backend.CPU      # Audio encoder on CPU

)

If you omit vision_backend or audio_backend, they default to Backend.UNSPECIFIED, allowing the engine to auto-select the appropriate hardware.

Enforcing Backend Constraints in Model Packaging

When building LiteRT-LM model files (.litertlm), you can embed backend constraints that restrict which hardware the model is allowed to run on. This is validated during model construction in schema/py/litertlm_builder.py via the _validate_backend_constraints function.

def _validate_backend_constraints(backend_constraint: str) -> None:
    backends = [b.strip().lower() for b in backend_constraint.split(",")]
    valid_backends = set(Backend)
    for backend in backends:
        if backend not in valid_backends:
            raise ValueError(...)

To restrict a model to GPU only during packaging:

builder.add_tflite_model(
    model_path="my_model.tflite",
    backend_constraint="gpu",   # Only GPU allowed at runtime

)

If a user attempts to load this model with --backend=cpu, the runtime will reject the configuration, preventing incompatible execution paths.

Complete Working Examples

Multimodal Inference with Mixed Backends

This example demonstrates running a multimodal conversation where vision processing happens on GPU while audio and text inference remain on CPU:

import litert_lm

engine = litert_lm.Engine(
    model_path="my_model.litertlm",
    backend=litert_lm.Backend.CPU,
    vision_backend=litert_lm.Backend.GPU,
    audio_backend=litert_lm.Backend.CPU
)

with engine.create_conversation() as conv:
    message = {
        "role": "user",
        "content": [
            {"type": "image", "path": "scene.jpg"},
            {"type": "audio", "path": "speech.wav"},
            {"type": "text", "text": "Describe the scene and transcribe the audio."}
        ],
    }
    response = conv.send_message(message)
    print(response["content"][0]["text"])

Benchmarking on Specific Hardware

To benchmark model performance on GPU from the command line:

litert-lm benchmark ./my_model.litertlm --backend=gpu --num_iterations=100

Running Audio-Only Examples

The repository includes a multimodal example that demonstrates audio backend configuration:

python -m litert_lm.examples.multimodal_main \
    --model_path=my_model.litertlm \
    --audio_path=sample.wav

According to the source in python/litert_lm/examples/multimodal_main.py, this example internally configures audio_backend=Backend.CPU when initializing the engine.

Summary

• Backend Enum: Hardware targets are defined in python/litert_lm/interfaces.py as UNSPECIFIED, CPU, and GPU.
• Inference Backend: Configure via CLI (--backend cpu|gpu) handled by python/litert_lm_cli/model.py, or via Python Engine(backend=...) argument.
• Encoder Backends: Set vision_backend and audio_backend arguments in the Python API only; these default to UNSPECIFIED if not provided.
• Model Constraints: Enforce hardware restrictions during model packaging in schema/py/litertlm_builder.py using the backend_constraint parameter.
• Defaults: All backends default to CPU (inference) or auto-selection (encoders) when not explicitly configured.

Frequently Asked Questions

Can I use different backends for inference and encoders in the same model?

Yes. As implemented in python/litert_lm/interfaces.py, the Engine class accepts separate backend, vision_backend, and audio_backend arguments. This allows you to run the main transformer on CPU while offloading vision encoders to GPU, which is useful for balancing memory constraints and compute latency.

What happens if I set vision_backend to UNSPECIFIED?

Setting vision_backend or audio_backend to Backend.UNSPECIFIED (or omitting the argument) allows the LiteRT-LM engine to automatically select the appropriate hardware based on availability and model requirements. This differs from explicitly setting Backend.CPU or Backend.GPU, which forces execution to that specific hardware.

How do I prevent my model from running on CPU?

During model packaging in schema/py/litertlm_builder.py, specify backend_constraint="gpu" when calling add_tflite_model(). This embeds a whitelist into the model file that causes the runtime to reject CPU initialization attempts, ensuring the model only executes on compatible GPU hardware.

Why can't I set vision or audio backends via the CLI?

The CLI interface in python/litert_lm_cli/main.py only exposes the --backend flag for the main inference engine. Vision and audio encoders require the Python API because they need to be configured as part of the Engine dataclass instantiation, as shown in the AbstractEngine definition in python/litert_lm/interfaces.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}