LiteRT Known Issues and Limitations: Critical Constraints for Edge AI Deployment

LiteRT currently supports only stateless LSTM conversion, enforces strict GPU operator whitelists, and requires strict API version matching for vendor NPUs, necessitating manual fallback strategies and careful model architecture design.

LiteRT (the lightweight TensorFlow Lite runtime) provides a unified inference engine for Edge AI across CPU, GPU, and vendor-specific NPUs. While it offers broad operator support and hardware acceleration, developers must navigate specific LiteRT known issues and limitations regarding model conversion, delegate coverage, and backend compatibility to avoid runtime failures in production environments.

Model Conversion Constraints

Stateless LSTM Only

LiteRT’s TensorFlow Lite converter can only transform stateless Keras LSTM layers, which is the default Keras behavior. According to tflite/g3doc/models/convert/rnn.md (lines 90-98), stateful LSTM conversion is not yet implemented. This means models relying on hidden-state persistence across invocations must be rewritten to manage state manually in client code or use stateless layers with explicit state inputs.

To work around this limitation, expose the previous hidden and cell states as additional inputs to your model:

import tensorflow as tf

lstm = tf.keras.layers.LSTM(64, return_sequences=False, stateful=False)

@tf.function(input_signature=[tf.TensorSpec([1, 10, 32], tf.float32)])
def run_one_step(x, h, c):
    out, new_h, new_c = lstm(x, initial_state=[h, c])
    return out, new_h, new_c

Bidirectional LSTM Decomposition

As documented in tflite/g3doc/models/convert/rnn.md (lines 96-98), bidirectional LSTM layers are lowered to two separate UnidirectionalSequenceLSTM operations rather than a single fused BidirectionalSequenceLSTM op. This decomposition increases graph size and may introduce performance penalties compared to a native bidirectional implementation.

GPU Delegate Limitations

Operator Whitelist and Version Constraints

The GPU delegate accelerates only a specific subset of TensorFlow operations. As detailed in tflite/g3doc/performance/gpu.md (lines 29-57), supported ops include ADD, CONV_2D, LSTM v2 (Basic LSTM only), PRELU, and others. Ops outside this whitelist execute on CPU, potentially causing significant latency from CPU/GPU synchronization overhead.

Furthermore, according to lines 58-60 of the same file, the GPU delegate supports only version 1 of each operator by default. Higher-version kernels are enabled automatically only when the model uses quantization (e.g., ADD v2).

CPU Fallback Warnings

When the GPU delegate cannot handle an operation, the interpreter emits a warning:

WARNING: op code #42 cannot be handled by this delegate.

As noted in tflite/g3doc/performance/gpu.md (lines 62-78), this indicates the graph has been split between GPU and CPU execution, often reducing overall throughput.

TensorFlow Select Ops Restrictions

When using SELECT_TF_OPS to include TensorFlow operations not natively supported in LiteRT, be aware that some TensorFlow ops do not support the full range of data types that core TensorFlow Lite ops do. According to tflite/g3doc/guide/ops_select.md (section "Known limitations", lines 86-88), this limitation can lead to conversion errors or silent fallbacks to less efficient implementations.

Android Deployment Limitations

Play Services Runtime Subset

The TensorFlow Lite runtime distributed via Google Play Services does not expose all operators, custom delegates, or experimental features available in the open-source LiteRT package. As stated in tflite/g3doc/android/play_services.md (line 38), models requiring advanced features must ship with a bundled LiteRT wheel rather than relying on the Play Services version.

Acceleration Service Op Restrictions

The on-device acceleration service (for Edge TPU and similar hardware) maintains a fixed list of supported ops and may reject models using newer or non-standard operators. The specific supported list is documented in tflite/g3doc/android/acceleration_service.md (line 256).

Vendor-Specific Backend Constraints

API Version Validation

When deploying to Qualcomm QNN, MediaTek Neuro, or other vendor runtimes, LiteRT validates that the compiler/dispatch library API version matches the runtime's expected version. According to litert/vendors/qualcomm/qnn_manager.cc and litert/vendors/qualcomm/dispatch/dispatch_api.cc (lines 345-352), incompatible versions cause a clear error and prevent model loading.

Block-wise Quantization Restrictions

Certain quantization schemes are explicitly unsupported for specific vendor backends. The kLiteRtQuantizationBlockWise scheme is marked "No" (unsupported) in litert/vendors/qualcomm/qnn_compiler.md, indicating that block-wise quantization cannot be compiled for Qualcomm NPUs.

Microcontroller (Micro-TFLite) Limitations

When targeting microcontrollers, LiteRT provides only a core subset of operators, omitting many higher-level operations such as certain TRANSPOSE variants and complex control-flow ops. As documented in tflite/g3doc/microcontrollers/index.md (line 97), developers must aggressively prune models to fit within these constraints.

Mitigation Strategies and Code Examples

Detecting GPU Delegate Rejections at Runtime

Use the C++ API to detect when the GPU delegate cannot handle specific operations, allowing graceful fallback to CPU execution:

#include "tensorflow/lite/interpreter.h"
#include "tensorflow/lite/delegates/gpu/gl_delegate.h"

std::unique_ptr<tflite::Interpreter> interpreter = ...;
auto* gpu_delegate = TfLiteGpuDelegateV2Create(TfLiteGpuDelegateOptionsV2Default());

if (interpreter->ModifyGraphWithDelegate(gpu_delegate) != kTfLiteOk) {
  std::cerr << "GPU delegate rejected some ops – running entire model on CPU.\n";
}
TfLiteGpuDelegateV2Delete(gpu_delegate);

ModifyGraphWithDelegate returns kTfLiteOk only if all ops are accepted; otherwise, the interpreter automatically partitions the graph between GPU and CPU.

Querying Supported Operations for Debugging

Verify GPU operator coverage before deployment using Python:

from tensorflow.lite.experimental import delegate

gpu_opts = delegate.GpuDelegateOptions()
gpu = delegate.GpuDelegate(gpu_opts)
supported_ops = gpu.supported_ops()
print("GPU supports:", supported_ops)

This API exposes the whitelist documented in tflite/g3doc/performance/gpu.md, allowing you to cross-check your model graph against hardware capabilities.

Guarding Against Play Services Limitations

Implement runtime detection to fallback from Play Services to a bundled LiteRT wheel when encountering unsupported operators:

import ai_edge_litert as litert

try:
    interpreter = litert.Interpreter(model_path="model.tflite")
except litert.UnsupportedOperatorError as e:
    print("Operator not supported by Play Services:", e)
    # Load bundled wheel implementation

    import importlib.util
    spec = importlib.util.spec_from_file_location("ai_edge_litert", "/opt/litert/ai_edge_litert/__init__.py")
    litert = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(litert)
    interpreter = litert.Interpreter(model_path="model.tflite")

Summary

• Stateless LSTM Only: LiteRT cannot convert stateful Keras LSTM layers; manage recurrent state manually in client code or rewrite models to be stateless.
• GPU Op Whitelist: The GPU delegate supports only specific operators (v1 by default) and falls back to CPU for unsupported ops, potentially adding synchronization overhead.
• Select Ops Type Restrictions: TensorFlow Select Ops may not support all data types available in core LiteRT ops, leading to conversion failures.
• Play Services Limitations: The Google Play Services LiteRT runtime excludes custom delegates and experimental features; bundle the full wheel for advanced models.
• Vendor Compatibility: Qualcomm and MediaTek backends require matching API versions between compiler and runtime, and explicitly block certain quantization schemes like block-wise quantization.
• Microcontroller Constraints: Micro-TFLite targets support only a minimal operator subset, requiring aggressive model pruning.

Frequently Asked Questions

Why does my stateful LSTM model fail to convert to LiteRT?

LiteRT only supports stateless Keras LSTM layers as of the current release. According to tflite/g3doc/models/convert/rnn.md, stateful LSTM conversion is not yet implemented. To resolve this, refactor your model to accept previous hidden and cell states as explicit inputs, or manage state persistence in your application code between inference calls.

What happens when a GPU delegate encounters an unsupported operator?

When the GPU delegate cannot handle an operation, the interpreter emits a warning (WARNING: op code #42 cannot be handled by this delegate) and automatically partitions the graph. The unsupported operation runs on CPU while compatible operations run on GPU. However, frequent CPU/GPU synchronization can degrade performance significantly compared to full GPU execution.

Does LiteRT via Google Play Services support all TensorFlow Lite operators?

No. The LiteRT runtime distributed through Google Play Services provides only a subset of operators and does not support custom delegates or experimental features. As documented in tflite/g3doc/android/play_services.md, models requiring operators outside the Play Services subset must include a bundled LiteRT wheel with the application.

Which quantization schemes are incompatible with Qualcomm and MediaTek NPUs?

Block-wise quantization (kLiteRtQuantizationBlockWise) is explicitly unsupported for Qualcomm and certain MediaTek backends. According to litert/vendors/qualcomm/qnn_compiler.md, this quantization scheme is marked as "No" in the compatibility matrix, and models using block-wise quantization will fail to compile for these vendor-specific NPUs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}