How LiteRT-LM Manages Multi-Turn Conversation State: Architecture and Implementation

LiteRT-LM treats conversation as a stateful object that maintains chat history and KV cache coherence through checkpointing and thread-safe history management in the native C++ Conversation class.

The google-ai-edge/LiteRT-LM runtime implements robust multi-turn dialogue capabilities by treating each conversation as an encapsulated entity that owns both the semantic history and the low-level inference state required to keep Large Language Model (LLM) Key-Value (KV) caches consistent across turns. Unlike stateless inference APIs, LiteRT-LM manages the complete lifecycle of conversational context within the Conversation class defined in runtime/conversation/conversation.cc, enabling efficient state persistence and selective cache invalidation when processing channel-specific content.

Conversation Lifecycle and Core Components

Every multi-turn session begins with Engine::CreateConversation, which instantiates a Conversation object configured with three critical elements. According to the source in runtime/conversation/conversation.cc, the initialization sequence calls ConversationConfig::CreateInternal at line 97 to establish the preface—containing system messages, tool definitions, and additional context—followed by Conversation::Create at line 308 to initialize the prompt template used for rendering messages into model-compatible text.

The constructor also initializes a model-data processor via CreateModelDataProcessor (line 18), which handles the conversion of JSON message objects into raw input tensors. Together, these components form the stateful foundation that persists across user interactions, eliminating the need for external history management.

Thread-Safe History Storage

LiteRT-LM protects concurrent access to dialogue history through an absl::Mutex named history_mutex_ that guards the mutable history_ vector. As implemented in runtime/conversation/conversation.cc (lines 17–24), this vector stores Message objects as std::variant types capable of holding nlohmann::ordered_json or other internal representations.

When SendMessage processes a new user input, it acquires the mutex to append the message to history_ before releasing the lock for inference operations. This design ensures that multiple threads can safely interact with the same conversation instance without corrupting the dialogue state, a critical feature for server deployments handling concurrent user sessions.

Prompt Rendering Strategies

The engine optimizes token generation by attempting a single-turn fast path before falling back to full-history rendering. In runtime/conversation/conversation.cc (lines 70–78), the method GetSingleTurnText attempts to render only the latest user message and preface, reducing computation when the model supports stateless context windows.

If the single-turn template is unavailable or insufficient, the system automatically switches to the full-history path, utilizing the Jinja-based prompt template defined in runtime/components/prompt_template.h to materialize the complete dialogue context. This dual-strategy approach minimizes latency for compatible model configurations while preserving coherence for complex multi-turn exchanges.

KV Cache Checkpointing and Rewinding

A distinctive feature of LiteRT-LM’s state management is its checkpointing mechanism for handling channel content that should not persist in the KV cache across turns. When the conversation configuration enables filter_channel_content_from_kv_cache, the system executes SaveCheckpoint at runtime/conversation/conversation.cc (lines 37–45) immediately before the assistant generates responses containing channel-specific data.

If the subsequent inbound message is a user turn, the session invokes RewindAndGetInputDataVector (lines 4–12) to restore the KV cache to the checkpointed state, effectively stripping previously emitted channel content from the model’s internal memory. This prevents ephemeral tool outputs or system channel data from polluting the attention mechanism during future turns, maintaining optimal generation quality.

Response Processing and State Advancement

Following the prefill phase, the engine constructs a DecodeConfig and executes session_->RunDecode (lines 48–52) to generate the assistant’s response. The raw decoder output undergoes processing through ExtractChannelContent (lines 54–56), which scans for channel placeholders and converts them into structured objects.

The method InsertChannelContentIntoMessage (lines 63–65) then embeds these structures into the final assistant message before it is appended to history_. If the generated message contains channel content, the system updates checkpoint_message_index_ (lines 74–81) to mark this message as the new rewind point for the next turn, ensuring the checkpointing strategy remains synchronized with the dialogue progression.

Python API Integration

While all state management logic resides in C++, the Python layer provides the AbstractConversation interface in python/litert_lm/interfaces.py, which forwards calls to the native implementation via the litert_lm_ext compiled extension. Developers interact with the conversation state through high-level methods that automatically handle history appending, checkpoint rewinding, and channel extraction.

Basic Multi-Turn Conversation

from litert_lm import Engine

# Initialize engine with TFLite model

engine = Engine(model_path="model.tflite", backend="CPU")

# Create conversation with system preface

conversation = engine.create_conversation(
    messages=[{"role": "system", "content": "You are a helpful assistant."}]
)

# First turn

resp1 = conversation.send_message("What is the capital of France?")
print(resp1["content"][0]["text"])  # Output: Paris

# Second turn maintains state automatically

resp2 = conversation.send_message("What’s its population?")
print(resp2["content"][0]["text"])  # Output: ~2.1 million

Asynchronous Streaming

def stream_callback(result):
    if result.ok():
        msg = result.message()
        print("Chunk:", msg["content"][0]["text"])
    else:
        print("Error:", result.status())

conversation.send_message_async(
    "Tell me a story about a robot.",
    user_callback=stream_callback,
    max_output_tokens=256,
)

The asynchronous path mirrors the synchronous flow, utilizing SendMessageAsync (lines 86–115) to stream partial decoder results while preserving the same mutex-protected history updates and checkpoint logic.

Summary

• Stateful Architecture: LiteRT-LM encapsulates multi-turn state in the C++ Conversation class, combining chat history with runtime inference state according to runtime/conversation/conversation.cc.
• Thread Safety: Concurrent access to dialogue history is protected by absl::Mutex guarding the history_ vector during append operations in SendMessage.
• Checkpointing: The KV cache supports selective rewinding via SaveCheckpoint and RewindAndGetInputDataVector to filter channel content when filter_channel_content_from_kv_cache is enabled.
• Rendering Optimization: The engine attempts GetSingleTurnText for low-latency single-turn processing before falling back to full-history templates.
• Channel Processing: Assistant outputs undergo ExtractChannelContent and InsertChannelContentIntoMessage to handle structured data, with checkpoint_message_index_ tracking rewind positions.

Frequently Asked Questions

How does LiteRT-LM ensure thread safety during multi-turn conversations?

The implementation uses an absl::Mutex named history_mutex_ to protect the history_ vector in runtime/conversation/conversation.cc. Every modification to the conversation history, including appends in SendMessage (lines 17–24), acquires this lock, allowing multiple threads to safely share conversation instances without race conditions.

What is the purpose of KV cache checkpointing in LiteRT-LM?

Checkpointing prevents channel-specific content—such as tool outputs or temporary system data—from permanently occupying the KV cache. When filter_channel_content_from_kv_cache is active, SaveCheckpoint preserves the cache state before channel generation, and RewindAndGetInputDataVector restores it before processing subsequent user messages, ensuring clean attention mechanisms for each turn.

How does LiteRT-LM handle prompt rendering for different conversation lengths?

The system employs a two-tier strategy via GetSingleTurnText in runtime/conversation/conversation.cc (lines 70–78), first attempting to render only the latest message and preface for efficiency. If the model requires full context, it automatically falls back to rendering the complete history_ vector using the Jinja-based template system.

Can Python developers manually manipulate the conversation history?

No. The AbstractConversation interface in python/litert_lm/interfaces.py exposes only high-level methods like send_message and send_message_async. History management—including appending, mutex locking, and checkpoint updates—is intentionally encapsulated within the C++ implementation to ensure consistency and prevent state corruption.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}