# How LiteRT-LM Manages Multi-Turn Conversation State: Architecture and Implementation

> Learn how LiteRT-LM manages multi turn conversation state using its C++ Conversation class for efficient chat history and KV cache management. Discover the architecture.

- Repository: [google-ai-edge/LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM)
- Tags: architecture
- Published: 2026-04-06

---

**LiteRT-LM treats conversation as a stateful object that maintains chat history and KV cache coherence through checkpointing and thread-safe history management in the native C++ `Conversation` class.**

The `google-ai-edge/LiteRT-LM` runtime implements robust multi-turn dialogue capabilities by treating each conversation as an encapsulated entity that owns both the semantic history and the low-level inference state required to keep Large Language Model (LLM) Key-Value (KV) caches consistent across turns. Unlike stateless inference APIs, LiteRT-LM manages the complete lifecycle of conversational context within the `Conversation` class defined in `runtime/conversation/conversation.cc`, enabling efficient state persistence and selective cache invalidation when processing channel-specific content.

## Conversation Lifecycle and Core Components

Every multi-turn session begins with `Engine::CreateConversation`, which instantiates a `Conversation` object configured with three critical elements. According to the source in `runtime/conversation/conversation.cc`, the initialization sequence calls `ConversationConfig::CreateInternal` at line 97 to establish the **preface**—containing system messages, tool definitions, and additional context—followed by `Conversation::Create` at line 308 to initialize the **prompt template** used for rendering messages into model-compatible text.

The constructor also initializes a **model-data processor** via `CreateModelDataProcessor` (line 18), which handles the conversion of JSON message objects into raw input tensors. Together, these components form the stateful foundation that persists across user interactions, eliminating the need for external history management.

## Thread-Safe History Storage

LiteRT-LM protects concurrent access to dialogue history through an `absl::Mutex` named `history_mutex_` that guards the mutable `history_` vector. As implemented in `runtime/conversation/conversation.cc` (lines 17–24), this vector stores `Message` objects as `std::variant` types capable of holding `nlohmann::ordered_json` or other internal representations.

When `SendMessage` processes a new user input, it acquires the mutex to append the message to `history_` before releasing the lock for inference operations. This design ensures that multiple threads can safely interact with the same conversation instance without corrupting the dialogue state, a critical feature for server deployments handling concurrent user sessions.

## Prompt Rendering Strategies

The engine optimizes token generation by attempting a **single-turn fast path** before falling back to full-history rendering. In `runtime/conversation/conversation.cc` (lines 70–78), the method `GetSingleTurnText` attempts to render only the latest user message and preface, reducing computation when the model supports stateless context windows.

If the single-turn template is unavailable or insufficient, the system automatically switches to the **full-history** path, utilizing the Jinja-based prompt template defined in [`runtime/components/prompt_template.h`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/runtime/components/prompt_template.h) to materialize the complete dialogue context. This dual-strategy approach minimizes latency for compatible model configurations while preserving coherence for complex multi-turn exchanges.

## KV Cache Checkpointing and Rewinding

A distinctive feature of LiteRT-LM’s state management is its **checkpointing mechanism** for handling channel content that should not persist in the KV cache across turns. When the conversation configuration enables `filter_channel_content_from_kv_cache`, the system executes `SaveCheckpoint` at `runtime/conversation/conversation.cc` (lines 37–45) immediately before the assistant generates responses containing channel-specific data.

If the subsequent inbound message is a user turn, the session invokes `RewindAndGetInputDataVector` (lines 4–12) to restore the KV cache to the checkpointed state, effectively stripping previously emitted channel content from the model’s internal memory. This prevents ephemeral tool outputs or system channel data from polluting the attention mechanism during future turns, maintaining optimal generation quality.

## Response Processing and State Advancement

Following the prefill phase, the engine constructs a `DecodeConfig` and executes `session_->RunDecode` (lines 48–52) to generate the assistant’s response. The raw decoder output undergoes processing through `ExtractChannelContent` (lines 54–56), which scans for channel placeholders and converts them into structured objects.

The method `InsertChannelContentIntoMessage` (lines 63–65) then embeds these structures into the final assistant message before it is appended to `history_`. If the generated message contains channel content, the system updates `checkpoint_message_index_` (lines 74–81) to mark this message as the new rewind point for the next turn, ensuring the checkpointing strategy remains synchronized with the dialogue progression.

## Python API Integration

While all state management logic resides in C++, the Python layer provides the `AbstractConversation` interface in [`python/litert_lm/interfaces.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/python/litert_lm/interfaces.py), which forwards calls to the native implementation via the `litert_lm_ext` compiled extension. Developers interact with the conversation state through high-level methods that automatically handle history appending, checkpoint rewinding, and channel extraction.

### Basic Multi-Turn Conversation

```python
from litert_lm import Engine

# Initialize engine with TFLite model

engine = Engine(model_path="model.tflite", backend="CPU")

# Create conversation with system preface

conversation = engine.create_conversation(
    messages=[{"role": "system", "content": "You are a helpful assistant."}]
)

# First turn

resp1 = conversation.send_message("What is the capital of France?")
print(resp1["content"][0]["text"])  # Output: Paris

# Second turn maintains state automatically

resp2 = conversation.send_message("What’s its population?")
print(resp2["content"][0]["text"])  # Output: ~2.1 million

```

### Asynchronous Streaming

```python
def stream_callback(result):
    if result.ok():
        msg = result.message()
        print("Chunk:", msg["content"][0]["text"])
    else:
        print("Error:", result.status())

conversation.send_message_async(
    "Tell me a story about a robot.",
    user_callback=stream_callback,
    max_output_tokens=256,
)

```

The asynchronous path mirrors the synchronous flow, utilizing `SendMessageAsync` (lines 86–115) to stream partial decoder results while preserving the same mutex-protected history updates and checkpoint logic.

## Summary

- **Stateful Architecture**: LiteRT-LM encapsulates multi-turn state in the C++ `Conversation` class, combining chat history with runtime inference state according to `runtime/conversation/conversation.cc`.
- **Thread Safety**: Concurrent access to dialogue history is protected by `absl::Mutex` guarding the `history_` vector during append operations in `SendMessage`.
- **Checkpointing**: The KV cache supports selective rewinding via `SaveCheckpoint` and `RewindAndGetInputDataVector` to filter channel content when `filter_channel_content_from_kv_cache` is enabled.
- **Rendering Optimization**: The engine attempts `GetSingleTurnText` for low-latency single-turn processing before falling back to full-history templates.
- **Channel Processing**: Assistant outputs undergo `ExtractChannelContent` and `InsertChannelContentIntoMessage` to handle structured data, with `checkpoint_message_index_` tracking rewind positions.

## Frequently Asked Questions

### How does LiteRT-LM ensure thread safety during multi-turn conversations?

The implementation uses an `absl::Mutex` named `history_mutex_` to protect the `history_` vector in `runtime/conversation/conversation.cc`. Every modification to the conversation history, including appends in `SendMessage` (lines 17–24), acquires this lock, allowing multiple threads to safely share conversation instances without race conditions.

### What is the purpose of KV cache checkpointing in LiteRT-LM?

Checkpointing prevents channel-specific content—such as tool outputs or temporary system data—from permanently occupying the KV cache. When `filter_channel_content_from_kv_cache` is active, `SaveCheckpoint` preserves the cache state before channel generation, and `RewindAndGetInputDataVector` restores it before processing subsequent user messages, ensuring clean attention mechanisms for each turn.

### How does LiteRT-LM handle prompt rendering for different conversation lengths?

The system employs a two-tier strategy via `GetSingleTurnText` in `runtime/conversation/conversation.cc` (lines 70–78), first attempting to render only the latest message and preface for efficiency. If the model requires full context, it automatically falls back to rendering the complete `history_` vector using the Jinja-based template system.

### Can Python developers manually manipulate the conversation history?

No. The `AbstractConversation` interface in [`python/litert_lm/interfaces.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/python/litert_lm/interfaces.py) exposes only high-level methods like `send_message` and `send_message_async`. History management—including appending, mutex locking, and checkpoint updates—is intentionally encapsulated within the C++ implementation to ensure consistency and prevent state corruption.