LiteRT-LM Engine, Session, and Conversation Objects: Key Differences Explained

TLDR: The Engine acts as a factory that creates both Session and Conversation objects, where Session provides low-level token-oriented inference control (managing prefill/decode cycles and KV-cache state), while Conversation offers a high-level chat abstraction that orchestrates Sessions to handle multi-turn history, prompt templates, and tool calling.

The google-ai-edge/LiteRT-LM runtime exposes three core Python abstractions for executing on-device LLM inference. Understanding the architectural boundaries between the Engine, Session, and Conversation objects is essential for choosing the appropriate API level, whether you need fine-grained control over token streams or a ready-to-use chat interface.

The Engine Object

The Engine serves as the root entry point for model initialization in LiteRT-LM. It owns the loaded model weights and backend configuration (CPU, GPU, or NPU), exposing factory methods that instantiate the two primary interaction patterns.

According to the source code in runtime/engine/engine.h, the Engine provides CreateSession (C++) and create_session() (Python) for low-level access, alongside CreateConversation (C++) and create_conversation() (Python) for high-level chat management. Both methods accept an optional apply_prompt_template parameter, though it defaults to False for Sessions and True for Conversations.

Session: Low-Level Token-Oriented API

The Session object wraps the underlying C++ Engine::Session class (defined in runtime/engine/engine.h lines 71-90) and provides direct access to the model's inference stages. In Python, this interface is abstracted through AbstractSession in python/litert_lm/interfaces.py (lines 28-78).

Core Responsibilities

• Prefill and Decode Management: Exposes run_prefill() and run_decode() methods that map directly to the C++ RunPrefill and RunDecode calls
• Text Scoring: Supports run_text_scoring() (C++ RunTextScoring) for perplexity calculations and token probability analysis
• KV-Cache State: Maintains only the KV-cache state required for the current inference cycle, with no concept of message history or roles
• Checkpoint Support: Basic implementations like SessionBasic support checkpointing and rewind operations for state management

Construction and Configuration

Create a Session when you need to bypass prompt template handling:

session = engine.create_session(apply_prompt_template=False)

This returns an object where you manually control tokenization and context windows, with the caller responsible for embedding any tool-related tokens directly into the prompt.

Conversation: High-Level Chat Abstraction

The Conversation object represents a significant step up in abstraction, wrapping a Session internally while adding conversation state management. The C++ implementation resides in runtime/conversation/conversation.h (lines 91-145), with the Python interface defined in python/litert_lm/interfaces.py (lines 56-84) as AbstractConversation.

Core Responsibilities

• History Management: Stores a std::vector<Message> history_ (per conversation.h) that tracks multi-turn exchanges with role attribution (system, user, assistant)
• Prompt Template Rendering: Automatically applies prompt templates via the ModelDataProcessor (see runtime/conversation/model_data_processor/model_data_processor.h)
• Tool and Constraint Support: Integrates ConstraintProvider for structured decoding and ToolEventHandler for function calling workflows
• Async Streaming: Provides send_message_async() that streams partial Message results, automatically managing task groups

Construction

Unlike Session, Conversation initialization handles template application and history initialization:

conversation = engine.create_conversation(
    messages=[{"role": "system", "content": "You are a helpful assistant."}]
)

Key Differences Between Session and Conversation

Feature	Session	Conversation
Abstraction Level	Low-level, token-oriented	High-level, message-oriented
C++ Class	Engine::Session in runtime/engine/engine.h (lines 71-90)	Conversation in runtime/conversation/conversation.h (lines 91-145)
Python Interface	AbstractSession in python/litert_lm/interfaces.py (lines 28-78)	AbstractConversation in python/litert_lm/interfaces.py (lines 56-84)
State Management	KV-cache only; no message history	Full conversation history (std::vector<Message>) with append tracking
Template Handling	Manual; caller manages formatting	Automatic via ModelDataProcessor
Tool Support	No built-in awareness; manual token injection	Native ToolEventHandler and ConstraintProvider integration
Async API	RunPrefillAsync/RunDecodeAsync returning TaskController	SendMessageAsync streaming partial messages with automatic task group management
Primary Methods	run_prefill(), run_decode(), run_text_scoring()	send_message(), send_message_async(), cancel_process()

Practical Code Examples

Low-Level Session Usage

For fine-grained control over token streams and custom prompt handling:

from litert_lm import Engine, Backend

engine = Engine(
    model_path="model.tflite",
    backend=Backend.CPU,
)

# Create session without template processing

session = engine.create_session(apply_prompt_template=False)

# Prefill the prompt

session.run_prefill(["Explain quantum computing in simple terms."])

# Decode response

responses = session.run_decode()
print(responses.texts[0])

Source: AbstractSession.run_prefill / run_decode – see python/litert_lm/interfaces.py lines 28-44.

High-Level Conversation Usage

For standard chat applications with history management:

from litert_lm import Engine, Backend

engine = Engine(
    model_path="model.tflite",
    backend=Backend.CPU,
)

# Initialize with system message

conversation = engine.create_conversation(
    messages=[{"role": "system", "content": "You are a helpful assistant."}]
)

# Synchronous message

msg = conversation.send_message(
    {"role": "user", "content": "What is quantum computing?"}
)
print(msg["content"][0]["text"])

# Asynchronous streaming

for chunk in conversation.send_message_async(
    {"role": "user", "content": "Explain it in one sentence."}
):
    print(chunk["content"][0]["text"], end="", flush=True)

Source: Conversation.send_message / send_message_async – see python/litert_lm/interfaces.py lines 56-84.

Cloning and Checkpointing

Both objects support cloning, but with different semantics. Cloning a Session creates an independent copy of the KV-cache state for parallel inference batches. Cloning a Conversation creates a new instance with a fresh Session and a deep copy of the conversation history:

# Clone a conversation to explore alternative continuations

clone_conv = conversation.clone()

# The clone includes a fresh Session and copied history, but task-group IDs are not cloned

clone_conv.send_message({"role": "assistant", "content": "Alternative view..."})

Source: Conversation.Clone – see runtime/conversation/conversation.h lines 158-165.

Summary

• Engine: The factory object that loads model weights and creates Session/Conversation instances via create_session() and create_conversation().
• Session: Low-level abstraction in runtime/engine/engine.h providing direct run_prefill() and run_decode() access with manual KV-cache management and no history tracking.
• Conversation: High-level abstraction in runtime/conversation/conversation.h that composes a Session to provide multi-turn history (std::vector<Message>), automatic prompt template rendering, and built-in tool handling via ConstraintProvider.
• Architecture: Conversation internally manages a Session instance; you cannot convert a Session to a Conversation, but Conversation relies on Session for actual token generation.
• Use Case: Choose Session for batch inference, custom prompting logic, or when running many independent inference batches in parallel; choose Conversation for interactive chat applications requiring history and tool support.

Frequently Asked Questions

When should I use Session instead of Conversation?

Use Session when you need fine-grained control over token streams, such as running batch inference across multiple independent prompts or implementing custom prompt formatting that doesn't match LiteRT-LM's built-in templates. Session exposes the raw run_prefill() and run_decode() methods from runtime/engine/engine.h and maintains only the KV-cache state, making it more memory-efficient for single-turn tasks without conversation overhead.

Can I convert an existing Session into a Conversation?

No. While Conversation internally composes a Session object created via Engine::CreateConversation (see conversation.h), there is no API to wrap an existing Session instance with Conversation functionality. The Conversation constructor initializes both a fresh Session and a ModelDataProcessor for template handling. To switch abstractions, you must create a new Conversation from the Engine and manually transfer any context.

How does cloning work in LiteRT-LM?

Both Session and Conversation support cloning, but with different behaviors. Cloning a Session (via Clone in engine.h) creates an independent copy of the KV-cache state for parallel inference. Cloning a Conversation (via Clone in conversation.h lines 158-165) creates a new Conversation with a fresh Session instance and a deep copy of the history_ vector, though task-group IDs are intentionally not cloned to prevent async task conflicts.

Does LiteRT-LM support asynchronous inference for both abstractions?

Yes, but with different interfaces. Session provides RunPrefillAsync and RunDecodeAsync (C++) or run_prefill_async and run_decode_async (Python) which return a TaskController for direct task management. Conversation offers send_message_async() which handles the underlying Session calls internally and streams partial Message results, automatically managing task groups and callback routing according to the implementation in runtime/conversation/conversation.h.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}