# LiteRT-LM Engine, Session, and Conversation Objects: Key Differences Explained

> Understand the key differences between LiteRT-LM Engine, Session, and Conversation objects. Learn how Engine creates Sessions for low-level control and Conversations for high-level chat abstraction and multi-turn history manage...

- Repository: [google-ai-edge/LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM)
- Tags: deep-dive
- Published: 2026-04-06

---

**TLDR:** The **Engine** acts as a factory that creates both **Session** and **Conversation** objects, where Session provides low-level token-oriented inference control (managing prefill/decode cycles and KV-cache state), while Conversation offers a high-level chat abstraction that orchestrates Sessions to handle multi-turn history, prompt templates, and tool calling.

The google-ai-edge/LiteRT-LM runtime exposes three core Python abstractions for executing on-device LLM inference. Understanding the architectural boundaries between the **Engine**, **Session**, and **Conversation** objects is essential for choosing the appropriate API level, whether you need fine-grained control over token streams or a ready-to-use chat interface.

## The Engine Object

The **Engine** serves as the root entry point for model initialization in LiteRT-LM. It owns the loaded model weights and backend configuration (CPU, GPU, or NPU), exposing factory methods that instantiate the two primary interaction patterns.

According to the source code in [`runtime/engine/engine.h`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/runtime/engine/engine.h), the Engine provides `CreateSession` (C++) and `create_session()` (Python) for low-level access, alongside `CreateConversation` (C++) and `create_conversation()` (Python) for high-level chat management. Both methods accept an optional `apply_prompt_template` parameter, though it defaults to `False` for Sessions and `True` for Conversations.

## Session: Low-Level Token-Oriented API

The **Session** object wraps the underlying C++ `Engine::Session` class (defined in [`runtime/engine/engine.h`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/runtime/engine/engine.h) lines 71-90) and provides direct access to the model's inference stages. In Python, this interface is abstracted through `AbstractSession` in [`python/litert_lm/interfaces.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/python/litert_lm/interfaces.py) (lines 28-78).

### Core Responsibilities

- **Prefill and Decode Management**: Exposes `run_prefill()` and `run_decode()` methods that map directly to the C++ `RunPrefill` and `RunDecode` calls
- **Text Scoring**: Supports `run_text_scoring()` (C++ `RunTextScoring`) for perplexity calculations and token probability analysis
- **KV-Cache State**: Maintains only the KV-cache state required for the current inference cycle, with no concept of message history or roles
- **Checkpoint Support**: Basic implementations like `SessionBasic` support checkpointing and rewind operations for state management

### Construction and Configuration

Create a Session when you need to bypass prompt template handling:

```python
session = engine.create_session(apply_prompt_template=False)

```

This returns an object where you manually control tokenization and context windows, with the caller responsible for embedding any tool-related tokens directly into the prompt.

## Conversation: High-Level Chat Abstraction

The **Conversation** object represents a significant step up in abstraction, wrapping a Session internally while adding conversation state management. The C++ implementation resides in [`runtime/conversation/conversation.h`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/runtime/conversation/conversation.h) (lines 91-145), with the Python interface defined in [`python/litert_lm/interfaces.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/python/litert_lm/interfaces.py) (lines 56-84) as `AbstractConversation`.

### Core Responsibilities

- **History Management**: Stores a `std::vector<Message> history_` (per [`conversation.h`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/conversation.h)) that tracks multi-turn exchanges with role attribution (system, user, assistant)
- **Prompt Template Rendering**: Automatically applies prompt templates via the `ModelDataProcessor` (see [`runtime/conversation/model_data_processor/model_data_processor.h`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/runtime/conversation/model_data_processor/model_data_processor.h))
- **Tool and Constraint Support**: Integrates `ConstraintProvider` for structured decoding and `ToolEventHandler` for function calling workflows
- **Async Streaming**: Provides `send_message_async()` that streams partial `Message` results, automatically managing task groups

### Construction

Unlike Session, Conversation initialization handles template application and history initialization:

```python
conversation = engine.create_conversation(
    messages=[{"role": "system", "content": "You are a helpful assistant."}]
)

```

## Key Differences Between Session and Conversation

| Feature | **Session** | **Conversation** |
|---------|-------------|------------------|
| **Abstraction Level** | Low-level, token-oriented | High-level, message-oriented |
| **C++ Class** | `Engine::Session` in [`runtime/engine/engine.h`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/runtime/engine/engine.h) (lines 71-90) | `Conversation` in [`runtime/conversation/conversation.h`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/runtime/conversation/conversation.h) (lines 91-145) |
| **Python Interface** | `AbstractSession` in [`python/litert_lm/interfaces.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/python/litert_lm/interfaces.py) (lines 28-78) | `AbstractConversation` in [`python/litert_lm/interfaces.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/python/litert_lm/interfaces.py) (lines 56-84) |
| **State Management** | KV-cache only; no message history | Full conversation history (`std::vector<Message>`) with append tracking |
| **Template Handling** | Manual; caller manages formatting | Automatic via `ModelDataProcessor` |
| **Tool Support** | No built-in awareness; manual token injection | Native `ToolEventHandler` and `ConstraintProvider` integration |
| **Async API** | `RunPrefillAsync`/`RunDecodeAsync` returning `TaskController` | `SendMessageAsync` streaming partial messages with automatic task group management |
| **Primary Methods** | `run_prefill()`, `run_decode()`, `run_text_scoring()` | `send_message()`, `send_message_async()`, `cancel_process()` |

## Practical Code Examples

### Low-Level Session Usage

For fine-grained control over token streams and custom prompt handling:

```python
from litert_lm import Engine, Backend

engine = Engine(
    model_path="model.tflite",
    backend=Backend.CPU,
)

# Create session without template processing

session = engine.create_session(apply_prompt_template=False)

# Prefill the prompt

session.run_prefill(["Explain quantum computing in simple terms."])

# Decode response

responses = session.run_decode()
print(responses.texts[0])

```

*Source: `AbstractSession.run_prefill` / `run_decode` – see [`python/litert_lm/interfaces.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/python/litert_lm/interfaces.py) lines 28-44.*

### High-Level Conversation Usage

For standard chat applications with history management:

```python
from litert_lm import Engine, Backend

engine = Engine(
    model_path="model.tflite",
    backend=Backend.CPU,
)

# Initialize with system message

conversation = engine.create_conversation(
    messages=[{"role": "system", "content": "You are a helpful assistant."}]
)

# Synchronous message

msg = conversation.send_message(
    {"role": "user", "content": "What is quantum computing?"}
)
print(msg["content"][0]["text"])

# Asynchronous streaming

for chunk in conversation.send_message_async(
    {"role": "user", "content": "Explain it in one sentence."}
):
    print(chunk["content"][0]["text"], end="", flush=True)

```

*Source: `Conversation.send_message` / `send_message_async` – see [`python/litert_lm/interfaces.py`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/python/litert_lm/interfaces.py) lines 56-84.*

### Cloning and Checkpointing

Both objects support cloning, but with different semantics. Cloning a **Session** creates an independent copy of the KV-cache state for parallel inference batches. Cloning a **Conversation** creates a new instance with a fresh Session and a deep copy of the conversation history:

```python

# Clone a conversation to explore alternative continuations

clone_conv = conversation.clone()

# The clone includes a fresh Session and copied history, but task-group IDs are not cloned

clone_conv.send_message({"role": "assistant", "content": "Alternative view..."})

```

*Source: `Conversation.Clone` – see [`runtime/conversation/conversation.h`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/runtime/conversation/conversation.h) lines 158-165.*

## Summary

- **Engine**: The factory object that loads model weights and creates Session/Conversation instances via `create_session()` and `create_conversation()`.
- **Session**: Low-level abstraction in [`runtime/engine/engine.h`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/runtime/engine/engine.h) providing direct `run_prefill()` and `run_decode()` access with manual KV-cache management and no history tracking.
- **Conversation**: High-level abstraction in [`runtime/conversation/conversation.h`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/runtime/conversation/conversation.h) that composes a Session to provide multi-turn history (`std::vector<Message>`), automatic prompt template rendering, and built-in tool handling via `ConstraintProvider`.
- **Architecture**: Conversation internally manages a Session instance; you cannot convert a Session to a Conversation, but Conversation relies on Session for actual token generation.
- **Use Case**: Choose Session for batch inference, custom prompting logic, or when running many independent inference batches in parallel; choose Conversation for interactive chat applications requiring history and tool support.

## Frequently Asked Questions

### When should I use Session instead of Conversation?

Use **Session** when you need fine-grained control over token streams, such as running batch inference across multiple independent prompts or implementing custom prompt formatting that doesn't match LiteRT-LM's built-in templates. Session exposes the raw `run_prefill()` and `run_decode()` methods from [`runtime/engine/engine.h`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/runtime/engine/engine.h) and maintains only the KV-cache state, making it more memory-efficient for single-turn tasks without conversation overhead.

### Can I convert an existing Session into a Conversation?

No. While **Conversation** internally composes a **Session** object created via `Engine::CreateConversation` (see [`conversation.h`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/conversation.h)), there is no API to wrap an existing Session instance with Conversation functionality. The Conversation constructor initializes both a fresh Session and a `ModelDataProcessor` for template handling. To switch abstractions, you must create a new Conversation from the Engine and manually transfer any context.

### How does cloning work in LiteRT-LM?

Both Session and Conversation support cloning, but with different behaviors. Cloning a **Session** (via `Clone` in [`engine.h`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/engine.h)) creates an independent copy of the KV-cache state for parallel inference. Cloning a **Conversation** (via `Clone` in [`conversation.h`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/conversation.h) lines 158-165) creates a new Conversation with a fresh Session instance and a deep copy of the `history_` vector, though task-group IDs are intentionally not cloned to prevent async task conflicts.

### Does LiteRT-LM support asynchronous inference for both abstractions?

Yes, but with different interfaces. **Session** provides `RunPrefillAsync` and `RunDecodeAsync` (C++) or `run_prefill_async` and `run_decode_async` (Python) which return a `TaskController` for direct task management. **Conversation** offers `send_message_async()` which handles the underlying Session calls internally and streams partial `Message` results, automatically managing task groups and callback routing according to the implementation in [`runtime/conversation/conversation.h`](https://github.com/google-ai-edge/LiteRT-LM/blob/main/runtime/conversation/conversation.h).