# LiteRT-LM Sampler Parameters: Complete Guide to TopK, TopP, Greedy, and Temperature

> Master LiteRT-LM sampler parameters like TopK, TopP, Greedy, and Temperature. Optimize your text generation with this complete guide to LiteRT-LM's SamplerParameters.

- Repository: [google-ai-edge/LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM)
- Tags: api-reference
- Published: 2026-04-06

---

**LiteRT-LM exposes four sampler types—`TOP_K`, `TOP_P`, `GREEDY`, and `TYPE_UNSPECIFIED`—controlled via the `SamplerParameters` protobuf message with fields for `k` (int), `p` (float), `temperature` (float), and `seed` (int).**

The google-ai-edge/LiteRT-LM repository implements a flexible sampling system for on-device language model inference. Understanding the available LiteRT-LM sampler parameters allows developers to fine-tune generation behavior from deterministic outputs to creative sampling across CPU, GPU, and NPU backends.

## SamplerParameters Protobuf Structure

LiteRT-LM defines all sampling configuration in the `SamplerParameters` message located in `runtime/proto/sampler_params.proto`. This protobuf serves as the single source of truth for decoding strategies.

### Core Configuration Fields

The message contains the following fields:

- **`type`** – Enum specifying the decoding algorithm. Valid values are `TOP_K`, `TOP_P`, `GREEDY`, or `TYPE_UNSPECIFIED`.
- **`k`** – Integer defining the number of highest-logit tokens retained for sampling. Used directly by `TOP_K` and as a pre-filter for `TOP_P`.
- **`p`** – Float representing the cumulative probability threshold (0.0–1.0) for nucleus sampling. Tokens are kept until their summed probability exceeds this value.
- **`temperature`** – Float scaling factor applied to logits before softmax. Values above 1.0 increase randomness; 1.0 leaves logits unchanged; values approaching 0.0 approach greedy behavior.
- **`seed`** – Optional integer for RNG initialization. Default is 0.

## Sampler Types and Algorithms

The `type` field determines which algorithm executes during token generation.

### TOP_K Sampling

**`TOP_K`** probabilistically selects from the `k` tokens with the highest logits. After identifying the top `k` candidates, the sampler applies temperature scaling and randomly selects based on the renormalized probabilities.

### TOP_P (Nucleus) Sampling

**`TOP_P`** first applies the `k` cutoff, then filters further to the smallest set of tokens whose cumulative probability meets or exceeds `p`. This dynamic vocabulary reduction allows for adaptive creativity—rare tokens appear when the model is confident, but only high-probability tokens are considered when uncertainty is low.

### GREEDY Decoding

**`GREEDY`** deterministically selects the token with the maximum logit (argmax). According to the source in `runtime/executor/llm_litert_mtp_drafter.cc` (lines 50-58), this is internally implemented by setting `type = TOP_P`, `k = 1`, and `p = 0.0`, effectively collapsing the probability distribution to a single choice.

### TYPE_UNSPECIFIED

**`TYPE_UNSPECIFIED`** signals that the backend should use its own sampling logic. This is the default path for NPU/ARTISAN backends that implement sampling in hardware or firmware, bypassing the CPU-based `TopPSampler` implementation.

## Backend-Specific Default Values

The runtime populates default `SamplerParameters` when model metadata lacks explicit configuration. As implemented in `runtime/engine/engine_settings.cc` (lines 36-49), defaults vary by execution target:

**CPU and GPU Backends:**
- `type`: `TOP_P`
- `k`: 1 (minimal filtering)
- `p`: 0.95 (95% cumulative probability)
- `temperature`: 1.0 (no scaling)
- `seed`: 0

**NPU/ARTISAN Backends:**
- `type`: `TYPE_UNSPECIFIED` (delegated to hardware implementation)

## Implementation Details

The sampling pipeline is constructed through several key components:

1. **Definition** – `runtime/proto/sampler_params.proto` declares the parameter structure and enum values.
2. **Factory Creation** – `runtime/components/sampler_factory.cc` reads the protobuf and instantiates concrete samplers (e.g., `TopPSampler` for CPU).
3. **CPU Execution** – `runtime/components/top_p_cpu_sampler.cc` validates parameters and implements the top-K/top-P logic, including temperature application.
4. **Greedy Shortcut** – `runtime/executor/llm_litert_mtp_drafter.cc` demonstrates constructing a greedy sampler via parameter manipulation.

## Configuration Examples

### C++: TOP-K Configuration

```cpp
#include "runtime/proto/sampler_params.pb.h"
#include "runtime/components/sampler_factory.h"

using litert::lm::proto::SamplerParameters;

// Configure TOP-K with 50 candidates and reduced temperature
SamplerParameters params;
params.set_type(SamplerParameters::TOP_K);
params.set_k(50);
params.set_p(0.0f);  // Ignored for TOP_K
params.set_temperature(0.7f);
params.set_seed(12345);

auto sampler = CreateSampler(
    Backend::CPU,
    /*output_heads=*/1,
    std::move(params),
    environment,
    /*sequence_size=*/1,
    vocab_size,
    std::nullopt);

```

### C++: Greedy Configuration

```cpp
SamplerParameters greedy;
greedy.set_type(SamplerParameters::GREEDY);
greedy.set_k(1);
greedy.set_p(0.0f);
greedy.set_temperature(1.0f);
greedy.set_seed(0);

auto greedy_sampler = CreateSampler(
    Backend::CPU, 1, std::move(greedy), env, 
    1, vocab_size, std::nullopt);

```

### Python: Protobuf Preparation

```python
from litert.lm.proto import sampler_params_pb2 as sp_pb2

# TOP-P configuration with pre-filtering

sampler_cfg = sp_pb2.SamplerParameters()
sampler_cfg.type = sp_pb2.SamplerParameters.TOP_P
sampler_cfg.k = 1              # Pre-filter to top token (minimal effect)

sampler_cfg.p = 0.9            # Nucleus threshold

sampler_cfg.temperature = 0.8  # Slightly conservative randomness

sampler_cfg.seed = 42

# Attach to model metadata

llm_metadata.sampler_params.CopyFrom(sampler_cfg)

```

## Summary

- **LiteRT-LM sampler parameters** are defined in the `SamplerParameters` protobuf with fields for `type`, `k`, `p`, `temperature`, and `seed`.
- **Four sampler types** are available: `TOP_K` (top-k sampling), `TOP_P` (nucleus sampling), `GREEDY` (argmax), and `TYPE_UNSPECIFIED` (backend-dependent).
- **CPU/GPU defaults** use `TOP_P` with `k=1`, `p=0.95`, and `temperature=1.0`, while NPUs default to `TYPE_UNSPECIFIED`.
- **Greedy decoding** is internally implemented as `TOP_P` with `k=1` and `p=0.0`.
- **Source files** governing this behavior include `runtime/proto/sampler_params.proto`, `runtime/engine/engine_settings.cc`, and `runtime/components/top_p_cpu_sampler.cc`.

## Frequently Asked Questions

### What is the difference between TOP_K and TOP_P in LiteRT-LM?

**TOP_K** restricts sampling to a fixed number (`k`) of highest-probability tokens regardless of their actual probability values. **TOP_P** (nucleus sampling) uses a dynamic cutoff based on cumulative probability, selecting the smallest set of tokens whose probabilities sum to at least `p`. TOP_P can adapt to the model's confidence, while TOP_K provides a fixed vocabulary size limit.

### How does temperature affect sampling in LiteRT-LM?

**Temperature** scales logits before the softmax operation via division: `logits / temperature`. A value of 1.0 applies no change. Values below 1.0 sharpen the distribution (more deterministic), while values above 1.0 flatten it (more random). In `runtime/components/top_p_cpu_sampler.cc`, this scaling occurs before the top-K filtering and probability normalization.

### When should I use TYPE_UNSPECIFIED?

Use **TYPE_UNSPECIFIED** when deploying on NPU or specialized hardware backends (ARTISAN) that implement their own optimized sampling logic. When this type is set, the LiteRT-LM runtime skips its internal `CreateSampler` factory logic and allows the hardware driver to handle token selection, which often provides better performance on accelerator hardware.

### How do I force deterministic (greedy) output in LiteRT-LM?

Set `type` to `GREEDY` or manually configure parameters to `k=1`, `p=0.0`, and `temperature=1.0`. According to the implementation in `runtime/executor/llm_litert_mtp_drafter.cc`, the greedy type is equivalent to TOP_P sampling with a single candidate and zero probability threshold, ensuring the highest-logit token is always selected.