LiteRT-LM Sampler Parameters: Complete Guide to TopK, TopP, Greedy, and Temperature

LiteRT-LM exposes four sampler types—TOP_K, TOP_P, GREEDY, and TYPE_UNSPECIFIED—controlled via the SamplerParameters protobuf message with fields for k (int), p (float), temperature (float), and seed (int).

The google-ai-edge/LiteRT-LM repository implements a flexible sampling system for on-device language model inference. Understanding the available LiteRT-LM sampler parameters allows developers to fine-tune generation behavior from deterministic outputs to creative sampling across CPU, GPU, and NPU backends.

SamplerParameters Protobuf Structure

LiteRT-LM defines all sampling configuration in the SamplerParameters message located in runtime/proto/sampler_params.proto. This protobuf serves as the single source of truth for decoding strategies.

Core Configuration Fields

The message contains the following fields:

• type – Enum specifying the decoding algorithm. Valid values are TOP_K, TOP_P, GREEDY, or TYPE_UNSPECIFIED.
• k – Integer defining the number of highest-logit tokens retained for sampling. Used directly by TOP_K and as a pre-filter for TOP_P.
• p – Float representing the cumulative probability threshold (0.0–1.0) for nucleus sampling. Tokens are kept until their summed probability exceeds this value.
• temperature – Float scaling factor applied to logits before softmax. Values above 1.0 increase randomness; 1.0 leaves logits unchanged; values approaching 0.0 approach greedy behavior.
• seed – Optional integer for RNG initialization. Default is 0.

Sampler Types and Algorithms

The type field determines which algorithm executes during token generation.

TOP_K Sampling

TOP_K probabilistically selects from the k tokens with the highest logits. After identifying the top k candidates, the sampler applies temperature scaling and randomly selects based on the renormalized probabilities.

TOP_P (Nucleus) Sampling

TOP_P first applies the k cutoff, then filters further to the smallest set of tokens whose cumulative probability meets or exceeds p. This dynamic vocabulary reduction allows for adaptive creativity—rare tokens appear when the model is confident, but only high-probability tokens are considered when uncertainty is low.

GREEDY Decoding

GREEDY deterministically selects the token with the maximum logit (argmax). According to the source in runtime/executor/llm_litert_mtp_drafter.cc (lines 50-58), this is internally implemented by setting type = TOP_P, k = 1, and p = 0.0, effectively collapsing the probability distribution to a single choice.

TYPE_UNSPECIFIED

TYPE_UNSPECIFIED signals that the backend should use its own sampling logic. This is the default path for NPU/ARTISAN backends that implement sampling in hardware or firmware, bypassing the CPU-based TopPSampler implementation.

Backend-Specific Default Values

The runtime populates default SamplerParameters when model metadata lacks explicit configuration. As implemented in runtime/engine/engine_settings.cc (lines 36-49), defaults vary by execution target:

CPU and GPU Backends:

• type: TOP_P
• k: 1 (minimal filtering)
• p: 0.95 (95% cumulative probability)
• temperature: 1.0 (no scaling)
• seed: 0

NPU/ARTISAN Backends:

• type: TYPE_UNSPECIFIED (delegated to hardware implementation)

Implementation Details

The sampling pipeline is constructed through several key components:

1. Definition – runtime/proto/sampler_params.proto declares the parameter structure and enum values.
2. Factory Creation – runtime/components/sampler_factory.cc reads the protobuf and instantiates concrete samplers (e.g., TopPSampler for CPU).
3. CPU Execution – runtime/components/top_p_cpu_sampler.cc validates parameters and implements the top-K/top-P logic, including temperature application.
4. Greedy Shortcut – runtime/executor/llm_litert_mtp_drafter.cc demonstrates constructing a greedy sampler via parameter manipulation.

Configuration Examples

C++: TOP-K Configuration

#include "runtime/proto/sampler_params.pb.h"
#include "runtime/components/sampler_factory.h"

using litert::lm::proto::SamplerParameters;

// Configure TOP-K with 50 candidates and reduced temperature
SamplerParameters params;
params.set_type(SamplerParameters::TOP_K);
params.set_k(50);
params.set_p(0.0f);  // Ignored for TOP_K
params.set_temperature(0.7f);
params.set_seed(12345);

auto sampler = CreateSampler(
    Backend::CPU,
    /*output_heads=*/1,
    std::move(params),
    environment,
    /*sequence_size=*/1,
    vocab_size,
    std::nullopt);

C++: Greedy Configuration

SamplerParameters greedy;
greedy.set_type(SamplerParameters::GREEDY);
greedy.set_k(1);
greedy.set_p(0.0f);
greedy.set_temperature(1.0f);
greedy.set_seed(0);

auto greedy_sampler = CreateSampler(
    Backend::CPU, 1, std::move(greedy), env, 
    1, vocab_size, std::nullopt);

Python: Protobuf Preparation

from litert.lm.proto import sampler_params_pb2 as sp_pb2

# TOP-P configuration with pre-filtering

sampler_cfg = sp_pb2.SamplerParameters()
sampler_cfg.type = sp_pb2.SamplerParameters.TOP_P
sampler_cfg.k = 1              # Pre-filter to top token (minimal effect)

sampler_cfg.p = 0.9            # Nucleus threshold

sampler_cfg.temperature = 0.8  # Slightly conservative randomness

sampler_cfg.seed = 42

# Attach to model metadata

llm_metadata.sampler_params.CopyFrom(sampler_cfg)

Summary

• LiteRT-LM sampler parameters are defined in the SamplerParameters protobuf with fields for type, k, p, temperature, and seed.
• Four sampler types are available: TOP_K (top-k sampling), TOP_P (nucleus sampling), GREEDY (argmax), and TYPE_UNSPECIFIED (backend-dependent).
• CPU/GPU defaults use TOP_P with k=1, p=0.95, and temperature=1.0, while NPUs default to TYPE_UNSPECIFIED.
• Greedy decoding is internally implemented as TOP_P with k=1 and p=0.0.
• Source files governing this behavior include runtime/proto/sampler_params.proto, runtime/engine/engine_settings.cc, and runtime/components/top_p_cpu_sampler.cc.

Frequently Asked Questions

What is the difference between TOP_K and TOP_P in LiteRT-LM?

TOP_K restricts sampling to a fixed number (k) of highest-probability tokens regardless of their actual probability values. TOP_P (nucleus sampling) uses a dynamic cutoff based on cumulative probability, selecting the smallest set of tokens whose probabilities sum to at least p. TOP_P can adapt to the model's confidence, while TOP_K provides a fixed vocabulary size limit.

How does temperature affect sampling in LiteRT-LM?

Temperature scales logits before the softmax operation via division: logits / temperature. A value of 1.0 applies no change. Values below 1.0 sharpen the distribution (more deterministic), while values above 1.0 flatten it (more random). In runtime/components/top_p_cpu_sampler.cc, this scaling occurs before the top-K filtering and probability normalization.

When should I use TYPE_UNSPECIFIED?

Use TYPE_UNSPECIFIED when deploying on NPU or specialized hardware backends (ARTISAN) that implement their own optimized sampling logic. When this type is set, the LiteRT-LM runtime skips its internal CreateSampler factory logic and allows the hardware driver to handle token selection, which often provides better performance on accelerator hardware.

How do I force deterministic (greedy) output in LiteRT-LM?

Set type to GREEDY or manually configure parameters to k=1, p=0.0, and temperature=1.0. According to the implementation in runtime/executor/llm_litert_mtp_drafter.cc, the greedy type is equivalent to TOP_P sampling with a single candidate and zero probability threshold, ensuring the highest-logit token is always selected.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}