Difference Between Greedy Sampling and Nucleus (Top-p) Sampling in Llama

Greedy sampling deterministically selects the single highest-probability token at each generation step, while nucleus (top-p) sampling stochastically chooses from the smallest set of tokens whose cumulative probability exceeds a threshold p, enabling more diverse and creative outputs.

Understanding the difference between greedy sampling and nucleus (top-p) sampling is essential for controlling text generation behavior in large language models. In the meta-llama/llama repository, these two strategies are implemented in llama/generation.py and govern how the Llama model selects the next token during autoregressive inference. While greedy sampling prioritizes consistency by always picking the most likely token, nucleus sampling introduces controlled randomness that produces more varied and human-like continuations.

Core Differences Between Greedy and Nucleus Sampling

Decision Logic and Selection Criteria

Greedy sampling applies an argmax operation to the model's output logits, selecting the single token with the highest probability mass. In contrast, nucleus (top-p) sampling identifies the smallest subset of tokens whose cumulative probability meets or exceeds the top-p threshold (typically 0.9), then samples proportionally from within that filtered distribution.

Determinism vs Stochasticity

Greedy sampling is deterministic—given the same prompt, model weights, and context, it always generates identical output. Nucleus sampling is stochastic, utilizing torch.multinomial to randomly draw from the truncated probability distribution, resulting in different continuations across generation runs.

Diversity and Output Quality Trade-offs

Greedy decoding often produces repetitive or overly "safe" text because it consistently follows the highest probability path through the distribution. Nucleus sampling increases output diversity by allowing lower-probability tokens to compete for selection when they reside within the high-cumulative-probability nucleus, creating more creative and natural-sounding responses.

Implementation in the meta-llama/llama Repository

The pivot between these strategies occurs in llama/generation.py. When temperature <= 0, the code executes greedy selection via torch.argmax; when temperature > 0, it applies temperature scaling followed by nucleus sampling via the sample_top_p function.

Greedy Sampling Path (Lines 88-93)

if temperature > 0:
    probs = torch.softmax(logits[:, -1] / temperature, dim=-1)
    next_token = sample_top_p(probs, top_p)
else:
    next_token = torch.argmax(logits[:, -1], dim=-1)

When temperature is zero or negative, the code bypasses probability distribution sampling and executes torch.argmax on the raw logits at position [:, -1], selecting the single highest-value index deterministically.

Nucleus Sampling Implementation (Lines 99-121)

def sample_top_p(probs, p):
    probs_sort, probs_idx = torch.sort(probs, dim=-1, descending=True)
    probs_sum = torch.cumsum(probs_sort, dim=-1)
    mask = probs_sum - probs_sort > p
    probs_sort[mask] = 0.0
    probs_sort.div_(probs_sort.sum(dim=-1, keepdim=True))
    next_token = torch.multinomial(probs_sort, num_samples=1)
    next_token = torch.gather(probs_idx, -1, next_token)
    return next_token

This function first sorts probabilities in descending order using torch.sort, computes the cumulative sum with torch.cumsum, and masks out tokens where the cumulative probability exceeds the top-p threshold. It then renormalizes the remaining probabilities and samples using torch.multinomial, mapping the sample back to the original token indices via torch.gather.

Practical Usage Examples

Here is how to invoke each sampling strategy using the Llama Python API:

Greedy (Deterministic) Generation:

from llama.generation import Llama

model = Llama.build(
    ckpt_dir="checkpoints",
    tokenizer_path="tokenizer.model",
    max_seq_len=2048,
    max_batch_size=4,
)

# Temperature 0.0 forces greedy argmax selection

output = model.text_completion(
    prompts=["Explain quantum entanglement:"],
    temperature=0.0,  # Triggers greedy path in generation.py

    top_p=1.0,        # Ignored when temperature <= 0

    max_gen_len=100,
)

Nucleus (Stochastic) Generation:

# Temperature > 0 enables nucleus sampling path

output = model.text_completion(
    prompts=["Explain quantum entanglement:"],
    temperature=0.7,  # Scales logits before softmax

    top_p=0.9,        # Keep smallest set with 90% cumulative probability

    max_gen_len=100,
)

Setting temperature=0.0 disables randomness entirely, forcing the model to select the argmax at every step. Conversely, temperature=0.7 with top_p=0.9 permits the model to explore high-probability alternatives within the nucleus, producing more varied responses.

Summary

• Greedy sampling uses torch.argmax in llama/generation.py (lines 88-93) to select the single highest-probability token, producing deterministic but potentially repetitive output.
• Nucleus (top-p) sampling filters the distribution to the smallest token set with cumulative probability ≥ p, then samples stochastically via sample_top_p (lines 99-121), yielding more diverse responses.
• The temperature parameter acts as a switch in the Llama implementation: values ≤ 0 trigger greedy decoding, while values > 0 enable temperature scaling and nucleus sampling.
• Use greedy sampling when you need consistent, reproducible results for tasks like code generation or factual retrieval; use nucleus sampling when you want creative, human-like text variation for creative writing or conversational AI.

Frequently Asked Questions

Does nucleus sampling work if I set temperature to 0.0 in Llama?

No. In the meta-llama/llama implementation, setting temperature <= 0 forces the code to execute the else branch containing torch.argmax rather than the sample_top_p function, effectively disabling nucleus sampling. The top_p parameter is ignored when temperature is zero or negative, resulting in purely greedy selection.

Why does greedy sampling sometimes produce repetitive text?

Greedy sampling always selects the highest-probability token at each step, causing the model to follow the most statistically likely path through the distribution. This deterministic trajectory often leads to loops or generic phrases because the model never deviates to explore lower-probability but contextually interesting alternatives that nucleus sampling would consider.

What is a good top-p value for nucleus sampling?

A top-p value of 0.9 is commonly used as it retains the smallest set of tokens comprising approximately 90% of the probability mass, filtering out the long tail of unlikely tokens while preserving meaningful diversity. Values closer to 1.0 allow more tokens (approaching full sampling), while lower values restrict the pool to only the most probable candidates, approaching greedy behavior.

How does temperature interact with nucleus sampling in the Llama codebase?

Temperature scales the logits before the softmax operation in lines 88-89 of generation.py, flattening or sharpening the probability distribution. Higher temperatures (> 1.0) increase randomness by making probabilities more uniform, while lower temperatures (approaching 0.1) make the distribution more peaked. This scaling occurs before the sample_top_p function processes the probabilities, directly influencing which tokens qualify for inclusion in the nucleus.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}