# Difference Between Greedy Sampling and Nucleus (Top-p) Sampling in Llama

> Understand the difference between greedy sampling and nucleus (top-p) sampling in Llama. Discover how top-p sampling creates more diverse and creative text generation.

- Repository: [Meta Llama/llama](https://github.com/meta-llama/llama)
- Tags: deep-dive
- Published: 2026-03-05

---

**Greedy sampling deterministically selects the single highest-probability token at each generation step, while nucleus (top-p) sampling stochastically chooses from the smallest set of tokens whose cumulative probability exceeds a threshold p, enabling more diverse and creative outputs.**

Understanding the difference between greedy sampling and nucleus (top-p) sampling is essential for controlling text generation behavior in large language models. In the meta-llama/llama repository, these two strategies are implemented in [`llama/generation.py`](https://github.com/meta-llama/llama/blob/main/llama/generation.py) and govern how the Llama model selects the next token during autoregressive inference. While greedy sampling prioritizes consistency by always picking the most likely token, nucleus sampling introduces controlled randomness that produces more varied and human-like continuations.

## Core Differences Between Greedy and Nucleus Sampling

### Decision Logic and Selection Criteria

**Greedy sampling** applies an `argmax` operation to the model's output logits, selecting the single token with the highest probability mass. In contrast, **nucleus (top-p) sampling** identifies the smallest subset of tokens whose cumulative probability meets or exceeds the top-p threshold (typically 0.9), then samples proportionally from within that filtered distribution.

### Determinism vs Stochasticity

Greedy sampling is **deterministic**—given the same prompt, model weights, and context, it always generates identical output. Nucleus sampling is **stochastic**, utilizing `torch.multinomial` to randomly draw from the truncated probability distribution, resulting in different continuations across generation runs.

### Diversity and Output Quality Trade-offs

Greedy decoding often produces **repetitive or overly "safe" text** because it consistently follows the highest probability path through the distribution. Nucleus sampling increases **output diversity** by allowing lower-probability tokens to compete for selection when they reside within the high-cumulative-probability nucleus, creating more creative and natural-sounding responses.

## Implementation in the meta-llama/llama Repository

The pivot between these strategies occurs in [`llama/generation.py`](https://github.com/meta-llama/llama/blob/main/llama/generation.py). When `temperature <= 0`, the code executes greedy selection via `torch.argmax`; when `temperature > 0`, it applies temperature scaling followed by nucleus sampling via the `sample_top_p` function.

### Greedy Sampling Path (Lines 88-93)

```python
if temperature > 0:
    probs = torch.softmax(logits[:, -1] / temperature, dim=-1)
    next_token = sample_top_p(probs, top_p)
else:
    next_token = torch.argmax(logits[:, -1], dim=-1)

```

When temperature is zero or negative, the code bypasses probability distribution sampling and executes `torch.argmax` on the raw logits at position `[:, -1]`, selecting the single highest-value index deterministically.

### Nucleus Sampling Implementation (Lines 99-121)

```python
def sample_top_p(probs, p):
    probs_sort, probs_idx = torch.sort(probs, dim=-1, descending=True)
    probs_sum = torch.cumsum(probs_sort, dim=-1)
    mask = probs_sum - probs_sort > p
    probs_sort[mask] = 0.0
    probs_sort.div_(probs_sort.sum(dim=-1, keepdim=True))
    next_token = torch.multinomial(probs_sort, num_samples=1)
    next_token = torch.gather(probs_idx, -1, next_token)
    return next_token

```

This function first sorts probabilities in descending order using `torch.sort`, computes the cumulative sum with `torch.cumsum`, and masks out tokens where the cumulative probability exceeds the top-p threshold. It then renormalizes the remaining probabilities and samples using `torch.multinomial`, mapping the sample back to the original token indices via `torch.gather`.

## Practical Usage Examples

Here is how to invoke each sampling strategy using the Llama Python API:

**Greedy (Deterministic) Generation:**

```python
from llama.generation import Llama

model = Llama.build(
    ckpt_dir="checkpoints",
    tokenizer_path="tokenizer.model",
    max_seq_len=2048,
    max_batch_size=4,
)

# Temperature 0.0 forces greedy argmax selection

output = model.text_completion(
    prompts=["Explain quantum entanglement:"],
    temperature=0.0,  # Triggers greedy path in generation.py

    top_p=1.0,        # Ignored when temperature <= 0

    max_gen_len=100,
)

```

**Nucleus (Stochastic) Generation:**

```python

# Temperature > 0 enables nucleus sampling path

output = model.text_completion(
    prompts=["Explain quantum entanglement:"],
    temperature=0.7,  # Scales logits before softmax

    top_p=0.9,        # Keep smallest set with 90% cumulative probability

    max_gen_len=100,
)

```

Setting `temperature=0.0` disables randomness entirely, forcing the model to select the argmax at every step. Conversely, `temperature=0.7` with `top_p=0.9` permits the model to explore high-probability alternatives within the nucleus, producing more varied responses.

## Summary

- **Greedy sampling** uses `torch.argmax` in [`llama/generation.py`](https://github.com/meta-llama/llama/blob/main/llama/generation.py) (lines 88-93) to select the single highest-probability token, producing deterministic but potentially repetitive output.
- **Nucleus (top-p) sampling** filters the distribution to the smallest token set with cumulative probability ≥ p, then samples stochastically via `sample_top_p` (lines 99-121), yielding more diverse responses.
- The **temperature parameter** acts as a switch in the Llama implementation: values ≤ 0 trigger greedy decoding, while values > 0 enable temperature scaling and nucleus sampling.
- **Use greedy sampling** when you need consistent, reproducible results for tasks like code generation or factual retrieval; **use nucleus sampling** when you want creative, human-like text variation for creative writing or conversational AI.

## Frequently Asked Questions

### Does nucleus sampling work if I set temperature to 0.0 in Llama?

No. In the meta-llama/llama implementation, setting `temperature <= 0` forces the code to execute the `else` branch containing `torch.argmax` rather than the `sample_top_p` function, effectively disabling nucleus sampling. The `top_p` parameter is ignored when temperature is zero or negative, resulting in purely greedy selection.

### Why does greedy sampling sometimes produce repetitive text?

Greedy sampling always selects the highest-probability token at each step, causing the model to follow the most statistically likely path through the distribution. This deterministic trajectory often leads to loops or generic phrases because the model never deviates to explore lower-probability but contextually interesting alternatives that nucleus sampling would consider.

### What is a good top-p value for nucleus sampling?

A top-p value of **0.9** is commonly used as it retains the smallest set of tokens comprising approximately 90% of the probability mass, filtering out the long tail of unlikely tokens while preserving meaningful diversity. Values closer to 1.0 allow more tokens (approaching full sampling), while lower values restrict the pool to only the most probable candidates, approaching greedy behavior.

### How does temperature interact with nucleus sampling in the Llama codebase?

Temperature scales the logits before the softmax operation in lines 88-89 of [`generation.py`](https://github.com/meta-llama/llama/blob/main/generation.py), flattening or sharpening the probability distribution. Higher temperatures (> 1.0) increase randomness by making probabilities more uniform, while lower temperatures (approaching 0.1) make the distribution more peaked. This scaling occurs before the `sample_top_p` function processes the probabilities, directly influencing which tokens qualify for inclusion in the nucleus.