# How to Customize Temperature and Top‑P Sampling Parameters During Generation in Llama

> Discover how to customize temperature and top_p sampling parameters for generation in Llama. Learn to control model output by adjusting these key settings in text generation.

- Repository: [Meta Llama/llama](https://github.com/meta-llama/llama)
- Tags: how-to-guide
- Published: 2026-03-05

---

**You can customize the `temperature` and `top_p` parameters by passing them to `Llama.text_completion()`, `Llama.chat_completion()`, or the low‑level `Llama.generate()` method in [`llama/generation.py`](https://github.com/meta-llama/llama/blob/main/llama/generation.py), where temperature scales logits before softmax and top_p enables nucleus sampling.**

The `meta-llama/llama` repository provides a flexible generation API that allows fine‑grained control over text randomness and diversity. Understanding how to adjust these sampling parameters is essential for balancing creative, varied outputs against deterministic, factual responses.

## Understanding Temperature and Top‑P Sampling

Two hyper‑parameters control the stochasticity of text generation in the Llama inference pipeline:

- **`temperature`** – Scales the logits before applying softmax. Lower values (e.g., 0.2) sharpen the probability distribution, making the model more deterministic. Higher values (e.g., 1.2) flatten the distribution, increasing randomness. The default is **0.6**.
- **`top_p`** (nucleus sampling) – Retains the smallest set of tokens whose cumulative probability exceeds the threshold, then renormalizes the distribution over that set. The default is **0.9**.

## Where Sampling Parameters Are Implemented

The core generation logic resides in **[`llama/generation.py`](https://github.com/meta-llama/llama/blob/main/llama/generation.py)**. The `Llama` class exposes three public methods that accept these parameters:

1. **`Llama.generate()`** – Low‑level API that implements the actual sampling loop.
2. **`Llama.text_completion()`** – High‑level helper for text completion tasks; forwards parameters to `generate()`.
3. **`Llama.chat_completion()`** – High‑level helper for conversational formats; also forwards to `generate()`.

The `sample_top_p` helper function (lines 98‑104 in [`generation.py`](https://github.com/meta-llama/llama/blob/main/generation.py)) implements the nucleus filtering logic.

## How Temperature and Top‑P Work During Generation

During each decoding step, `Llama.generate` follows this execution flow:

1. **Logit computation** – The transformer produces raw logits via `self.model.forward()`.
2. **Temperature scaling** – If `temperature > 0`, logits are divided by the temperature value before softmax:

   ```python
   probs = torch.softmax(logits[:, -1] / temperature, dim=-1)
   ```

   This operation occurs at line 88 in [`generation.py`](https://github.com/meta-llama/llama/blob/main/generation.py).
3. **Nucleus sampling** – The `sample_top_p(probs, top_p)` function filters the distribution, removing low‑probability tails and sampling from the remaining token set (lines 98‑104).
4. **Greedy fallback** – When `temperature == 0`, the model bypasses sampling and selects the token with the highest logit via `torch.argmax`.

## Practical Code Examples

### Text Completion with High Temperature

Use a temperature above 1.0 to encourage creative, diverse outputs:

```python
from llama import Llama

# Initialize model (adjust paths to your checkpoint)

llama = Llama.build(
    ckpt_dir="checkpoints/llama-2-7b",
    tokenizer_path="tokenizer.model",
    max_seq_len=2048,
    max_batch_size=4,
)

prompt = "Once upon a time in a distant galaxy"
result = llama.text_completion(
    prompts=[prompt],
    temperature=1.2,   # Higher randomness

    top_p=0.95,        # Broader nucleus

    max_gen_len=100,
)

print(result[0]["generation"])

```

### Chat Completion with Deterministic Settings

Reduce temperature for factual, consistent responses in conversational contexts:

```python
dialog = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What are the benefits of electric cars?"},
]

responses = llama.chat_completion(
    dialogs=[dialog],
    temperature=0.2,   # Sharper distribution

    top_p=0.8,         # Tighter nucleus

    max_gen_len=150,
)

print(responses[0]["generation"]["content"])

```

### Direct Batch Generation

For advanced use cases, call `generate()` directly to process multiple prompts with shared parameters:

```python
prompt_tokens = [
    llama.tokenizer.encode("Explain quantum tunneling.", bos=True, eos=False),
    llama.tokenizer.encode("Give me a quick recipe for pancakes.", bos=True, eos=False),
]

tokens, _ = llama.generate(
    prompt_tokens=prompt_tokens,
    max_gen_len=80,
    temperature=0.7,
    top_p=0.85,
)

for t in tokens:
    print(llama.tokenizer.decode(t))

```

## Summary

- The `meta-llama/llama` repository implements sampling controls in **[`llama/generation.py`](https://github.com/meta-llama/llama/blob/main/llama/generation.py)** within the `Llama` class.
- **`temperature`** scales logits before softmax; default is **0.6**, with lower values producing deterministic output and higher values increasing randomness.
- **`top_p`** enables nucleus sampling; default is **0.9**, filtering the token distribution to the most probable subset.
- When `temperature` is set to **0**, the model uses greedy decoding (`torch.argmax`) instead of sampling.
- All public methods—`generate()`, `text_completion()`, and `chat_completion()`—accept these parameters and forward them to the core sampling logic.

## Frequently Asked Questions

### What happens when temperature is set to 0?

When `temperature` is **0**, the `Llama.generate` method bypasses probabilistic sampling entirely. Instead of scaling logits and applying softmax, the code executes `torch.argmax` to select the token with the highest logit score at each step. This produces deterministic, greedy decoding where the same prompt always yields the identical output.

### How does top_p differ from top_k sampling?

**Top‑p** (nucleus sampling) selects the smallest set of tokens whose cumulative probability exceeds the threshold `p`, which means the number of tokens kept varies dynamically based on the distribution shape. **Top‑k** sampling, by contrast, always retains exactly the `k` highest‑probability tokens regardless of their cumulative mass. The Llama implementation uses `top_p` exclusively, as implemented in the `sample_top_p` function in [`llama/generation.py`](https://github.com/meta-llama/llama/blob/main/llama/generation.py).

### Can I use different temperature values for different prompts in a batch?

No. The current implementation in [`llama/generation.py`](https://github.com/meta-llama/llama/blob/main/llama/generation.py) applies a single `temperature` and `top_p` value to all sequences in a batch. The parameters are scalars passed to `generate()`, and the sampling logic processes the entire batch with these shared values. To use different temperatures, you must process prompts in separate calls or modify the source code to accept per‑sequence parameters.

### Where are the default values for temperature and top_p defined?

The default values—**0.6** for `temperature` and **0.9** for `top_p`—are defined in the method signature of `Llama.generate` in [`llama/generation.py`](https://github.com/meta-llama/llama/blob/main/llama/generation.py) (lines 31‑36). These defaults propagate to the high‑level convenience methods `text_completion` and `chat_completion`, which forward the values to `generate` when not explicitly overridden by the caller.