How to Customize Temperature and Top‑P Sampling Parameters During Generation in Llama

You can customize the temperature and top_p parameters by passing them to Llama.text_completion(), Llama.chat_completion(), or the low‑level Llama.generate() method in llama/generation.py, where temperature scales logits before softmax and top_p enables nucleus sampling.

The meta-llama/llama repository provides a flexible generation API that allows fine‑grained control over text randomness and diversity. Understanding how to adjust these sampling parameters is essential for balancing creative, varied outputs against deterministic, factual responses.

Understanding Temperature and Top‑P Sampling

Two hyper‑parameters control the stochasticity of text generation in the Llama inference pipeline:

• temperature – Scales the logits before applying softmax. Lower values (e.g., 0.2) sharpen the probability distribution, making the model more deterministic. Higher values (e.g., 1.2) flatten the distribution, increasing randomness. The default is 0.6.
• top_p (nucleus sampling) – Retains the smallest set of tokens whose cumulative probability exceeds the threshold, then renormalizes the distribution over that set. The default is 0.9.

Where Sampling Parameters Are Implemented

The core generation logic resides in llama/generation.py. The Llama class exposes three public methods that accept these parameters:

1. Llama.generate() – Low‑level API that implements the actual sampling loop.
2. Llama.text_completion() – High‑level helper for text completion tasks; forwards parameters to generate().
3. Llama.chat_completion() – High‑level helper for conversational formats; also forwards to generate().

The sample_top_p helper function (lines 98‑104 in generation.py) implements the nucleus filtering logic.

How Temperature and Top‑P Work During Generation

During each decoding step, Llama.generate follows this execution flow:

1. Logit computation – The transformer produces raw logits via self.model.forward().

2. Temperature scaling – If temperature > 0, logits are divided by the temperature value before softmax:

   probs = torch.softmax(logits[:, -1] / temperature, dim=-1)

   This operation occurs at line 88 in generation.py.

3. Nucleus sampling – The sample_top_p(probs, top_p) function filters the distribution, removing low‑probability tails and sampling from the remaining token set (lines 98‑104).

4. Greedy fallback – When temperature == 0, the model bypasses sampling and selects the token with the highest logit via torch.argmax.

Practical Code Examples

Text Completion with High Temperature

Use a temperature above 1.0 to encourage creative, diverse outputs:

from llama import Llama

# Initialize model (adjust paths to your checkpoint)

llama = Llama.build(
    ckpt_dir="checkpoints/llama-2-7b",
    tokenizer_path="tokenizer.model",
    max_seq_len=2048,
    max_batch_size=4,
)

prompt = "Once upon a time in a distant galaxy"
result = llama.text_completion(
    prompts=[prompt],
    temperature=1.2,   # Higher randomness

    top_p=0.95,        # Broader nucleus

    max_gen_len=100,
)

print(result[0]["generation"])

Chat Completion with Deterministic Settings

Reduce temperature for factual, consistent responses in conversational contexts:

dialog = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What are the benefits of electric cars?"},
]

responses = llama.chat_completion(
    dialogs=[dialog],
    temperature=0.2,   # Sharper distribution

    top_p=0.8,         # Tighter nucleus

    max_gen_len=150,
)

print(responses[0]["generation"]["content"])

Direct Batch Generation

For advanced use cases, call generate() directly to process multiple prompts with shared parameters:

prompt_tokens = [
    llama.tokenizer.encode("Explain quantum tunneling.", bos=True, eos=False),
    llama.tokenizer.encode("Give me a quick recipe for pancakes.", bos=True, eos=False),
]

tokens, _ = llama.generate(
    prompt_tokens=prompt_tokens,
    max_gen_len=80,
    temperature=0.7,
    top_p=0.85,
)

for t in tokens:
    print(llama.tokenizer.decode(t))

Summary

• The meta-llama/llama repository implements sampling controls in llama/generation.py within the Llama class.
• temperature scales logits before softmax; default is 0.6, with lower values producing deterministic output and higher values increasing randomness.
• top_p enables nucleus sampling; default is 0.9, filtering the token distribution to the most probable subset.
• When temperature is set to 0, the model uses greedy decoding (torch.argmax) instead of sampling.
• All public methods—generate(), text_completion(), and chat_completion()—accept these parameters and forward them to the core sampling logic.

Frequently Asked Questions

What happens when temperature is set to 0?

When temperature is 0, the Llama.generate method bypasses probabilistic sampling entirely. Instead of scaling logits and applying softmax, the code executes torch.argmax to select the token with the highest logit score at each step. This produces deterministic, greedy decoding where the same prompt always yields the identical output.

How does top_p differ from top_k sampling?

Top‑p (nucleus sampling) selects the smallest set of tokens whose cumulative probability exceeds the threshold p, which means the number of tokens kept varies dynamically based on the distribution shape. Top‑k sampling, by contrast, always retains exactly the k highest‑probability tokens regardless of their cumulative mass. The Llama implementation uses top_p exclusively, as implemented in the sample_top_p function in llama/generation.py.

Can I use different temperature values for different prompts in a batch?

No. The current implementation in llama/generation.py applies a single temperature and top_p value to all sequences in a batch. The parameters are scalars passed to generate(), and the sampling logic processes the entire batch with these shared values. To use different temperatures, you must process prompts in separate calls or modify the source code to accept per‑sequence parameters.

Where are the default values for temperature and top_p defined?

The default values—0.6 for temperature and 0.9 for top_p—are defined in the method signature of Llama.generate in llama/generation.py (lines 31‑36). These defaults propagate to the high‑level convenience methods text_completion and chat_completion, which forward the values to generate when not explicitly overridden by the caller.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}