# How DFlash Handles Temperature and Sampling During Text Generation

> Discover how DFlash manages text generation randomness using its unified sample function, blending greedy decoding with probabilistic sampling for optimal results.

- Repository: [Z Lab/dflash](https://github.com/z-lab/dflash)
- Tags: deep-dive
- Published: 2026-04-17

---

**DFlash controls text generation randomness through a unified `sample` function that switches between greedy decoding at near-zero temperatures and probabilistic sampling via softmax-scaled logits at higher temperatures.**

DFlash is a speculative decoding framework developed by the `z-lab/dflash` repository that accelerates large language model inference. Understanding how DFlash handles temperature and sampling during text generation is essential for controlling output randomness across both draft and target model stages.

## Temperature-Controlled Sampling Logic

The core sampling mechanism resides in [`dflash/model.py`](https://github.com/z-lab/dflash/blob/main/dflash/model.py), where a lightweight helper function determines whether to generate tokens deterministically or stochastically based on the temperature parameter.

### The Core Sampling Function

The `sample` function (lines 48-55) implements the temperature scaling logic:

```python
def sample(logits, temperature):
    if temperature < 1e-5:
        # Greedy decoding: select highest probability token

        return torch.argmax(logits, dim=-1)
    else:
        # Stochastic sampling: scale logits by temperature

        probs = torch.softmax(logits / temperature, dim=-1)
        return torch.multinomial(probs, num_samples=1).squeeze(-1)

```

When `temperature` falls below `1e-5`, DFlash executes **greedy decoding** using `torch.argmax`. For any positive temperature value, the function **scales the logits** by dividing by the temperature parameter, applies `softmax` to obtain probabilities, and samples via `torch.multinomial`.

### Greedy vs. Stochastic Decoding

The temperature parameter acts as a global scaling factor across the entire generation pipeline:

- **Temperature ≈ 0**: Deterministic output; the model always selects the highest probability token
- **Temperature > 0**: Stochastic output; lower values (0.1-0.5) produce focused, conservative text while higher values (0.8-1.2) increase diversity and creativity

## How Temperature Flows Through the Generation Pipeline

DFlash propagates the temperature argument through the entire call chain: `spec_generate` → `dflash_generate` → `sample`. This ensures consistent sampling behavior across different generation stages.

### First-Token Generation

After the initial prefill pass, DFlash generates the first new token using the same `sample` function (lines 96-99 in [`dflash/model.py`](https://github.com/z-lab/dflash/blob/main/dflash/model.py)):

```python

# First token sampling

first_token = sample(logits, temperature)
output_ids = torch.cat([input_ids, first_token.unsqueeze(0)], dim=1)

```

The temperature supplied to `dflash_generate` directly influences this initial sampling step, ensuring that the randomness level is established from the first generated token.

### Draft Model Sampling

When operating with `block_size > 1`, DFlash samples **draft logits** produced by the draft model. The draft tokens are generated using the identical `sample(draft_logits, temperature)` call before being validated against the target model's posterior.

This means both the smaller draft model and the larger target model use the same temperature-controlled sampling logic, maintaining coherence between speculative and verification phases.

### Posterior Sampling

For each decoding step, the target model's logits undergo sampling again (lines 134-136):

```python

# Posterior sampling after verification

new_token = sample(target_logits, temperature)
output_ids = torch.cat([output_ids, new_token.unsqueeze(0)], dim=1)

```

The sampled token becomes the final output for that position, and the acceptance length determines how many draft tokens are retained. Throughout this process, the temperature parameter remains constant, ensuring uniform randomness across the speculative decoding loop.

## Practical Usage Examples

The following example demonstrates how to invoke DFlash generation with temperature control:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
from dflash.model import DFlashDraftModel

# Load models

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.8B")
target = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.8B")
draft = DFlashDraftModel.from_pretrained("Qwen/Qwen3-1.8B")

# Prepare input

prompt = "Explain quantum entanglement in simple terms."
input_ids = tokenizer(prompt, return_tensors="pt").input_ids

# Generate with temperature 0.7 (stochastic)

output_ids = draft.spec_generate(
    target=target,
    input_ids=input_ids,
    max_new_tokens=100,
    stop_token_ids=[tokenizer.eos_token_id],
    temperature=0.7,
)

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

```

To adjust randomness levels:

```python

# Greedy decoding for deterministic output

greedy_output = draft.spec_generate(..., temperature=0.0)

# High temperature for creative, diverse generation

creative_output = draft.spec_generate(..., temperature=1.2)

```

## Summary

- **Unified sampling function**: DFlash uses a single `sample` function in [`dflash/model.py`](https://github.com/z-lab/dflash/blob/main/dflash/model.py) (lines 48-55) that switches between greedy `argmax` and temperature-scaled multinomial sampling based on the temperature threshold of `1e-5`.

- **Temperature propagation**: The temperature parameter flows through `spec_generate` → `dflash_generate` → `sample`, ensuring consistent randomness across first-token generation, draft model sampling, and posterior verification.

- **Deterministic vs. stochastic**: Temperatures below `1e-5` trigger deterministic greedy decoding, while positive temperatures enable probabilistic sampling where lower values increase focus and higher values increase diversity.

## Frequently Asked Questions

### What happens when I set temperature to 0 in DFlash?

When you set temperature to 0 (or any value below `1e-5`), DFlash executes greedy decoding in the `sample` function. The code path calls `torch.argmax(logits, dim=-1)` to select the highest probability token at each step, resulting in deterministic, reproducible output regardless of how many times you run the generation.

### How does DFlash temperature sampling differ from standard Hugging Face transformers?

While both frameworks use temperature scaling (`logits / temperature`), DFlash implements this within a speculative decoding context where the same temperature must synchronize both draft and target model sampling. In [`dflash/model.py`](https://github.com/z-lab/dflash/blob/main/dflash/model.py), the `sample` function is reused across draft token generation, first-token selection, and posterior verification, ensuring consistent randomness that standard transformers handles independently in their `generate` method.

### Can I use different temperatures for draft and target models in DFlash?

According to the source code in [`dflash/model.py`](https://github.com/z-lab/dflash/blob/main/dflash/model.py), DFlash uses a single temperature parameter that propagates through the entire call chain from `spec_generate` through `dflash_generate` to the `sample` function. There is no mechanism to specify separate temperatures for draft versus target model sampling; both models share the same temperature value to maintain coherence in the speculative decoding acceptance criteria.

### Where is the sampling logic implemented in the DFlash codebase?

The core sampling logic resides in [`dflash/model.py`](https://github.com/z-lab/dflash/blob/main/dflash/model.py) at lines 48-55, within the `sample` helper function. This function handles both greedy decoding (temperature < 1e-5) and temperature-scaled multinomial sampling. The same file contains the generation loop at `dflash_generate` (starting around line 70) and the high-level API `spec_generate`, both of which invoke the `sample` function for token selection.