# How Temperature, Top-p, and Seed Sampling Strategy Controls Model Output in gpullama3.java

> Control gpullama3.java model output with temperature, top-p, and seed sampling. Learn how hierarchical sampling affects determinism and token selection for precise results.

- Repository: [Beehive lab/gpullama3.java](https://github.com/beehive-lab/gpullama3.java)
- Tags: deep-dive
- Published: 2026-02-26

---

**The gpullama3.java repository implements a hierarchical sampling strategy in `Sampler.selectSampler` that applies temperature scaling to logits, optionally filters tokens via nucleus (top-p) sampling, and uses seeded random number generation to control determinism.**

The gpullama3.java inference engine provides precise control over text generation randomness through three configurable parameters managed by the sampling strategy. Located in [`org/beehive/gpullama3/inference/sampler/Sampler.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/org/beehive/gpullama3/inference/sampler/Sampler.java), the implementation balances computational efficiency with statistical flexibility, allowing developers to tune outputs from deterministic argmax selection to highly stochastic generation.

## The Sampling Strategy Architecture

The sampling pipeline is constructed once per generation session via `Sampler.selectSampler(int vocabSize, float temperature, float topp, long rngSeed)`. This factory method returns a functional interface that transforms raw model logits into discrete token selections through a three-stage pipeline: temperature scaling, softmax normalization, and stochastic or greedy selection.

### Temperature Scaling Implementation

**Temperature** controls the sharpness of the probability distribution by scaling logits before softmax normalization. In the source code, this is implemented via `divideInPlace` operations on tensor structures:

```java
if (logits instanceof FloatTensor) {
    FloatTensor t = (FloatTensor) logits;
    t.divideInPlace(0, t.size(), temperature);
    t.softmaxInPlace(0, t.size());
}

```

When `temperature == 0.0f`, the system bypasses stochastic sampling entirely and uses `Sampler.TENSOR_ARGMAX` for greedy selection. Values between 0.0 and 1.0 make the distribution peakier (more deterministic), while values greater than 1.0 flatten the distribution, increasing randomness.

### Top-p (Nucleus) Filtering Logic

**Top-p** (nucleus sampling) dynamically truncates the vocabulary to the smallest set of tokens whose cumulative probability exceeds the threshold `topp`. The `selectSampler` method chooses between two inner samplers based on this parameter:

- **`CategoricalSampler`** – Used when `topp <= 0 || topp >= 1`, sampling from the full softmax distribution.
- **`ToppSampler`** – Activated when `0 < topp < 1`, implementing a heap-based algorithm that discards low-probability tails while preserving high-likelihood tokens.

This approach maintains vocabulary flexibility unlike static top-k sampling, adapting the candidate pool to the model's confidence for each position.

### Seed-Based Random Generation

The **seed** parameter ensures reproducible stochasticity by instantiating a `RandomGenerator` via `RandomGeneratorFactory.getDefault().create(rngSeed)`. This generator is shared across all inner samplers:

```java
RandomGenerator rng = RandomGeneratorFactory.getDefault().create(rngSeed);
Sampler inner = (topp <= 0 || topp >= 1)
    ? new CategoricalSampler(rng)
    : new ToppSampler(vocabSize, topp, rng);

```

With identical seeds, temperature values above 0.0, and valid top-p settings, the generated token sequence remains bitwise identical across runs.

## Building the Sampler Pipeline

The factory method implements a decision tree that prioritizes deterministic behavior when possible. If temperature equals exactly 0.0, it returns the argmax sampler immediately. Otherwise, it constructs a lambda that applies temperature scaling, computes softmax probabilities, and delegates to the appropriate inner sampler:

```java
static Sampler selectSampler(int vocabSize,
                            float temperature,
                            float topp,
                            long rngSeed) {
    if (temperature == 0.0f) {
        return Sampler.TENSOR_ARGMAX;
    }

    RandomGenerator rng = RandomGeneratorFactory.getDefault().create(rngSeed);
    Sampler inner = (topp <= 0 || topp >= 1)
            ? new CategoricalSampler(rng)
            : new ToppSampler(vocabSize, topp, rng);

    return logits -> {
        // Temperature scaling and softmax application
        if (logits instanceof FloatTensor) {
            FloatTensor t = (FloatTensor) logits;
            t.divideInPlace(0, t.size(), temperature);
            t.softmaxInPlace(0, t.size());
        } else if (logits instanceof FloatArray) {
            FloatArray a = (FloatArray) logits;
            FloatArrayUtils.divideInPlace(a, 0, a.getSize(), temperature);
            FloatArrayUtils.softmaxInPlace(a, 0, a.getSize());
        } else {
            throw new IllegalArgumentException("Unsupported logits type");
        }
        return inner.sampleToken(logits);
    };
}

```

This design separates the mathematical transformation of logits from the stochastic selection mechanism, enabling both CPU (`FloatTensor`) and GPU (`FloatArray`) code paths to share identical sampling logic.

## Practical Implementation Examples

### Parsing CLI Arguments in Options.java

The sampling parameters originate from command-line arguments parsed in [`Options.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Options.java). The `Options.parseOptions(String[] args)` method extracts temperature, top-p, and seed values for injection into the sampler factory:

```java
Options options = Options.parseOptions(args);
Model model = ...;
Sampler sampler = Sampler.createSampler(model, options);

```

Source: [`Options.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Options.java) handles argument parsing for `--temperature`, `--top-p`, and `--seed` flags.

### Integrating with Model Generation Loops

The configured sampler participates in the autoregressive generation cycle via `Model.generateTokens`. This method accepts the sampler as a strategy object and invokes it for each token position:

```java
State state = model.createNewState();
int startPos = 0;
List<Integer> prompt = ...;
Set<Integer> stop = model.chatFormat().getStopTokens();

List<Integer> generated = model.generateTokens(
        state,
        startPos,
        prompt,
        stop,
        options.maxTokens(),
        sampler,
        options.echo(),
        token -> System.out.print(model.tokenizer().decode(List.of(token)))
);

```

Source: [`Model.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Model.java) defines the generation contract at line 66-68, while concrete implementations (e.g., [`Llama.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Llama.java)) provide the actual inference loop.

### Command-Line Usage Example

To exercise all three sampling parameters simultaneously:

```bash
jbang Llama3.java \
    --model /path/to/llama3.gguf \
    --prompt "Explain quantum computing" \
    --temperature 0.7 \
    --top-p 0.9 \
    --seed 12345

```

Executing this command twice produces identical outputs because the seed fixes the random sequence, while the temperature (0.7) and top-p (0.9) values determine the statistical properties of the selection process.

## Summary

- **Temperature** scales logits via `divideInPlace` operations; 0.0 triggers greedy argmax, while values above 0.0 enable stochastic sampling with adjustable entropy.
- **Top-p** selects between `CategoricalSampler` (full distribution) and `ToppSampler` (nucleus filtering) based on whether the parameter falls within the exclusive range (0, 1).
- **Seed** initializes a `RandomGenerator` that ensures reproducible stochastic behavior across identical runs when combined with non-zero temperature.
- The `Sampler.selectSampler` factory in [`Sampler.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Sampler.java) orchestrates these components, supporting both CPU and GPU tensor types through polymorphic dispatch.

## Frequently Asked Questions

### What happens when temperature is set to 0.0 in gpullama3.java?

When temperature equals exactly 0.0f, `Sampler.selectSampler` returns `Sampler.TENSOR_ARGMAX`, which performs deterministic greedy selection by choosing the token with the highest logit value. This bypasses all random number generation and ignores the top-p and seed parameters, producing identical outputs for identical inputs.

### How does top-p differ from top-k sampling in this implementation?

Top-p (nucleus) sampling dynamically adjusts the candidate pool size based on cumulative probability mass, whereas top-k uses a fixed integer cutoff. In gpullama3.java, `ToppSampler` constructs a min-heap of token probabilities and truncates once the cumulative probability exceeds the threshold, effectively implementing variable-width nucleus filtering without hard vocabulary limits.

### Why does the sampler accept a seed even when using greedy decoding?

The seed parameter is accepted for API consistency but only affects behavior when `temperature > 0.0f`. The `RandomGenerator` instantiation occurs inside the stochastic branch of `selectSampler`, meaning greedy mode (temperature 0.0) never initializes the RNG, making the seed irrelevant for deterministic outputs.

### Can sampling parameters be changed during text generation?

The current architecture constructs the `Sampler` instance once before the generation loop begins in `Model.generateTokens`. To modify parameters mid-generation, you would need to implement custom state management that calls `Sampler.selectSampler` with new parameters and replaces the sampler instance in the inference state, though this is not supported by the default CLI interface.