How Temperature, Top-p, and Seed Sampling Strategy Controls Model Output in gpullama3.java

The gpullama3.java repository implements a hierarchical sampling strategy in Sampler.selectSampler that applies temperature scaling to logits, optionally filters tokens via nucleus (top-p) sampling, and uses seeded random number generation to control determinism.

The gpullama3.java inference engine provides precise control over text generation randomness through three configurable parameters managed by the sampling strategy. Located in org/beehive/gpullama3/inference/sampler/Sampler.java, the implementation balances computational efficiency with statistical flexibility, allowing developers to tune outputs from deterministic argmax selection to highly stochastic generation.

The Sampling Strategy Architecture

The sampling pipeline is constructed once per generation session via Sampler.selectSampler(int vocabSize, float temperature, float topp, long rngSeed). This factory method returns a functional interface that transforms raw model logits into discrete token selections through a three-stage pipeline: temperature scaling, softmax normalization, and stochastic or greedy selection.

Temperature Scaling Implementation

Temperature controls the sharpness of the probability distribution by scaling logits before softmax normalization. In the source code, this is implemented via divideInPlace operations on tensor structures:

if (logits instanceof FloatTensor) {
    FloatTensor t = (FloatTensor) logits;
    t.divideInPlace(0, t.size(), temperature);
    t.softmaxInPlace(0, t.size());
}

When temperature == 0.0f, the system bypasses stochastic sampling entirely and uses Sampler.TENSOR_ARGMAX for greedy selection. Values between 0.0 and 1.0 make the distribution peakier (more deterministic), while values greater than 1.0 flatten the distribution, increasing randomness.

Top-p (Nucleus) Filtering Logic

Top-p (nucleus sampling) dynamically truncates the vocabulary to the smallest set of tokens whose cumulative probability exceeds the threshold topp. The selectSampler method chooses between two inner samplers based on this parameter:

• CategoricalSampler – Used when topp <= 0 || topp >= 1, sampling from the full softmax distribution.
• ToppSampler – Activated when 0 < topp < 1, implementing a heap-based algorithm that discards low-probability tails while preserving high-likelihood tokens.

This approach maintains vocabulary flexibility unlike static top-k sampling, adapting the candidate pool to the model's confidence for each position.

Seed-Based Random Generation

The seed parameter ensures reproducible stochasticity by instantiating a RandomGenerator via RandomGeneratorFactory.getDefault().create(rngSeed). This generator is shared across all inner samplers:

RandomGenerator rng = RandomGeneratorFactory.getDefault().create(rngSeed);
Sampler inner = (topp <= 0 || topp >= 1)
    ? new CategoricalSampler(rng)
    : new ToppSampler(vocabSize, topp, rng);

With identical seeds, temperature values above 0.0, and valid top-p settings, the generated token sequence remains bitwise identical across runs.

Building the Sampler Pipeline

The factory method implements a decision tree that prioritizes deterministic behavior when possible. If temperature equals exactly 0.0, it returns the argmax sampler immediately. Otherwise, it constructs a lambda that applies temperature scaling, computes softmax probabilities, and delegates to the appropriate inner sampler:

static Sampler selectSampler(int vocabSize,
                            float temperature,
                            float topp,
                            long rngSeed) {
    if (temperature == 0.0f) {
        return Sampler.TENSOR_ARGMAX;
    }

    RandomGenerator rng = RandomGeneratorFactory.getDefault().create(rngSeed);
    Sampler inner = (topp <= 0 || topp >= 1)
            ? new CategoricalSampler(rng)
            : new ToppSampler(vocabSize, topp, rng);

    return logits -> {
        // Temperature scaling and softmax application
        if (logits instanceof FloatTensor) {
            FloatTensor t = (FloatTensor) logits;
            t.divideInPlace(0, t.size(), temperature);
            t.softmaxInPlace(0, t.size());
        } else if (logits instanceof FloatArray) {
            FloatArray a = (FloatArray) logits;
            FloatArrayUtils.divideInPlace(a, 0, a.getSize(), temperature);
            FloatArrayUtils.softmaxInPlace(a, 0, a.getSize());
        } else {
            throw new IllegalArgumentException("Unsupported logits type");
        }
        return inner.sampleToken(logits);
    };
}

This design separates the mathematical transformation of logits from the stochastic selection mechanism, enabling both CPU (FloatTensor) and GPU (FloatArray) code paths to share identical sampling logic.

Practical Implementation Examples

Parsing CLI Arguments in Options.java

The sampling parameters originate from command-line arguments parsed in Options.java. The Options.parseOptions(String[] args) method extracts temperature, top-p, and seed values for injection into the sampler factory:

Options options = Options.parseOptions(args);
Model model = ...;
Sampler sampler = Sampler.createSampler(model, options);

Source: Options.java handles argument parsing for --temperature, --top-p, and --seed flags.

Integrating with Model Generation Loops

The configured sampler participates in the autoregressive generation cycle via Model.generateTokens. This method accepts the sampler as a strategy object and invokes it for each token position:

State state = model.createNewState();
int startPos = 0;
List<Integer> prompt = ...;
Set<Integer> stop = model.chatFormat().getStopTokens();

List<Integer> generated = model.generateTokens(
        state,
        startPos,
        prompt,
        stop,
        options.maxTokens(),
        sampler,
        options.echo(),
        token -> System.out.print(model.tokenizer().decode(List.of(token)))
);

Source: Model.java defines the generation contract at line 66-68, while concrete implementations (e.g., Llama.java) provide the actual inference loop.

Command-Line Usage Example

To exercise all three sampling parameters simultaneously:

jbang Llama3.java \
    --model /path/to/llama3.gguf \
    --prompt "Explain quantum computing" \
    --temperature 0.7 \
    --top-p 0.9 \
    --seed 12345

Executing this command twice produces identical outputs because the seed fixes the random sequence, while the temperature (0.7) and top-p (0.9) values determine the statistical properties of the selection process.

Summary

• Temperature scales logits via divideInPlace operations; 0.0 triggers greedy argmax, while values above 0.0 enable stochastic sampling with adjustable entropy.
• Top-p selects between CategoricalSampler (full distribution) and ToppSampler (nucleus filtering) based on whether the parameter falls within the exclusive range (0, 1).
• Seed initializes a RandomGenerator that ensures reproducible stochastic behavior across identical runs when combined with non-zero temperature.
• The Sampler.selectSampler factory in Sampler.java orchestrates these components, supporting both CPU and GPU tensor types through polymorphic dispatch.

Frequently Asked Questions

What happens when temperature is set to 0.0 in gpullama3.java?

When temperature equals exactly 0.0f, Sampler.selectSampler returns Sampler.TENSOR_ARGMAX, which performs deterministic greedy selection by choosing the token with the highest logit value. This bypasses all random number generation and ignores the top-p and seed parameters, producing identical outputs for identical inputs.

How does top-p differ from top-k sampling in this implementation?

Top-p (nucleus) sampling dynamically adjusts the candidate pool size based on cumulative probability mass, whereas top-k uses a fixed integer cutoff. In gpullama3.java, ToppSampler constructs a min-heap of token probabilities and truncates once the cumulative probability exceeds the threshold, effectively implementing variable-width nucleus filtering without hard vocabulary limits.

Why does the sampler accept a seed even when using greedy decoding?

The seed parameter is accepted for API consistency but only affects behavior when temperature > 0.0f. The RandomGenerator instantiation occurs inside the stochastic branch of selectSampler, meaning greedy mode (temperature 0.0) never initializes the RNG, making the seed irrelevant for deterministic outputs.

Can sampling parameters be changed during text generation?

The current architecture constructs the Sampler instance once before the generation loop begins in Model.generateTokens. To modify parameters mid-generation, you would need to implement custom state management that calls Sampler.selectSampler with new parameters and replaces the sampler instance in the inference state, though this is not supported by the default CLI interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}