# How Streaming Token Generation Works in the GPULlama3.java Inference Pipeline

> Discover how streaming token generation works in the GPULlama3.java inference pipeline. Learn how tokens are emitted immediately after sampling via an IntConsumer callback for efficient inference.

- Repository: [Beehive lab/gpullama3.java](https://github.com/beehive-lab/gpullama3.java)
- Tags: internals
- Published: 2026-02-26

---

**Streaming token generation in GPULlama3.java emits tokens immediately after sampling through an `IntConsumer` callback passed from the high-level Model API to the GPU inference loop in `InferenceEngine.generateTokensGPULlama`.**

The beehive-lab/gpullama3.java repository provides a GPU-accelerated LLaMA inference implementation using TornadoVM. Understanding how streaming token generation functions within this pipeline is essential for building responsive applications that display output in real-time rather than waiting for complete sequences.

## Enabling Streaming via the Options Flag

Streaming behavior is controlled by the `--stream` CLI flag, which defaults to `true`. In [`Options.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Options.java), the flag is parsed and stored in the `stream` field.

```java
// Options.java – line 45‑46 (source)
out.println("  --stream <boolean>            print tokens during generation; may cause encoding artifacts for non ASCII text, default true");
...
boolean stream = true;                 // default
...
case "--stream" -> stream = Boolean.parseBoolean(nextArg);

```

*Source:* [Options.java (lines 45‑46)](https://github.com/beehive‑lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/Options.java#L45-L46)

## Creating the Token Consumer in the Model Layer

Both the chat loop (`runChatLoop`) and single-prompt execution (`runInstructOnce`) construct an `IntConsumer` callback that handles token display. This consumer checks the `stream` option and filters special tokens before printing.

```java
// Model.java – creation of tokenConsumer (snippet)
IntConsumer tokenConsumer = token -> {
    if (options.stream()) {
        if (tokenizer().shouldDisplayToken(token)) {
            System.out.print(tokenizer().decode(List.of(token)));
        }
    }
};

```

*Source:* [Model.java (lines 124‑132)](https://github.com/beehive‑lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/model/Model.java#L124-L132)

## Dispatching to the GPU Inference Engine

The model layer conditionally passes the consumer to the GPU path only when streaming is enabled. If `options.useTornadovm()` returns `true` and streaming is active, the `tokenConsumer` is supplied to `generateTokensGPU`; otherwise, `null` is passed.

```java
// Model.java – dispatching to GPU or CPU (snippet)
if (options.useTornadovm()) {
    // GPU path using TornadoVM
    responseTokens = generateTokensGPU(state, startPosition,
        conversationTokens.subList(startPosition, conversationTokens.size()),
        stopTokens, options.maxTokens(), sampler,
        options.echo(), options.stream() ? tokenConsumer : null, tornadoVMPlan);
} else {
    // CPU path
    responseTokens = generateTokens(state, startPosition,
        conversationTokens.subList(startPosition, conversationTokens.size()),
        stopTokens, options.maxTokens(), sampler,
        options.echo(), tokenConsumer);
}

```

*Source:* [Model.java (lines 133‑141)](https://github.com/beehive‑lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/model/Model.java#L133-L141)

## The GPU Generation Loop and Real-Time Callbacks

Inside `InferenceEngine.generateTokensGPULlama`, the streaming callback is invoked immediately after token sampling. The GPU forward pass (`InferenceCore.forwardTornadoVM`) executes independently, and once the logits are available, the sampler produces the next token. If `onTokenGenerated` is non-null, it accepts the token instantly.

```java
// InferenceEngine.java – inside the generation loop (excerpt)
nextToken = sampler.sampleToken(logits);

/* Stream the token if a consumer is present */
if (onTokenGenerated != null) {
    onTokenGenerated.accept(nextToken);
}

/* Echo to stderr only when not streaming (i.e. consumer is null) */
if (echo && onTokenGenerated == null) {
    System.err.print(Tokenizer.replaceControlCharacters(
        model.tokenizer().decode(List.of(nextToken))));
}

/* Store the token for the returned list */
generatedTokens.add(nextToken);

```

*Source:* [InferenceEngine.java (lines 32‑37 within the loop)](https://github.com/beehive‑lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/inference/InferenceEngine.java#L32-L37)

This architecture ensures that **streaming token generation** adds no latency to the GPU computation; the callback fires immediately after sampling while the next forward pass proceeds asynchronously.

## Filtering Special Tokens

The consumer delegates to `Tokenizer.shouldDisplayToken(int)` to suppress special control tokens (e.g., BOS, EOS, padding) from appearing in the output stream. Concrete implementations such as `LlamaTokenizer` perform a simple exclusion check.

```java
// LlamaTokenizer.java – shouldDisplayToken implementation
@Override
public boolean shouldDisplayToken(int token) {
    return !isSpecialToken(token);
}

```

*Source:* [LlamaTokenizer.java (lines 35‑38)](https://github.com/beehive‑lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tokenizer/LlamaTokenizer.java#L35-L38)

Other tokenizers (Phi3, Qwen3, etc.) follow the same pattern, ensuring that only meaningful text reaches the user during streaming.

## Practical Example: Implementing Streaming Inference

Below is a minimal program that demonstrates streaming token generation for a LLaMA model on the GPU:

```java
// Example: streaming generation with GPULlama3
import org.beehive.gpullama3.model.llama.Llama;
import org.beehive.gpullama3.Options;
import org.beehive.gpullama3.inference.sampler.ToppSampler;
import java.nio.file.Paths;

public class StreamDemo {
    public static void main(String[] args) {
        // 1️⃣ Build Options – enable streaming (default) and GPU
        Options opts = new Options.Builder()
                .modelPath(Paths.get("models/llama-2-7b-fp16.gguf"))
                .prompt("Write a haiku about sunrise.")
                .stream(true)          // <-- streaming enabled
                .useTornadovm(true)    // <-- GPU path
                .maxTokens(100)
                .build();

        // 2️⃣ Load the model
        Llama model = new Llama(opts);

        // 3️⃣ Choose a sampler (top‑p = 0.9)
        ToppSampler sampler = new ToppSampler(0.9f, 42L);

        // 4️⃣ Run – tokens are printed as they are produced
        model.runInstructOnce(sampler, opts);
    }
}

```

Running the program prints each token **as soon as it is sampled**, e.g.:

```

A golden light
...

```

If you disable streaming (`opts.stream(false)`), the same call will first collect all tokens and then print the whole response at once.

## Summary

- **Streaming token generation** in GPULlama3.java is controlled by the `--stream` flag parsed in [`Options.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Options.java) (default `true`).
- The `Model` class constructs an `IntConsumer` callback that decodes and prints tokens only when streaming is enabled and tokens are displayable.
- For GPU inference, the consumer is passed to `generateTokensGPU` and invoked immediately after sampling inside `InferenceEngine.generateTokensGPULlama`.
- Special tokens are filtered via `Tokenizer.shouldDisplayToken()` to prevent control characters from appearing in the output stream.
- The architecture separates GPU computation from output logic, allowing real-time streaming without blocking the inference loop.

## Frequently Asked Questions

### What controls whether tokens are streamed in GPULlama3.java?

The `Options.stream` boolean field controls streaming behavior. Parsed from the `--stream` CLI argument (defaulting to `true` in [`Options.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Options.java) lines 45‑46), this flag determines whether the `Model` class passes a non-null `IntConsumer` to the generation methods. When `stream` is `false`, the consumer is `null` and tokens accumulate silently until the full response is ready.

### How does the GPU inference engine communicate tokens back to the main thread?

The `InferenceEngine.generateTokensGPULlama` method accepts an `IntConsumer` parameter named `onTokenGenerated`. After each forward pass on the GPU and subsequent token sampling, the engine invokes `onTokenGenerated.accept(nextToken)` immediately. This callback-based approach decouples the GPU computation from output logic, allowing the main thread to print tokens in real-time without blocking the next GPU iteration.

### Why are special tokens filtered during streaming?

Special tokens—such as BOS (beginning of sequence), EOS (end of sequence), and padding tokens—carry structural meaning for the model but should not appear in user-facing output. The `Tokenizer.shouldDisplayToken(int)` method (implemented in [`LlamaTokenizer.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/LlamaTokenizer.java) lines 35‑38) checks `!isSpecialToken(token)` to ensure only meaningful text characters are decoded and printed during the streaming process.

### Can streaming be disabled for non-interactive batch processing?

Yes. Setting `Options.stream` to `false` (via `--stream false` or the builder API) disables streaming for both GPU and CPU paths. In this mode, the `tokenConsumer` is not passed to `generateTokensGPU`, causing the inference loop to accumulate tokens in a list without invoking the print callback. After the loop finishes, the caller decodes the entire token list at once and prints the complete response, which is optimal for batch processing or API responses.