How TornadoVM Acceleration Works with the Llama Architecture in GPULlama3.java

TornadoVM acceleration in GPULlama3.java offloads the compute-intensive transformer layers to GPU task graphs while keeping token embedding lookup and sampling on the CPU, delivering high-performance LLaMA inference through a three-stage pipeline.

GPULlama3.java leverages TornadoVM to execute the LLaMA transformer architecture on heterogeneous hardware. According to the beehive-lab/gpullama3.java source code, the implementation splits inference between CPU preprocessing and GPU-accelerated layer computation, using quantization-aware task graphs to maximize throughput.

The Three-Stage TornadoVM Acceleration Pipeline

The acceleration strategy follows a distinct pipeline that separates initialization, data preparation, and computation. This architecture minimizes host-device transfer overhead by keeping model weights resident on the GPU across token generation steps.

Stage 1: Plan Creation and Warm-Up

Before inference begins, TornadoVMMasterPlan.initializeTornadoVMPlan constructs a TornadoExecutionPlan and performs JIT compilation of GPU kernels. This method, located in src/main/java/org/beehive/gpullama3/tornadovm/TornadoVMMasterPlan.java, also transfers read-only model weights to device memory during the warm-up phase.

TornadoVMMasterPlan tornadoPlan = 
    TornadoVMMasterPlan.initializeTornadoVMPlan(state, model);
model.setTornadoVMPlan(tornadoPlan);

By compiling kernels upfront and persisting weights on the GPU, the system avoids per-token compilation overhead and reduces memory transfer latency during the generation loop.

Stage 2: CPU-Based Token Embedding Lookup

The InferenceCore.forwardTornadoVM method handles the embedding lookup on the CPU. This step reads token embeddings from the model's weight tables—whether stored as FP16 or Q8_0—into a host buffer called state.embeddingX.

The CPU remains responsible for this operation because the embedding table resides in regular host memory. The source code in src/main/java/org/beehive/gpullama3/inference/InferenceCore.java (lines 704-724) implements this lookup before handing control to the GPU for transformer computation.

Stage 3: Layer-Wise GPU Execution

The core acceleration occurs in TornadoVMMasterPlan.tornadoVMForwardExecuteLayered, which executes three distinct groups of TornadoVM task graphs:

• Pre-processing graph: Initializes position holders and temporary buffers
• Per-layer transformer graphs: Each graph executes attention and feed-forward network operations for a single layer, iterating over config.numberOfLayers()
• Final logits graph: Projects the last hidden state to the output vocabulary space

After the final graph completes, state.wrapLogits (a FloatArray) returns to the caller. The CPU then handles token sampling from these logits before initiating the next forward pass.

Quantization-Aware Task Graph Generation

The system adapts to different weight formats through the QuantizationPlannerFactory class in src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/base/QuantizationPlannerFactory.java. This factory selects the appropriate GenericLayerPlanner implementation based on the model's quantization type (FP16, Q8_0, etc.) and model family (LLaMA-3, Mistral, etc.).

The planner ensures TornadoVM generates specialized kernels for each quantization scheme, optimizing memory bandwidth and compute utilization for the specific weight format without changing the high-level inference API.

Implementation Examples

Initializing the TornadoVM Execution Plan

Create and store the execution plan immediately after model loading to enable GPU acceleration:

// After model and state instantiation
TornadoVMMasterPlan tornadoPlan = 
    TornadoVMMasterPlan.initializeTornadoVMPlan(state, model);
model.setTornadoVMPlan(tornadoPlan);   // Store for subsequent calls

Running GPU-Accelerated Token Generation

Use the high-level InferenceEngine API to generate tokens with TornadoVM acceleration:

List<Integer> generated = InferenceEngine.generateTokensGPULlama(
        model,                     // Llama or Mistral model instance
        state,                     // Holds KV cache and buffers
        0,                         // Start position
        promptTokens,              // List<Integer> of input tokens
        Set.of(2),                 // Stop token IDs
        512,                       // Maximum tokens to generate
        sampler,                   // Sampler implementation
        true,                      // Echo output to stderr
        null,                      // Optional per-token callback
        model.tornadoVMPlan());    // Pre-built TornadoVM plan

Low-Level Forward Pass Call

For custom inference loops, call the forward method directly:

FloatArray logits = InferenceCore.forwardTornadoVM(
        model, state, currentToken, position, tornadoPlan);

This method combines the CPU embedding lookup with the layered GPU execution before returning the raw logits for sampling.

Key Source Files and Architecture Components

• src/main/java/org/beehive/gpullama3/tornadovm/TornadoVMMasterPlan.java: Creates the TornadoExecutionPlan, manages kernel warm-up, and orchestrates the layered forward pass through tornadoVMForwardExecuteLayered
• src/main/java/org/beehive/gpullama3/inference/InferenceCore.java: Implements forwardTornadoVM, handling CPU-side embedding extraction and GPU execution triggering
• src/main/java/org/beehive/gpullama3/inference/InferenceEngine.java: Provides the public API generateTokensGPULlama for GPU-based token generation
• src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/base/QuantizationPlannerFactory.java: Factory thatinstantiates quantization-specific task graph planners based on weight type and model family
• src/main/java/org/beehive/gpullama3/model/llama/Llama.java: Model class that delegates GPU inference operations to the InferenceEngine
• src/main/java/org/beehive/gpullama3/model/ModelType.java: Enum defining supported architectures including LLAMA_3 and MISTRAL that work with the TornadoVM acceleration stack

Summary

• TornadoVM acceleration in GPULlama3.java executes transformer layers on the GPU while keeping embedding lookup and token sampling on the CPU.
• The three-stage pipeline consists of plan initialization with JIT compilation (initializeTornadoVMPlan), CPU embedding extraction (forwardTornadoVM), and layer-wise GPU task graph execution (tornadoVMForwardExecuteLayered).
• Quantization-aware planning via QuantizationPlannerFactory automatically selects optimized kernels for FP16, Q8_0, and other formats.
• Model weights remain persistently resident on the GPU across generation steps to minimize memory transfers.
• The high-level API InferenceEngine.generateTokensGPULlama abstracts the complexity while providing full GPU acceleration for LLaMA and compatible architectures.

Frequently Asked Questions

What is TornadoVM and why is it used in GPULlama3.java?

TornadoVM is a parallel programming framework that enables Java applications to offload compute-intensive workloads to GPUs and other accelerators. GPULlama3.java uses TornadoVM to translate transformer layer operations into GPU kernels, providing significant speedup over CPU-only inference while maintaining the safety and productivity of the Java ecosystem.

Which parts of the LLaMA inference run on the GPU versus the CPU?

The GPU handles all transformer layer computations including attention mechanisms and feed-forward networks through TornadoVM task graphs. The CPU manages token embedding lookups from weight tables, token sampling from output logits, and overall generation loop orchestration. This separation optimizes the use of each processor type's strengths.

How does GPULlama3 handle different quantization formats with TornadoVM?

The QuantizationPlannerFactory inspects the model's weight type and model family during initialization. Based on this metadata, it instantiates the appropriate layer planner that generates TornadoVM task graphs specific to the quantization scheme—whether FP16, Q8_0, or other supported formats—ensuring kernels are optimized for the specific memory layout and computation patterns.

What is the role of the TornadoExecutionPlan in the acceleration pipeline?

The TornadoExecutionPlan serves as the compiled execution graph that TornadoVM uses to dispatch work to the GPU. Created once during initializeTornadoVMPlan, it encapsulates the JIT-compiled kernels and device memory allocations for model weights. This plan persists across multiple forward passes, allowing the system to reuse GPU resources without recompilation or repeated memory transfers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}