# How TornadoVM Acceleration Works with the Llama Architecture in GPULlama3.java > Discover how TornadoVM acceleration optimizes Llama architecture in GPULlama3.java. Achieve high-performance LLaMA inference by offloading transformer layers to GPU task graphs. - Repository: [Beehive lab/gpullama3.java](https://github.com/beehive-lab/gpullama3.java) - Tags: deep-dive - Published: 2026-02-26 --- **TornadoVM acceleration in GPULlama3.java offloads the compute-intensive transformer layers to GPU task graphs while keeping token embedding lookup and sampling on the CPU, delivering high-performance LLaMA inference through a three-stage pipeline.** GPULlama3.java leverages TornadoVM to execute the LLaMA transformer architecture on heterogeneous hardware. According to the [`beehive-lab/gpullama3.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/beehive-lab/gpullama3.java) source code, the implementation splits inference between CPU preprocessing and GPU-accelerated layer computation, using quantization-aware task graphs to maximize throughput. ## The Three-Stage TornadoVM Acceleration Pipeline The acceleration strategy follows a distinct pipeline that separates initialization, data preparation, and computation. This architecture minimizes host-device transfer overhead by keeping model weights resident on the GPU across token generation steps. ### Stage 1: Plan Creation and Warm-Up Before inference begins, `TornadoVMMasterPlan.initializeTornadoVMPlan` constructs a `TornadoExecutionPlan` and performs JIT compilation of GPU kernels. This method, located in [`src/main/java/org/beehive/gpullama3/tornadovm/TornadoVMMasterPlan.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/TornadoVMMasterPlan.java), also transfers read-only model weights to device memory during the warm-up phase. ```java TornadoVMMasterPlan tornadoPlan = TornadoVMMasterPlan.initializeTornadoVMPlan(state, model); model.setTornadoVMPlan(tornadoPlan); ``` By compiling kernels upfront and persisting weights on the GPU, the system avoids per-token compilation overhead and reduces memory transfer latency during the generation loop. ### Stage 2: CPU-Based Token Embedding Lookup The `InferenceCore.forwardTornadoVM` method handles the embedding lookup on the CPU. This step reads token embeddings from the model's weight tables—whether stored as FP16 or Q8_0—into a host buffer called `state.embeddingX`. The CPU remains responsible for this operation because the embedding table resides in regular host memory. The source code in [`src/main/java/org/beehive/gpullama3/inference/InferenceCore.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/inference/InferenceCore.java) (lines 704-724) implements this lookup before handing control to the GPU for transformer computation. ### Stage 3: Layer-Wise GPU Execution The core acceleration occurs in `TornadoVMMasterPlan.tornadoVMForwardExecuteLayered`, which executes three distinct groups of TornadoVM task graphs: - **Pre-processing graph**: Initializes position holders and temporary buffers - **Per-layer transformer graphs**: Each graph executes attention and feed-forward network operations for a single layer, iterating over `config.numberOfLayers()` - **Final logits graph**: Projects the last hidden state to the output vocabulary space After the final graph completes, `state.wrapLogits` (a `FloatArray`) returns to the caller. The CPU then handles token sampling from these logits before initiating the next forward pass. ## Quantization-Aware Task Graph Generation The system adapts to different weight formats through the `QuantizationPlannerFactory` class in [`src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/base/QuantizationPlannerFactory.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/base/QuantizationPlannerFactory.java). This factory selects the appropriate `GenericLayerPlanner` implementation based on the model's quantization type (FP16, Q8_0, etc.) and model family (LLaMA-3, Mistral, etc.). The planner ensures TornadoVM generates specialized kernels for each quantization scheme, optimizing memory bandwidth and compute utilization for the specific weight format without changing the high-level inference API. ## Implementation Examples ### Initializing the TornadoVM Execution Plan Create and store the execution plan immediately after model loading to enable GPU acceleration: ```java // After model and state instantiation TornadoVMMasterPlan tornadoPlan = TornadoVMMasterPlan.initializeTornadoVMPlan(state, model); model.setTornadoVMPlan(tornadoPlan); // Store for subsequent calls ``` ### Running GPU-Accelerated Token Generation Use the high-level `InferenceEngine` API to generate tokens with TornadoVM acceleration: ```java List generated = InferenceEngine.generateTokensGPULlama( model, // Llama or Mistral model instance state, // Holds KV cache and buffers 0, // Start position promptTokens, // List of input tokens Set.of(2), // Stop token IDs 512, // Maximum tokens to generate sampler, // Sampler implementation true, // Echo output to stderr null, // Optional per-token callback model.tornadoVMPlan()); // Pre-built TornadoVM plan ``` ### Low-Level Forward Pass Call For custom inference loops, call the forward method directly: ```java FloatArray logits = InferenceCore.forwardTornadoVM( model, state, currentToken, position, tornadoPlan); ``` This method combines the CPU embedding lookup with the layered GPU execution before returning the raw logits for sampling. ## Key Source Files and Architecture Components - **[`src/main/java/org/beehive/gpullama3/tornadovm/TornadoVMMasterPlan.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/TornadoVMMasterPlan.java)**: Creates the `TornadoExecutionPlan`, manages kernel warm-up, and orchestrates the layered forward pass through `tornadoVMForwardExecuteLayered` - **[`src/main/java/org/beehive/gpullama3/inference/InferenceCore.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/inference/InferenceCore.java)**: Implements `forwardTornadoVM`, handling CPU-side embedding extraction and GPU execution triggering - **[`src/main/java/org/beehive/gpullama3/inference/InferenceEngine.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/inference/InferenceEngine.java)**: Provides the public API `generateTokensGPULlama` for GPU-based token generation - **[`src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/base/QuantizationPlannerFactory.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/base/QuantizationPlannerFactory.java)**: Factory thatinstantiates quantization-specific task graph planners based on weight type and model family - **[`src/main/java/org/beehive/gpullama3/model/llama/Llama.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/model/llama/Llama.java)**: Model class that delegates GPU inference operations to the `InferenceEngine` - **[`src/main/java/org/beehive/gpullama3/model/ModelType.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/model/ModelType.java)**: Enum defining supported architectures including LLAMA_3 and MISTRAL that work with the TornadoVM acceleration stack ## Summary - **TornadoVM acceleration** in GPULlama3.java executes transformer layers on the GPU while keeping embedding lookup and token sampling on the CPU. - The **three-stage pipeline** consists of plan initialization with JIT compilation (`initializeTornadoVMPlan`), CPU embedding extraction (`forwardTornadoVM`), and layer-wise GPU task graph execution (`tornadoVMForwardExecuteLayered`). - **Quantization-aware planning** via `QuantizationPlannerFactory` automatically selects optimized kernels for FP16, Q8_0, and other formats. - Model weights remain **persistently resident on the GPU** across generation steps to minimize memory transfers. - The high-level API `InferenceEngine.generateTokensGPULlama` abstracts the complexity while providing full GPU acceleration for LLaMA and compatible architectures. ## Frequently Asked Questions ### What is TornadoVM and why is it used in GPULlama3.java? TornadoVM is a parallel programming framework that enables Java applications to offload compute-intensive workloads to GPUs and other accelerators. GPULlama3.java uses TornadoVM to translate transformer layer operations into GPU kernels, providing significant speedup over CPU-only inference while maintaining the safety and productivity of the Java ecosystem. ### Which parts of the LLaMA inference run on the GPU versus the CPU? The GPU handles all transformer layer computations including attention mechanisms and feed-forward networks through TornadoVM task graphs. The CPU manages token embedding lookups from weight tables, token sampling from output logits, and overall generation loop orchestration. This separation optimizes the use of each processor type's strengths. ### How does GPULlama3 handle different quantization formats with TornadoVM? The `QuantizationPlannerFactory` inspects the model's weight type and model family during initialization. Based on this metadata, it instantiates the appropriate layer planner that generates TornadoVM task graphs specific to the quantization scheme—whether FP16, Q8_0, or other supported formats—ensuring kernels are optimized for the specific memory layout and computation patterns. ### What is the role of the TornadoExecutionPlan in the acceleration pipeline? The `TornadoExecutionPlan` serves as the compiled execution graph that TornadoVM uses to dispatch work to the GPU. Created once during `initializeTornadoVMPlan`, it encapsulates the JIT-compiled kernels and device memory allocations for model weights. This plan persists across multiple forward passes, allowing the system to reuse GPU resources without recompilation or repeated memory transfers.