TornadoWeights vs StandardWeights Backends in GPULlama3.java: CPU vs GPU Inference Architecture

TornadoWeights utilizes GPU-resident TornadoTensor objects for TornadoVM acceleration and requires FP32 loading for numerical stability, while StandardWeights employs CPU-resident FloatTensor arrays for pure Java inference without native GPU dependencies.

GPULlama3.java is a high-performance Java inference engine for Large Language Models that abstracts model weights behind a unified interface. The repository provides two distinct backend implementations—TornadoWeights and StandardWeights—that determine whether inference executes on GPU via TornadoVM or remains on the CPU using standard Java arrays. Understanding the architectural differences between these backends is essential for optimizing model performance and memory layout.

Core Architectural Differences

The Common Weights Interface

Both backends implement the Weights interface defined in src/main/java/org/beehive/gpullama3/inference/weights/Weights.java.

public interface Weights {
    GGMLType getWeightType();
}

This contract exposes the underlying GGML type (e.g., F16, Q8_0) while allowing the inference engine to remain agnostic of the concrete tensor implementation.

StandardWeights CPU Backend

Located in src/main/java/org/beehive/gpullama3/inference/weights/standard/StandardWeights.java, this backend uses FloatTensor objects backed by Java float[] arrays.

public abstract class StandardWeights implements Weights {
    public final FloatTensor token_embedding_table;
    public final FloatTensor[] rms_att_weight;
    public final FloatTensor[] wq, wk, wv, wo;
    public final FloatTensor[] rms_ffn_weight;
    public final FloatTensor[] w1, w2, w3;
    public final FloatTensor wcls;
    public final FloatTensor rms_final_weight;
    public final FloatTensor freq_cis_real, freq_cis_imag;
    protected final GGMLType weightType;
}

The LlamaModelLoader.createStandardWeights method populates these fields using loadTensor and loadArrayOfTensors, producing ArrayFloatTensor instances suitable for debugging and CPU-only environments.

TornadoWeights GPU Backend

Defined in src/main/java/org/beehive/gpullama3/inference/weights/tornado/TornadoWeights.java, this backend employs TornadoTensor objects that reside in device memory.

public abstract class TornadoWeights implements Weights {
    public final TornadoTensor tokenEmbeddingTable;
    public final TornadoTensor[] rms_att_weightLayered;
    public final TornadoTensor[] wqLayered, wkLayered, wvLayered, woLayered;
    public final TornadoTensor[] rms_ffn_weightLayered;
    public final TornadoTensor[] w1Layered, w2Layered, w3Layered;
    public final TornadoTensor wclsByteArray;
    public final TornadoTensor rms_final_weight_as_floatArray;
    public final TornadoTensor freq_cis_realFlat, freq_cis_imagFlat;
    protected final GGMLType weightType;
}

The concrete implementation LlamaTornadoWeights holds these GPU-resident buffers, enabling direct execution of transformer kernels on the device without host-to-device transfers.

Memory Layout and Quantization Support

Tensor Representation Differences

StandardWeights stores weights as FloatTensor instances containing contiguous Java float arrays. TornadoWeights utilizes TornadoVM's FloatArray or custom GPU buffers through TornadoTensor, maintaining data in device-compatible memory throughout the inference lifecycle.

GGML Type Constraints

The TornadoWeights backend validates supported formats during initialization in LlamaModelLoader.createTornadoVMWeights, accepting only F16 and Q8_0 GGML types. StandardWeights imposes no such hardware restrictions, handling any format convertible to float arrays, though both backends ultimately expose the weight type via getWeightType().

Loading Mechanisms and Numerical Precision

Standard Loading Path

When useTornadovm is false, LlamaModelLoader invokes createStandardWeights, which deserializes tensors directly into ArrayFloatTensor objects using standard Java I/O operations.

Tornado-Specific Loading Requirements

TornadoWeights requires special handling for numerical stability. According to the Javadoc in TornadoWeights.java, embeddings and RMSNorm weights must be loaded as FP32 using loadTornadoTensorAsFP32, ensuring accurate reduction operations on GPUs that may lack native FP16 arithmetic support. This constraint does not apply to attention or FFN matrices, which can remain in their original quantized formats.

Runtime Selection and Usage Examples

Configuring the Backend

The LlamaModelLoader selects the implementation based on the useTornadovm flag provided to the ModelLoaderFactory:

// Inside LlamaModelLoader (lines 101-112 & 124-136)
if (useTornadovm) {
    return createTornadoVMWeights(...);
} else {
    return createStandardWeights(...);
}

Loading a CPU Model with StandardWeights

Path ggufPath = Paths.get("models/llama-3.1-8B.gguf");
Model model = ModelLoaderFactory
        .builder()
        .ggufFile(ggufPath)
        .useTornadovm(false)          // CPU backend
        .build()
        .loadModel();
// Model contains StandardWeights with FloatTensor fields

Loading a GPU Model with TornadoWeights

Path ggufPath = Paths.get("models/llama-3.1-8B.gguf");
Model model = ModelLoaderFactory
        .builder()
        .ggufFile(ggufPath)
        .useTornadovm(true)          // GPU backend
        .build()
        .loadModel();
// Model contains LlamaTornadoWeights with TornadoTensor fields

Runtime Inspection

Regardless of backend, you can query the weight type:

Weights weights = model.getWeights();
System.out.println("Weight type: " + weights.getWeightType());

Performance Characteristics

StandardWeights executes entirely within the JVM, offering predictable behavior and simplified debugging but limited to CPU-parallel execution. TornadoWeights eliminates costly host-to-device memory copies by keeping tensors in GPU memory, enabling parallel execution across thousands of threads through TornadoVM's OpenCL or CUDA backends. However, this requires compatible GPU hardware and restricts quantization support to F16 and Q8_0 formats.

Summary

• StandardWeights implements the Weights interface using FloatTensor objects backed by Java float arrays for CPU-only inference.
• TornadoWeights provides GPU acceleration through TornadoTensor objects that reside in device memory managed by TornadoVM.
• The useTornadovm flag in LlamaModelLoader determines which backend instantiates without changing the high-level model API.
• TornadoWeights requires FP32 loading for embeddings and RMSNorm weights to ensure numerical stability on GPUs.
• Only F16 and Q8_0 GGML types are supported by TornadoWeights, while StandardWeights handles any float-convertible format.
• Both backends share identical logical layouts (embedding tables, attention matrices, FFN weights, RoPE frequencies) but differ in physical memory representation.

Frequently Asked Questions

What is the primary advantage of TornadoWeights over StandardWeights?

TornadoWeights enables GPU-accelerated inference by maintaining weights in device memory through TornadoTensor objects, eliminating host-to-device transfer overhead and leveraging thousands of parallel threads. StandardWeights remains confined to CPU execution using standard Java arrays.

Why must RMSNorm and embedding weights be loaded as FP32 in TornadoWeights?

According to the source code in TornadoWeights.java, these specific layers require FP32 precision to guarantee numerical stability during reduction operations, as some GPU architectures lack native support for accurate FP16 arithmetic in normalization contexts. This constraint is enforced through the loadTornadoTensorAsFP32 method during model loading.

Which GGML quantization formats are supported by each backend?

TornadoWeights explicitly validates and supports only F16 and Q8_0 formats in LlamaModelLoader.createTornadoVMWeights. StandardWeights imposes no format restrictions, handling any GGML type that can be deserialized into Java float arrays, though performance characteristics vary by precision.

Can I switch between backends without modifying the model files?

Yes. The GGUF model files remain unchanged; you simply toggle the useTornadovm boolean flag when building the ModelLoaderFactory. The LlamaModelLoader dynamically instantiates either StandardWeights or TornadoWeights (specifically LlamaTornadoWeights) based on this flag, while exposing the same Model interface to the inference pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}