# TornadoWeights vs StandardWeights Backends in GPULlama3.java: CPU vs GPU Inference Architecture

> Explore TornadoWeights vs StandardWeights in GPULlama3.java. Understand CPU vs GPU inference architectures and choose the best backend for your needs. Learn about performance differences.

- Repository: [Beehive lab/gpullama3.java](https://github.com/beehive-lab/gpullama3.java)
- Tags: architecture
- Published: 2026-02-26

---

**TornadoWeights utilizes GPU-resident TornadoTensor objects for TornadoVM acceleration and requires FP32 loading for numerical stability, while StandardWeights employs CPU-resident FloatTensor arrays for pure Java inference without native GPU dependencies.**

GPULlama3.java is a high-performance Java inference engine for Large Language Models that abstracts model weights behind a unified interface. The repository provides two distinct backend implementations—**TornadoWeights** and **StandardWeights**—that determine whether inference executes on GPU via TornadoVM or remains on the CPU using standard Java arrays. Understanding the architectural differences between these backends is essential for optimizing model performance and memory layout.

## Core Architectural Differences

### The Common Weights Interface

Both backends implement the `Weights` interface defined in [`src/main/java/org/beehive/gpullama3/inference/weights/Weights.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/inference/weights/Weights.java).

```java
public interface Weights {
    GGMLType getWeightType();
}

```

This contract exposes the underlying GGML type (e.g., `F16`, `Q8_0`) while allowing the inference engine to remain agnostic of the concrete tensor implementation.

### StandardWeights CPU Backend

Located in [`src/main/java/org/beehive/gpullama3/inference/weights/standard/StandardWeights.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/inference/weights/standard/StandardWeights.java), this backend uses `FloatTensor` objects backed by Java `float[]` arrays.

```java
public abstract class StandardWeights implements Weights {
    public final FloatTensor token_embedding_table;
    public final FloatTensor[] rms_att_weight;
    public final FloatTensor[] wq, wk, wv, wo;
    public final FloatTensor[] rms_ffn_weight;
    public final FloatTensor[] w1, w2, w3;
    public final FloatTensor wcls;
    public final FloatTensor rms_final_weight;
    public final FloatTensor freq_cis_real, freq_cis_imag;
    protected final GGMLType weightType;
}

```

The `LlamaModelLoader.createStandardWeights` method populates these fields using `loadTensor` and `loadArrayOfTensors`, producing `ArrayFloatTensor` instances suitable for debugging and CPU-only environments.

### TornadoWeights GPU Backend

Defined in [`src/main/java/org/beehive/gpullama3/inference/weights/tornado/TornadoWeights.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/inference/weights/tornado/TornadoWeights.java), this backend employs `TornadoTensor` objects that reside in device memory.

```java
public abstract class TornadoWeights implements Weights {
    public final TornadoTensor tokenEmbeddingTable;
    public final TornadoTensor[] rms_att_weightLayered;
    public final TornadoTensor[] wqLayered, wkLayered, wvLayered, woLayered;
    public final TornadoTensor[] rms_ffn_weightLayered;
    public final TornadoTensor[] w1Layered, w2Layered, w3Layered;
    public final TornadoTensor wclsByteArray;
    public final TornadoTensor rms_final_weight_as_floatArray;
    public final TornadoTensor freq_cis_realFlat, freq_cis_imagFlat;
    protected final GGMLType weightType;
}

```

The concrete implementation `LlamaTornadoWeights` holds these GPU-resident buffers, enabling direct execution of transformer kernels on the device without host-to-device transfers.

## Memory Layout and Quantization Support

### Tensor Representation Differences

**StandardWeights** stores weights as `FloatTensor` instances containing contiguous Java float arrays. **TornadoWeights** utilizes TornadoVM's `FloatArray` or custom GPU buffers through `TornadoTensor`, maintaining data in device-compatible memory throughout the inference lifecycle.

### GGML Type Constraints

The TornadoWeights backend validates supported formats during initialization in `LlamaModelLoader.createTornadoVMWeights`, accepting only **F16** and **Q8_0** GGML types. StandardWeights imposes no such hardware restrictions, handling any format convertible to float arrays, though both backends ultimately expose the weight type via `getWeightType()`.

## Loading Mechanisms and Numerical Precision

### Standard Loading Path

When `useTornadovm` is false, `LlamaModelLoader` invokes `createStandardWeights`, which deserializes tensors directly into `ArrayFloatTensor` objects using standard Java I/O operations.

### Tornado-Specific Loading Requirements

TornadoWeights requires special handling for numerical stability. According to the Javadoc in [`TornadoWeights.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TornadoWeights.java), embeddings and RMSNorm weights must be loaded as FP32 using `loadTornadoTensorAsFP32`, ensuring accurate reduction operations on GPUs that may lack native FP16 arithmetic support. This constraint does not apply to attention or FFN matrices, which can remain in their original quantized formats.

## Runtime Selection and Usage Examples

### Configuring the Backend

The `LlamaModelLoader` selects the implementation based on the `useTornadovm` flag provided to the `ModelLoaderFactory`:

```java
// Inside LlamaModelLoader (lines 101-112 & 124-136)
if (useTornadovm) {
    return createTornadoVMWeights(...);
} else {
    return createStandardWeights(...);
}

```

### Loading a CPU Model with StandardWeights

```java
Path ggufPath = Paths.get("models/llama-3.1-8B.gguf");
Model model = ModelLoaderFactory
        .builder()
        .ggufFile(ggufPath)
        .useTornadovm(false)          // CPU backend
        .build()
        .loadModel();
// Model contains StandardWeights with FloatTensor fields

```

### Loading a GPU Model with TornadoWeights

```java
Path ggufPath = Paths.get("models/llama-3.1-8B.gguf");
Model model = ModelLoaderFactory
        .builder()
        .ggufFile(ggufPath)
        .useTornadovm(true)          // GPU backend
        .build()
        .loadModel();
// Model contains LlamaTornadoWeights with TornadoTensor fields

```

### Runtime Inspection

Regardless of backend, you can query the weight type:

```java
Weights weights = model.getWeights();
System.out.println("Weight type: " + weights.getWeightType());

```

## Performance Characteristics

**StandardWeights** executes entirely within the JVM, offering predictable behavior and simplified debugging but limited to CPU-parallel execution. **TornadoWeights** eliminates costly host-to-device memory copies by keeping tensors in GPU memory, enabling parallel execution across thousands of threads through TornadoVM's OpenCL or CUDA backends. However, this requires compatible GPU hardware and restricts quantization support to F16 and Q8_0 formats.

## Summary

- **StandardWeights** implements the `Weights` interface using `FloatTensor` objects backed by Java float arrays for CPU-only inference.
- **TornadoWeights** provides GPU acceleration through `TornadoTensor` objects that reside in device memory managed by TornadoVM.
- The `useTornadovm` flag in `LlamaModelLoader` determines which backend instantiates without changing the high-level model API.
- TornadoWeights requires FP32 loading for embeddings and RMSNorm weights to ensure numerical stability on GPUs.
- Only F16 and Q8_0 GGML types are supported by TornadoWeights, while StandardWeights handles any float-convertible format.
- Both backends share identical logical layouts (embedding tables, attention matrices, FFN weights, RoPE frequencies) but differ in physical memory representation.

## Frequently Asked Questions

### What is the primary advantage of TornadoWeights over StandardWeights?

TornadoWeights enables GPU-accelerated inference by maintaining weights in device memory through `TornadoTensor` objects, eliminating host-to-device transfer overhead and leveraging thousands of parallel threads. StandardWeights remains confined to CPU execution using standard Java arrays.

### Why must RMSNorm and embedding weights be loaded as FP32 in TornadoWeights?

According to the source code in [`TornadoWeights.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TornadoWeights.java), these specific layers require FP32 precision to guarantee numerical stability during reduction operations, as some GPU architectures lack native support for accurate FP16 arithmetic in normalization contexts. This constraint is enforced through the `loadTornadoTensorAsFP32` method during model loading.

### Which GGML quantization formats are supported by each backend?

TornadoWeights explicitly validates and supports only **F16** and **Q8_0** formats in `LlamaModelLoader.createTornadoVMWeights`. StandardWeights imposes no format restrictions, handling any GGML type that can be deserialized into Java float arrays, though performance characteristics vary by precision.

### Can I switch between backends without modifying the model files?

Yes. The GGUF model files remain unchanged; you simply toggle the `useTornadovm` boolean flag when building the `ModelLoaderFactory`. The `LlamaModelLoader` dynamically instantiates either `StandardWeights` or `TornadoWeights` (specifically `LlamaTornadoWeights`) based on this flag, while exposing the same `Model` interface to the inference pipeline.