Why JDK 21 Is Required for GPULlama3.java: Java Vector API Features Explained

GPULlama3.java requires JDK 21 or newer because it depends on the incubating Java Vector API (jdk.incubator.vector) to perform SIMD-accelerated tensor operations, which is only available in JDK 21+ with the --add-modules flag enabled.

The beehive-lab/gpullama3.java repository implements a high-performance, GPU-accelerated inference engine for Llama 3 models. To achieve optimal CPU-side performance for tensor operations, the project leverages the Java Vector API, making JDK 21 a strict requirement for compilation and runtime.

Why JDK 21 Is Required for GPULlama3.java

The Incubating Java Vector API Module

The core dependency is the jdk.incubator.vector module, which remains in incubation status. JDK 21 (JEP 426) introduced the first incubating version of this API, while JDK 25 continues to provide it (JEP 508). The module is not available in JDK 20 or earlier, making compilation impossible on older versions because imports such as jdk.incubator.vector.FloatVector cannot be resolved.

Required JVM Flags and Build Configuration

The project's pom.xml configures Maven profiles for JDK 21 and JDK 25 that automatically inject the necessary compiler and runtime flags:

<arg>--add-modules</arg>
<arg>jdk.incubator.vector</arg>

Without these flags, the JVM cannot resolve Vector API classes, resulting in ClassNotFoundException at runtime. The CLI entry point in LlamaTornadoCli.java also documents the requirement to run with --enable-preview --add-modules jdk.incubator.vector.

Java Vector API Features Used in GPULlama3.java

Vector Species and Runtime Shape Selection

In src/main/java/org/beehive/gpullama3/tensor/standard/FloatTensor.java, the code dynamically selects the optimal vector size based on hardware capabilities:

static final VectorSpecies<Float> F_SPECIES;
static {
    if (USE_VECTOR_API) {
        // Pick the best-size float vector for the current CPU/GPU
        F_SPECIES = VectorShape.forBitSize(VECTOR_BIT_SIZE).withLanes(float.class);
    } else {
        F_SPECIES = null;
    }
}

This uses VectorShape.forBitSize() and withLanes() to create a VectorSpecies<Float> that defines the lane count and shape at runtime.

Typed SIMD Vectors (FloatVector and ByteVector)

The implementation uses FloatVector for single-precision tensor math and ByteVector for quantized weight handling. In Q8_0FloatTensor.java, bytes are loaded directly from memory into vectors:

var wBytes = ByteVector.fromMemorySegment(ByteVector.SPECIES_256,
                                          thiz.memorySegment,
                                          blockOffset + Float16.BYTES,
                                          ByteOrder.LITTLE_ENDIAN);

Memory Operations and Vector Broadcasting

The code leverages FloatVector.broadcast() to replicate scalar values (such as weight scales) across all vector lanes for SIMD multiplication:

var sum0 = that.getFloatVector(F_SPECIES,
                               thatOffset + j + 0 * F_SPECIES.length())
                .mul(wBytes.castShape(F_SPECIES, 0));
...
val = sum0.add(sum1).add(sum2).add(sum3).fma(wScale, val);

Arithmetic, FMA, and Reduction Operations

Critical for performance, the implementation uses fused multiply-add (fma) and reduction operations. The VectorOperators.FMA method performs high-throughput dot-product accumulation, while reduceLanes collapses vectors to scalars:

result += val.reduceLanes(VectorOperators.ADD);

Summary

• JDK 21 Requirement: GPULlama3.java requires JDK 21 or newer because it depends on the incubating jdk.incubator.vector module (Java Vector API), which first appeared in JDK 21 (JEP 426).
• Module Access: Compilation and runtime require the --add-modules jdk.incubator.vector flag, configured automatically in the Maven profiles for JDK 21 and JDK 25.
• Core Vector Features: The implementation uses VectorSpecies, FloatVector, ByteVector, VectorShape, memory segment loading, broadcasting, fma operations, and lane reduction to achieve SIMD-accelerated tensor math.
• Performance Critical Files: Key implementations reside in FloatTensor.java (species selection) and Q8_0FloatTensor.java (quantized SIMD dot products).

Frequently Asked Questions

Can I run GPULlama3.java on JDK 17 or JDK 20?

No. The project requires the jdk.incubator.vector module, which was first introduced as an incubating feature in JDK 21 (JEP 426). Attempting to compile on JDK 20 or earlier will result in unresolved import errors for jdk.incubator.vector.FloatVector and related classes.

What happens if I don't enable the jdk.incubator.vector module?

Without the --add-modules jdk.incubator.vector JVM flag, the runtime will throw ClassNotFoundException or IllegalAccessError when attempting to load Vector API classes. The Maven profiles in pom.xml automatically add this flag for JDK 21 and JDK 25 builds to prevent this issue.

Does GPULlama3.java work with JDK 25?

Yes. JDK 25 continues to provide the Java Vector API as an incubating module (JEP 508, the 10th incubator). The project's Maven configuration includes a specific profile for JDK 25 that applies the necessary --add-modules flag, ensuring compatibility with newer feature releases.

Which specific Vector API classes provide the SIMD acceleration?

The SIMD acceleration is primarily driven by FloatVector and ByteVector for data parallelism, VectorSpecies for runtime vector shape selection, and VectorOperators for arithmetic operations like FMA (fused multiply-add) and ADD for reductions. These classes work together in Q8_0FloatTensor.java to perform quantized matrix multiplication using hardware SIMD instructions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}