GPULlama3.java Quantization Formats: Q8_0, Q4_0, and FP16 GPU Memory Guide

GPULlama3.java fully supports FP16 and Q8_0 quantization formats, with Q4_0 declared but unimplemented, where Q8_0 reduces GPU memory usage by approximately 50% compared to the FP16 baseline.

GPULlama3.java is a GPU-accelerated Java inference engine for Llama 3 models that leverages TornadoVM for parallel execution on heterogeneous hardware. The framework implements multiple quantization formats to optimize memory consumption, enabling deployment of larger models on constrained GPU hardware. Understanding the differences between FP16, Q8_0, and the planned Q4_0 formats is critical for configuring efficient inference pipelines.

Supported Quantization Formats in GPULlama3.java

The GGMLType enum in src/main/java/org/beehive/gpullama3/tensor/GGMLType.java defines the available precision formats, though the QuantizationPlannerFactory class determines actual runtime support.

FP16 (16-bit Floating Point)

FP16 serves as the baseline precision format, storing each weight as a 2-byte half-precision float. The QuantizationPlannerFactory.createFP16Planner() method instantiates full-precision layer planners for models using this format. While offering the highest numerical fidelity, FP16 consumes the most GPU memory, requiring approximately 2 bytes per parameter with no compression overhead.

Q8_0 (8-bit Integer with Block Scaling)

Q8_0 compresses weights to 1-byte integers accompanied by per-block scaling factors. According to the implementation in QuantizationPlannerFactory.createQ8_0Planner(), this format stores 2 bytes of metadata per 256-element block (scale and zero-point), yielding an effective size of roughly 1.007 bytes per weight. This achieves approximately 50% memory reduction compared to FP16. The concrete GPU kernels for Q8_0 matrix operations reside in src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/.

Q4_0 (4-bit Integer - Future Support)

While GGMLType.Q4_0 is declared in the enum, the format remains unimplemented in the current release. Calling QuantizationPlannerFactory.createQ4_0Planner() throws an UnsupportedOperationException, as explicitly noted in the repository README. When eventually supported, Q4_0 would store weights as 4-bit integers with the same 2-byte block metadata structure, theoretically achieving approximately 75% memory reduction (to ~25% of FP16 size). A skeleton tensor implementation exists in src/main/java/org/beehive/gpullama3/tensor/standard/Q4_0FloatTensor.java but lacks planner integration and GPU kernel support.

How Quantization Formats Impact GPU Memory Usage

The relationship between quantization precision and memory consumption follows predictable compression ratios based on bit-width and per-block overhead:

Format	Bytes per Weight	Block Overhead	Total Memory vs FP16
FP16	2.0 B	None	100% (baseline)
Q8_0	1.0 B	~0.007 B per weight	~50%
Q4_0	0.5 B	~0.008 B per weight	~25% (planned)

For practical deployment scenarios, a 1B-parameter model requires approximately 2GB in FP16, 1GB in Q8_0, and would require only 0.5GB once Q4_0 support arrives. Larger models (3B-8B parameters) often necessitate the --gpu-memory CLI flag to increase allocation beyond the default 7GB limit, though lower quantization formats proportionally reduce these requirements.

Configuring Quantization Formats in GPULlama3.java

The framework automatically detects quantization formats from GGUF model filenames during initialization. Users select formats by loading appropriately quantized model files.

Run a model with FP16 precision:

./llama-tornado --gpu --model beehive-llama-3.2-1b-instruct-fp16.gguf \
    --prompt "Explain neural networks"

Execute with Q8_0 quantization for reduced memory footprint:

./llama-tornado --gpu --model beehive-llama-3.2-1b-instruct-q8_0.gguf \
    --prompt "Explain neural networks"

For models exceeding default GPU memory limits, increase allocation regardless of quantization format:

./llama-tornado --gpu --model beehive-llama-3.2-3b-instruct-q8_0.gguf \
    --gpu-memory 15GB --prompt "Explain neural networks"

Attempting to use Q4_0 models currently results in runtime exceptions until QuantizationPlannerFactory implements the corresponding planner method.

Summary

• FP16 provides full precision but requires 2 bytes per weight, serving as the baseline for GPU memory calculations.
• Q8_0 is fully implemented and production-ready, reducing memory usage by approximately 50% through 8-bit quantization with per-block scaling factors.
• Q4_0 is declared in GGMLType.java but unimplemented; createQ4_0Planner() throws UnsupportedOperationException as noted in the README.
• Key implementation files include QuantizationPlannerFactory.java for planner creation and format-specific layer classes in tornadovm/layers/type/q8_0/.
• Use the --gpu-memory flag to accommodate larger models when quantization alone is insufficient for your hardware constraints.

Frequently Asked Questions

Which quantization formats can I use today in GPULlama3.java?

You can currently use FP16 and Q8_0 quantization formats. FP16 offers full precision while Q8_0 provides a balanced 50% memory reduction. The Q4_0 format appears in the codebase enum but remains unimplemented, with QuantizationPlannerFactory.createQ4_0Planner() explicitly throwing an UnsupportedOperationException when invoked.

How much GPU memory does Q8_0 save compared to FP16?

Q8_0 reduces memory consumption by approximately 50% compared to FP16. While FP16 requires 2 bytes per weight, Q8_0 uses 1 byte per weight plus negligible block metadata overhead (roughly 1.007 bytes effective per weight), allowing you to run models twice as large on the same GPU hardware.

Why does attempting to use Q4_0 quantization fail?

Q4_0 support is listed as a future feature in the GPULlama3.java README. Although GGMLType.Q4_0 exists in the enum definition, the QuantizationPlannerFactory lacks a functional implementation for createQ4_0Planner(), causing immediate runtime exceptions. The skeleton class Q4_0FloatTensor.java indicates planned support but requires additional kernel development and planner integration before activation.

How do I specify GPU memory allocation when running quantized models?

Use the --gpu-memory command-line flag followed by the desired allocation (e.g., --gpu-memory 15GB). This flag works independently of quantization format and is often necessary when running 3B or larger parameter models, even with Q8_0 compression, to exceed the default 7GB GPU memory limit.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}