# GPULlama3.java Quantization Formats: Q8_0, Q4_0, and FP16 GPU Memory Guide

> Explore GPULlama3.java quantization formats Q8_0 and FP16. Learn how Q8_0 cuts GPU memory in half and understand Q4_0 status for efficient AI model deployment.

- Repository: [Beehive lab/gpullama3.java](https://github.com/beehive-lab/gpullama3.java)
- Tags: deep-dive
- Published: 2026-02-26

---

**GPULlama3.java fully supports FP16 and Q8_0 quantization formats, with Q4_0 declared but unimplemented, where Q8_0 reduces GPU memory usage by approximately 50% compared to the FP16 baseline.**

GPULlama3.java is a GPU-accelerated Java inference engine for Llama 3 models that leverages TornadoVM for parallel execution on heterogeneous hardware. The framework implements multiple **quantization formats** to optimize memory consumption, enabling deployment of larger models on constrained GPU hardware. Understanding the differences between FP16, Q8_0, and the planned Q4_0 formats is critical for configuring efficient inference pipelines.

## Supported Quantization Formats in GPULlama3.java

The `GGMLType` enum in [`src/main/java/org/beehive/gpullama3/tensor/GGMLType.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tensor/GGMLType.java) defines the available precision formats, though the `QuantizationPlannerFactory` class determines actual runtime support.

### FP16 (16-bit Floating Point)

**FP16** serves as the baseline precision format, storing each weight as a 2-byte half-precision float. The `QuantizationPlannerFactory.createFP16Planner()` method instantiates full-precision layer planners for models using this format. While offering the highest numerical fidelity, FP16 consumes the most GPU memory, requiring approximately 2 bytes per parameter with no compression overhead.

### Q8_0 (8-bit Integer with Block Scaling)

**Q8_0** compresses weights to 1-byte integers accompanied by per-block scaling factors. According to the implementation in `QuantizationPlannerFactory.createQ8_0Planner()`, this format stores 2 bytes of metadata per 256-element block (scale and zero-point), yielding an effective size of roughly 1.007 bytes per weight. This achieves approximately **50% memory reduction** compared to FP16. The concrete GPU kernels for Q8_0 matrix operations reside in `src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/`.

### Q4_0 (4-bit Integer - Future Support)

While `GGMLType.Q4_0` is declared in the enum, the format remains unimplemented in the current release. Calling `QuantizationPlannerFactory.createQ4_0Planner()` throws an `UnsupportedOperationException`, as explicitly noted in the repository README. When eventually supported, Q4_0 would store weights as 4-bit integers with the same 2-byte block metadata structure, theoretically achieving approximately **75% memory reduction** (to ~25% of FP16 size). A skeleton tensor implementation exists in [`src/main/java/org/beehive/gpullama3/tensor/standard/Q4_0FloatTensor.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tensor/standard/Q4_0FloatTensor.java) but lacks planner integration and GPU kernel support.

## How Quantization Formats Impact GPU Memory Usage

The relationship between quantization precision and memory consumption follows predictable compression ratios based on bit-width and per-block overhead:

| Format | Bytes per Weight | Block Overhead | Total Memory vs FP16 |
|--------|-----------------|----------------|---------------------|
| **FP16** | 2.0 B | None | 100% (baseline) |
| **Q8_0** | 1.0 B | ~0.007 B per weight | ~50% |
| **Q4_0** | 0.5 B | ~0.008 B per weight | ~25% (planned) |

For practical deployment scenarios, a 1B-parameter model requires approximately 2GB in FP16, 1GB in Q8_0, and would require only 0.5GB once Q4_0 support arrives. Larger models (3B-8B parameters) often necessitate the `--gpu-memory` CLI flag to increase allocation beyond the default 7GB limit, though lower quantization formats proportionally reduce these requirements.

## Configuring Quantization Formats in GPULlama3.java

The framework automatically detects quantization formats from GGUF model filenames during initialization. Users select formats by loading appropriately quantized model files.

Run a model with FP16 precision:

```bash
./llama-tornado --gpu --model beehive-llama-3.2-1b-instruct-fp16.gguf \
    --prompt "Explain neural networks"

```

Execute with Q8_0 quantization for reduced memory footprint:

```bash
./llama-tornado --gpu --model beehive-llama-3.2-1b-instruct-q8_0.gguf \
    --prompt "Explain neural networks"

```

For models exceeding default GPU memory limits, increase allocation regardless of quantization format:

```bash
./llama-tornado --gpu --model beehive-llama-3.2-3b-instruct-q8_0.gguf \
    --gpu-memory 15GB --prompt "Explain neural networks"

```

Attempting to use Q4_0 models currently results in runtime exceptions until `QuantizationPlannerFactory` implements the corresponding planner method.

## Summary

- **FP16** provides full precision but requires 2 bytes per weight, serving as the baseline for GPU memory calculations.
- **Q8_0** is fully implemented and production-ready, reducing memory usage by approximately 50% through 8-bit quantization with per-block scaling factors.
- **Q4_0** is declared in [`GGMLType.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/GGMLType.java) but unimplemented; `createQ4_0Planner()` throws `UnsupportedOperationException` as noted in the README.
- Key implementation files include [`QuantizationPlannerFactory.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/QuantizationPlannerFactory.java) for planner creation and format-specific layer classes in `tornadovm/layers/type/q8_0/`.
- Use the `--gpu-memory` flag to accommodate larger models when quantization alone is insufficient for your hardware constraints.

## Frequently Asked Questions

### Which quantization formats can I use today in GPULlama3.java?

You can currently use **FP16** and **Q8_0** quantization formats. FP16 offers full precision while Q8_0 provides a balanced 50% memory reduction. The Q4_0 format appears in the codebase enum but remains unimplemented, with `QuantizationPlannerFactory.createQ4_0Planner()` explicitly throwing an `UnsupportedOperationException` when invoked.

### How much GPU memory does Q8_0 save compared to FP16?

Q8_0 reduces memory consumption by approximately **50%** compared to FP16. While FP16 requires 2 bytes per weight, Q8_0 uses 1 byte per weight plus negligible block metadata overhead (roughly 1.007 bytes effective per weight), allowing you to run models twice as large on the same GPU hardware.

### Why does attempting to use Q4_0 quantization fail?

Q4_0 support is listed as a future feature in the GPULlama3.java README. Although `GGMLType.Q4_0` exists in the enum definition, the `QuantizationPlannerFactory` lacks a functional implementation for `createQ4_0Planner()`, causing immediate runtime exceptions. The skeleton class [`Q4_0FloatTensor.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Q4_0FloatTensor.java) indicates planned support but requires additional kernel development and planner integration before activation.

### How do I specify GPU memory allocation when running quantized models?

Use the `--gpu-memory` command-line flag followed by the desired allocation (e.g., `--gpu-memory 15GB`). This flag works independently of quantization format and is often necessary when running 3B or larger parameter models, even with Q8_0 compression, to exceed the default 7GB GPU memory limit.