# How to Troubleshoot and Resolve TornadoOutOfMemoryException When Running Large Models in GPULLama3

> Resolve TornadoOutOfMemoryException for large models in GPULLama3. Clamp work-group size, switch quantization to Q8_0, and adjust batch size or context length to fit GPU memory.

- Repository: [Beehive lab/gpullama3.java](https://github.com/beehive-lab/gpullama3.java)
- Tags: how-to-guide
- Published: 2026-02-26

---

**You can resolve TornadoOutOfMemoryException by clamping the work-group size (localSize) to 128 or below, switching from FP16 to Q8_0 quantization, and ensuring your model weights fit within available GPU global memory by reducing batch size or context length.**

Running Llama-3-style models on TornadoVM through the GPULLama3 Java repository often triggers `TornadoVMException: Out of Memory` when transformer layers exhaust device limits. According to the [`beehive-lab/gpullama3.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/beehive-lab/gpullama3.java) source code, this occurs when **global GPU buffers** for model weights or **local memory** for per-work-group reductions exceed hardware capacity. Understanding how `TransformerComputeKernelsLayered` and `WorkerGridFactory` allocate memory is critical to troubleshoot and resolve TornadoOutOfMemoryException.

## Understand Memory Allocation in GPULLama3

GPULLama3 allocates two distinct memory regions that can trigger OOM errors: global memory for model weights and local memory for kernel scratch space.

### Global Weight Tensors

Model weights (FP16, Q8_0, etc.) are stored in **global GPU memory** via classes in `org.beehive.gpullama3.tensor.tornado.*`. For example, [`FP16TornadoTensor.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/FP16TornadoTensor.java) wraps native `ByteArray` objects that map directly to the GPU's global address space. When you load a 7B parameter model at FP16 precision (approximately 14 GB), allocation fails immediately on devices with less VRAM.

### Local Work-Group Buffers

Kernels allocate **local memory** (_shared memory on NVIDIA, local on OpenCL)_ for reductions and RMS-normalization. In [`TransformerComputeKernelsLayered.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TransformerComputeKernelsLayered.java) at lines 1496-1502, the code calls `context.allocateFloatLocalArray(localSize)` where `localSize` is derived from `WorkerGridFactory.createRmsNormWorker` (lines 14-18). If `localSize` exceeds the device's maximum work-group size (often 256 or 512) or the local memory per compute unit, TornadoVM throws an OOM.

## Diagnose TornadoVM Memory Failures

Systematic diagnosis requires inspecting both runtime configuration and hardware limits before inference begins.

### Enable Diagnostic Logging

Run the JVM with debug flags to see exact allocation sizes:

```bash
java -Dtornado.debug=true -Dtornado.profiler=true -jar gpullama3.jar

```

The console output displays the amount of global memory requested versus the device's total available bytes.

### Query Hardware Limits Programmatically

Query device capabilities directly in Java to determine safe upper bounds:

```java
import uk.ac.manchester.tornado.api.TornadoRuntime;

var runtime = TornadoRuntime.getRuntime();
var device = runtime.getDevice(0);

System.out.println("Global memory: " + device.getDeviceMemorySize() / (1024*1024) + " MB");
System.out.println("Max work-group size: " + device.getDeviceMaxWorkGroupSize());

```

If your model size (from the GGUF loader reports) exceeds the first value, or your `localSize` exceeds the second, you will encounter TornadoOutOfMemoryException.

## Common Causes and Fixes for TornadoOutOfMemoryException

### OOM on Global Memory (Model Too Large)

**Symptom:** Exception occurs during `ModelLoader.loadModel()` or initial tensor allocation.

**Likely cause:** The model weights exceed available VRAM (e.g., loading a 7B FP16 model on a 4 GB GPU).

**Fixes:**
- Switch to **Q8_0 quantization** to reduce memory by roughly 50%. See [`Qwen3Q8_0FFNLayers.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Qwen3Q8_0FFNLayers.java) (lines 27-30) for implementation details.
- Reduce **context length** via `Options.contextLength`.
- Decrease **batch size** via `Options.batchSize`.

### OOM on Local Memory (Work-Group Size Too Large)

**Symptom:** Exception thrown inside `TransformerComputeKernelsLayered` during kernel execution, specifically at `allocateFloatLocalArray`.

**Likely cause:** The `localSize` parameter passed to `WorkerGridFactory.createRmsNormWorker` exceeds the device's per-work-group local memory limit.

**Fix:** Cap `localSize` at 128 or below. Edit [`WorkerGridFactory.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/WorkerGridFactory.java):

```java
public static WorkerGrid createRmsNormWorker(int dim, int requestedLocalSize) {
    int maxLocal = 128;  // Safe upper bound for most GPUs
    int localSize = Math.min(requestedLocalSize, maxLocal);
    WorkerGrid worker = new WorkerGrid();
    worker.setLocalWork(localSize, 1, 1);
    return worker;
}

```

Alternatively, pass `--workers=128` on the CLI to override the configuration default.

### Memory Leaks Between Inference Runs

**Symptom:** OOM occurs only after multiple inference iterations despite fitting in memory initially.

**Likely cause:** Global tensors are not freed between runs.

**Fix:** Ensure `TornadoVMMasterPlan.freeDeviceMemory()` (lines 129-135) is invoked after each inference. The `InferenceEngine` already calls this, but if implementing custom loops, manually invoke:

```java
TornadoRuntime.getRuntime().reset();

```

## Code-Level Adjustments to Prevent OOM

### Switch to Q8_0 Quantization

Change the model loader to request quantized weights:

```java
import org.beehive.gpullama3.gguf.GGMLType;
import org.beehive.gpullama3.ModelLoader;

Model model = ModelLoader.loadModel(
    Paths.get("model.gguf"),
    2048,                   // contextLength
    true,                   // loadWeights
    true,                   // useTornadovm
    GGMLType.Q8_0           // quantization (not F16)
);

```

This loads the Q8_0 FFNLayers implementation instead of FP16 tensors, cutting memory usage in half.

### Reduce Runtime Parameters via CLI

Pass conservative limits at runtime:

```bash
java -jar gpullama3.jar \
  --model model.gguf \
  --batch 1 \
  --seq_len 1024 \
  --precision q8_0 \
  --workers 128 \
  -Dtornado.debug=true

```

### Pre-Allocation Sanity Check

Run this diagnostic before loading the model to validate your configuration:

```java
public class MemoryCheck {
    public static void main(String[] args) {
        var runtime = uk.ac.manchester.tornado.api.TornadoRuntime.getRuntime();
        var dev = runtime.getDevice(0);
        
        long globalMB = dev.getDeviceMemorySize() / (1024 * 1024);
        int maxWG = dev.getDeviceMaxWorkGroupSize();
        int safeLocal = Math.min(128, maxWG);
        
        System.out.println("Device memory: " + globalMB + " MB");
        System.out.println("Max work-group: " + maxWG);
        System.out.println("Recommended localSize: " + safeLocal);
    }
}

```

## Summary

- **Global memory** exhaustion occurs when model weights exceed VRAM; use Q8_0 quantization or reduce context/batch size via [`Options.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Options.java).
- **Local memory** OOM happens when `localSize` in `WorkerGridFactory` exceeds device limits; clamp to 128 or query `getDeviceMaxWorkGroupSize()`.
- **TornadoVMMasterPlan.java** manages device memory lifecycle; ensure `freeDeviceMemory()` is called after inference to prevent leaks.
- Enable `-Dtornado.debug=true` to see exact allocation sizes during `TransformerComputeKernelsLayered` execution.

## Frequently Asked Questions

### What causes TornadoOutOfMemoryException in GPULLama3?

The exception occurs when either **global GPU memory** (for storing model weights in `FP16TornadoTensor`) or **local work-group memory** (for reduction buffers in [`TransformerComputeKernelsLayered.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TransformerComputeKernelsLayered.java)) exceeds hardware limits. This typically happens when running large FP16 models on limited VRAM or using a `localSize` larger than the device's maximum work-group size (usually 256 or 512).

### How do I check my GPU memory limits in TornadoVM?

Query the device at runtime using `TornadoRuntime.getRuntime().getDevice(0).getDeviceMemorySize()` for global memory and `getDeviceMaxWorkGroupSize()` for local memory limits. Run this check before allocating tensors to ensure your model configuration fits within available resources.

### Can I run large models on GPUs with limited VRAM?

Yes, by switching to **Q8_0 quantization** (implemented in [`Qwen3Q8_0FFNLayers.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Qwen3Q8_0FFNLayers.java)) you reduce memory usage by approximately 50%. Additionally, reduce `Options.batchSize` and `Options.contextLength` to minimize activation memory during inference. These adjustments allow running 7B models on 4-6 GB GPUs.

### How do I fix local memory OOM errors?

Local memory errors stem from excessive `localSize` values in `WorkerGridFactory.createRmsNormWorker`. Edit the factory to clamp `localSize` to 128 or lower, or pass `--workers=128` on the command line. This ensures `allocateFloatLocalArray` requests in `TransformerComputeKernelsLayered` stay within the GPU's per-work-group shared memory limit.