# How to Configure JVM Heap Sizes (-Xms, -Xmx) for Optimal GPU Inference Performance in gpullama3.java

> Optimize Java GPU inference with gpullama3.java by setting JVM heap sizes (-Xms, -Xmx). Learn recommended values to prevent GC pauses and maximize LLM performance. Boost your inference speed today.

- Repository: [Beehive lab/gpullama3.java](https://github.com/beehive-lab/gpullama3.java)
- Tags: how-to-guide
- Published: 2026-02-26

---

**Set `-Xms` to 4–6 GB and `-Xmx` to 8–12 GB (or up to available host RAM) to ensure the JVM can hold LLM tensors and token buffers without triggering GC pauses that stall the GPU pipeline.**

The **gpullama3.java** project accelerates large language model (LLM) inference by offloading compute kernels to the GPU via **TornadoVM**. Because TornadoVM operates on Java heap arrays that are later copied to GPU memory, configuring JVM heap sizes correctly is critical for throughput. This guide explains how to tune `-Xms` and `-Xmx` based on the actual source code architecture found in [`beehive-lab/gpullama3.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/beehive-lab/gpullama3.java).

## Why JVM Heap Size Matters for GPU Inference

In [`gpullama3.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/gpullama3.java), the JVM heap serves as the staging area for all data that moves to the GPU.

- **Model weights** are loaded from GGUF files into `FloatTensor` and `Q8_0FloatTensor` objects allocated on the heap (see [`StandardWeights.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/StandardWeights.java)).
- **Input tokens and attention caches** reside in heap arrays before being wrapped by `TornadoTensor` objects in [`TransformerComputeKernels.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TransformerComputeKernels.java).
- **Intermediate results** are copied back to the heap for string decoding in `InferenceEngine`.

If `-Xmx` is too small, the runtime throws `OutOfMemoryError` during model loading or forces frequent full-heap GC cycles that pause the GPU pipeline. If `-Xms` is left at the default, the JVM expands the heap incrementally, causing resize pauses exactly when the first large tensors are allocated.

## Recommended JVM Heap Configuration for LLM Inference

For a 12 GB-class GPU running 7B-parameter models, the following settings balance stability and throughput:

| Flag | Recommended Value | Purpose |
|------|------------------|---------|
| `-Xms` | `4g` to `6g` | Pre-allocates heap space equal to the model size (typically 3–5 GB for quantized 7B models) plus safety margin, eliminating expansion pauses. |
| `-Xmx` | `8g` to `12g` (or up to 75% of available host RAM) | Provides headroom for large prompt batches, token streaming, and temporary work buffers while leaving memory for the OS and GPU driver. |
| `-XX:+UseParallelGC` | Already set in [`TornadoFlags.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TornadoFlags.java) | Parallel GC handles the large, short-lived allocations typical of token generation more efficiently than G1 for this workload. |

These values scale linearly with model size: for a 13B model, add approximately 2–3 GB to both `-Xms` and `-Xmx`.

## Where to Configure Heap Settings in gpullama3.java

### TornadoFlags.java (Primary Configuration)

The project centralizes JVM options in **[`TornadoFlags.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TornadoFlags.java)**. Lines 7–14 define the `//JAVA_OPTIONS` directive that JBang reads when launching the CLI:

```java
//JAVA_OPTIONS --enable-preview --enable-native-access=ALL-UNNAMED
//JAVA_OPTIONS -XX:+UseParallelGC
//JAVA_OPTIONS -Djava.library.path=...

```

Add your heap settings here to ensure every execution uses consistent memory limits:

```java
//JAVA_OPTIONS -Xms6g -Xmx10g

```

### Command-Line Overrides with JBang

When experimenting with different model sizes, pass heap flags directly via JBang’s `-J` prefix without editing source files:

```bash
jbang \
  -J-Xms6g \
  -J-Xmx10g \
  LlamaTornadoCli.java \
  --model models/llama3-7b.gguf \
  --prompt "Explain JVM heap tuning"

```

The `-J` flag forwards everything after it to the underlying `java` command, overriding any defaults in [`TornadoFlags.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TornadoFlags.java).

### Direct Java Execution

For production deployments that bypass JBang, include the flags in the `java` invocation:

```bash
java \
  -Xms6g \
  -Xmx10g \
  --enable-preview \
  --enable-native-access=ALL-UNNAMED \
  -XX:+UseParallelGC \
  -cp target/gpullama3-0.3.2.jar \
  org.beehive.gpullama3.cli.LlamaTornadoCli \
  --model path/to/model.gguf \
  --prompt "Test inference"

```

Ensure the classpath includes the compiled JAR and all TornadoVM dependencies.

## Architecture Impact of Heap Sizing

Understanding the data flow explains why heap configuration directly impacts GPU throughput:

1. **Model Loading ([`StandardWeights.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/StandardWeights.java))** – The `StandardWeights` implementation allocates `FloatTensor` objects on the heap to hold de-quantized weights. If `-Xms` is smaller than the model file, the JVM must expand the heap while parsing the GGUF header, causing I/O stalls.

2. **Kernel Preparation ([`TransformerComputeKernels.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TransformerComputeKernels.java))** – Before GPU execution, TornadoVM wraps heap arrays in `TornadoTensor` objects. These wrappers reference the same backing storage; no copy occurs until the kernel is scheduled. A constrained heap limits the maximum batch size that can be prepared.

3. **GPU Execution** – TornadoVM copies data from the heap to GPU memory via the TornadoVM runtime. If the heap is too small to hold both input and output buffers, the pipeline serializes transfers, reducing GPU utilization.

4. **Result Handling (`InferenceEngine`)** – Generated tokens are copied back to heap arrays for string decoding. Streaming mode (`--stream`) allocates new char arrays per token; sufficient `-Xmx` prevents allocation stalls during long generations.

## Practical Configuration Examples

### Launching from a Java Process

When integrating [`gpullama3.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/gpullama3.java) into a larger application, use `ProcessBuilder` to ensure heap settings are isolated:

```java
ProcessBuilder pb = new ProcessBuilder(
    "java",
    "-Xms6g",
    "-Xmx10g",
    "--enable-preview",
    "-cp", "target/gpullama3-0.3.2.jar",
    "org.beehive.gpullama3.cli.LlamaTornadoCli",
    "--model", "models/llama3-7b.gguf",
    "--prompt", "Summarize JVM heap tuning",
    "--stream"
);
pb.inheritIO();
Process process = pb.start();
process.waitFor();

```

### Bash Alias for Daily Use

Create a shell alias to avoid typing flags repeatedly:

```bash
alias gpullama='jbang -J-Xms6g -J-Xmx10g LlamaTornadoCli.java'
gpullama --model models/phi3.gguf --prompt "What does -Xms do?"

```

## Summary

- **Pre-size the heap** with `-Xms` equal to roughly the model size (4–6 GB for 7B models) to avoid expansion pauses during `StandardWeights` loading.
- **Cap the heap** with `-Xmx` at 8–12 GB (or 75% of host RAM) to accommodate large prompt batches and streaming generation without starving the OS or GPU driver.
- **Configure in [`TornadoFlags.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TornadoFlags.java)** for persistent settings, or use `jbang -J-…` for temporary overrides.
- **Use ParallelGC** (already enabled in the project) for efficient handling of short-lived tensor allocations.
- **Verify host RAM availability**; insufficient physical memory causes swapping that stalls GPU data transfers.

## Frequently Asked Questions

### What happens if -Xmx is too small for GPU inference?

If `-Xmx` is insufficient, the JVM throws `OutOfMemoryError` during model loading in [`StandardWeights.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/StandardWeights.java) or while allocating attention buffers in [`TransformerComputeKernels.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TransformerComputeKernels.java). Even if the error is avoided, frequent full-heap garbage collection pauses will stall the GPU pipeline, causing erratic inference latency and reduced throughput.

### Should -Xms equal -Xmx for LLM inference?

Setting `-Xms` equal to `-Xmx` (e.g., `-Xms10g -Xmx10g`) is generally safe and eliminates heap-resize pauses, but it forces the JVM to commit all memory upfront. For development machines with limited RAM, start with `-Xms` at the model size and `-Xmx` higher, then monitor GC logs to find the minimum stable `-Xmx` before locking both values in production.

### Does TornadoVM use off-heap memory for GPU buffers?

TornadoVM copies data from the Java heap to GPU device memory at kernel launch time; it does not use off-heap `DirectByteBuffer` for the primary tensor storage in this codebase. Consequently, the model weights in [`StandardWeights.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/StandardWeights.java) and input tensors in [`TransformerComputeKernels.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TransformerComputeKernels.java) must fit comfortably within the configured `-Xmx`, plus overhead for the GC and TornadoVM runtime.

### How do I check if my heap settings are working correctly?

Enable GC logging by adding `-Xlog:gc*:file=gc.log:time,uptime,level,tags:filecount=5,filesize=100m` to your `java` or `jbang` command. After running inference, inspect the log for `Pause Full` events or `OutOfMemoryError` traces. Ideally, you should see only young-generation collections (`Pause Young`) and no heap expansion messages (`Heap resized`). If the log shows frequent full GCs, increase `-Xmx`; if it shows heap expansion at startup, raise `-Xms`.