How to Configure JVM Heap Sizes (-Xms, -Xmx) for Optimal GPU Inference Performance in gpullama3.java

Set -Xms to 4–6 GB and -Xmx to 8–12 GB (or up to available host RAM) to ensure the JVM can hold LLM tensors and token buffers without triggering GC pauses that stall the GPU pipeline.

The gpullama3.java project accelerates large language model (LLM) inference by offloading compute kernels to the GPU via TornadoVM. Because TornadoVM operates on Java heap arrays that are later copied to GPU memory, configuring JVM heap sizes correctly is critical for throughput. This guide explains how to tune -Xms and -Xmx based on the actual source code architecture found in beehive-lab/gpullama3.java.

Why JVM Heap Size Matters for GPU Inference

In gpullama3.java, the JVM heap serves as the staging area for all data that moves to the GPU.

• Model weights are loaded from GGUF files into FloatTensor and Q8_0FloatTensor objects allocated on the heap (see StandardWeights.java).
• Input tokens and attention caches reside in heap arrays before being wrapped by TornadoTensor objects in TransformerComputeKernels.java.
• Intermediate results are copied back to the heap for string decoding in InferenceEngine.

If -Xmx is too small, the runtime throws OutOfMemoryError during model loading or forces frequent full-heap GC cycles that pause the GPU pipeline. If -Xms is left at the default, the JVM expands the heap incrementally, causing resize pauses exactly when the first large tensors are allocated.

Recommended JVM Heap Configuration for LLM Inference

For a 12 GB-class GPU running 7B-parameter models, the following settings balance stability and throughput:

Flag	Recommended Value	Purpose
-Xms	4g to 6g	Pre-allocates heap space equal to the model size (typically 3–5 GB for quantized 7B models) plus safety margin, eliminating expansion pauses.
-Xmx	8g to 12g (or up to 75% of available host RAM)	Provides headroom for large prompt batches, token streaming, and temporary work buffers while leaving memory for the OS and GPU driver.
-XX:+UseParallelGC	Already set in TornadoFlags.java	Parallel GC handles the large, short-lived allocations typical of token generation more efficiently than G1 for this workload.

These values scale linearly with model size: for a 13B model, add approximately 2–3 GB to both -Xms and -Xmx.

Where to Configure Heap Settings in gpullama3.java

TornadoFlags.java (Primary Configuration)

The project centralizes JVM options in TornadoFlags.java. Lines 7–14 define the //JAVA_OPTIONS directive that JBang reads when launching the CLI:

//JAVA_OPTIONS --enable-preview --enable-native-access=ALL-UNNAMED
//JAVA_OPTIONS -XX:+UseParallelGC
//JAVA_OPTIONS -Djava.library.path=...

Add your heap settings here to ensure every execution uses consistent memory limits:

//JAVA_OPTIONS -Xms6g -Xmx10g

Command-Line Overrides with JBang

When experimenting with different model sizes, pass heap flags directly via JBang’s -J prefix without editing source files:

jbang \
  -J-Xms6g \
  -J-Xmx10g \
  LlamaTornadoCli.java \
  --model models/llama3-7b.gguf \
  --prompt "Explain JVM heap tuning"

The -J flag forwards everything after it to the underlying java command, overriding any defaults in TornadoFlags.java.

Direct Java Execution

For production deployments that bypass JBang, include the flags in the java invocation:

java \
  -Xms6g \
  -Xmx10g \
  --enable-preview \
  --enable-native-access=ALL-UNNAMED \
  -XX:+UseParallelGC \
  -cp target/gpullama3-0.3.2.jar \
  org.beehive.gpullama3.cli.LlamaTornadoCli \
  --model path/to/model.gguf \
  --prompt "Test inference"

Ensure the classpath includes the compiled JAR and all TornadoVM dependencies.

Architecture Impact of Heap Sizing

Understanding the data flow explains why heap configuration directly impacts GPU throughput:

1. Model Loading (StandardWeights.java) – The StandardWeights implementation allocates FloatTensor objects on the heap to hold de-quantized weights. If -Xms is smaller than the model file, the JVM must expand the heap while parsing the GGUF header, causing I/O stalls.

2. Kernel Preparation (TransformerComputeKernels.java) – Before GPU execution, TornadoVM wraps heap arrays in TornadoTensor objects. These wrappers reference the same backing storage; no copy occurs until the kernel is scheduled. A constrained heap limits the maximum batch size that can be prepared.

3. GPU Execution – TornadoVM copies data from the heap to GPU memory via the TornadoVM runtime. If the heap is too small to hold both input and output buffers, the pipeline serializes transfers, reducing GPU utilization.

4. Result Handling (InferenceEngine) – Generated tokens are copied back to heap arrays for string decoding. Streaming mode (--stream) allocates new char arrays per token; sufficient -Xmx prevents allocation stalls during long generations.

Practical Configuration Examples

Launching from a Java Process

When integrating gpullama3.java into a larger application, use ProcessBuilder to ensure heap settings are isolated:

ProcessBuilder pb = new ProcessBuilder(
    "java",
    "-Xms6g",
    "-Xmx10g",
    "--enable-preview",
    "-cp", "target/gpullama3-0.3.2.jar",
    "org.beehive.gpullama3.cli.LlamaTornadoCli",
    "--model", "models/llama3-7b.gguf",
    "--prompt", "Summarize JVM heap tuning",
    "--stream"
);
pb.inheritIO();
Process process = pb.start();
process.waitFor();

Bash Alias for Daily Use

Create a shell alias to avoid typing flags repeatedly:

alias gpullama='jbang -J-Xms6g -J-Xmx10g LlamaTornadoCli.java'
gpullama --model models/phi3.gguf --prompt "What does -Xms do?"

Summary

• Pre-size the heap with -Xms equal to roughly the model size (4–6 GB for 7B models) to avoid expansion pauses during StandardWeights loading.
• Cap the heap with -Xmx at 8–12 GB (or 75% of host RAM) to accommodate large prompt batches and streaming generation without starving the OS or GPU driver.
• Configure in TornadoFlags.java for persistent settings, or use jbang -J-… for temporary overrides.
• Use ParallelGC (already enabled in the project) for efficient handling of short-lived tensor allocations.
• Verify host RAM availability; insufficient physical memory causes swapping that stalls GPU data transfers.

Frequently Asked Questions

What happens if -Xmx is too small for GPU inference?

If -Xmx is insufficient, the JVM throws OutOfMemoryError during model loading in StandardWeights.java or while allocating attention buffers in TransformerComputeKernels.java. Even if the error is avoided, frequent full-heap garbage collection pauses will stall the GPU pipeline, causing erratic inference latency and reduced throughput.

Should -Xms equal -Xmx for LLM inference?

Setting -Xms equal to -Xmx (e.g., -Xms10g -Xmx10g) is generally safe and eliminates heap-resize pauses, but it forces the JVM to commit all memory upfront. For development machines with limited RAM, start with -Xms at the model size and -Xmx higher, then monitor GC logs to find the minimum stable -Xmx before locking both values in production.

Does TornadoVM use off-heap memory for GPU buffers?

TornadoVM copies data from the Java heap to GPU device memory at kernel launch time; it does not use off-heap DirectByteBuffer for the primary tensor storage in this codebase. Consequently, the model weights in StandardWeights.java and input tensors in TransformerComputeKernels.java must fit comfortably within the configured -Xmx, plus overhead for the GC and TornadoVM runtime.

How do I check if my heap settings are working correctly?

Enable GC logging by adding -Xlog:gc*:file=gc.log:time,uptime,level,tags:filecount=5,filesize=100m to your java or jbang command. After running inference, inspect the log for Pause Full events or OutOfMemoryError traces. Ideally, you should see only young-generation collections (Pause Young) and no heap expansion messages (Heap resized). If the log shows frequent full GCs, increase -Xmx; if it shows heap expansion at startup, raise -Xms.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}