Best Practices for Deploying GPULlama3.java in Docker Containers with GPU Support

Deploy GPULlama3.java using the official pre-built Docker images with the --gpus flag, mount your GGUF models as read-only volumes, and select the correct backend (--opencl or --ptx) based on your GPU architecture while allocating sufficient device memory via --gpu-memory.

GPULlama3.java is a Java 21-based LLM inference engine that leverages TornadoVM to execute transformer kernels on GPUs. When deploying in production environments, containerization provides reproducible environments and isolates the complex native dependencies required by TornadoVM. This guide covers the best practices for deploying GPULlama3.java in Docker containers with GPU support, based on the official beehive-lab/gpullama3.java repository.

Understanding the Docker Image Architecture

Pre-built Images for OpenCL and PTX Backends

The project maintains two distinct production images optimized for different GPU backends. The beehivelab/gpullama3.java-nvidia-openjdk-opencl image targets the OpenCL backend, which works across NVIDIA, Intel, and AMD GPUs. The beehivelab/gpullama3.java-nvidia-openjdk-ptx image targets the PTX/CUDA backend, delivering optimal performance on NVIDIA hardware that exposes the PTX driver.

Both images bundle JDK 21, the complete TornadoVM SDK (native libraries and Java modules), the compiled gpu-llama3 JAR, and the llama-tornado CLI script located at /gpullama3/GPULlama3.java/llama-tornado.

Internal Structure and Key Files

The container's runtime behavior is governed by several key source files that define how CLI options translate into JVM execution:

• LlamaTornadoCli.java: The JBang-compatible entry point that parses arguments via the Options record and launches inference. It constructs the execution pipeline by loading models through ModelLoader and invoking either runInteractive or runInstructOnce.
• Options.java: An immutable record storing all CLI flags, including the critical useTornadovm boolean derived from the use.tornadovm system property.
• TornadoFlags.java: Contains JBang directives that inject the required --add-modules, --add-exports, and native library paths (-Djava.library.path) necessary for TornadoVM initialization.

Essential Deployment Configuration

GPU Runtime Access with --gpus

To expose host GPUs to the container, you must use the Docker --gpus flag. The standard configuration --gpus all makes every GPU available, while --gpus device=0 restricts access to a specific device. Without this flag, the container falls back to CPU execution via the Vector API, bypassing GPU acceleration entirely.

Volume Mounting for Model Files

Never embed GGUF model files inside the image. Instead, mount them as read-only volumes to keep images lightweight and allow runtime model swapping. The standard convention maps the host working directory to /data inside the container:

-v "$PWD":/data:ro

Reference models using absolute paths like /data/beehive-llama-3.2-1b-instruct-fp16.gguf.

Memory Allocation with --gpu-memory

TornadoVM requires explicit device memory configuration. The --gpu-memory CLI flag sets the Dtornado.device.memory property, defaulting to 7 GB. For production deployments:

• 1B-3B models: 7-10 GB
• 7B models: 15 GB
• 8B+ models: ≥20 GB

Insufficient memory causes CL_MEM_OBJECT_ALLOCATION_FAILURE or similar native errors during kernel execution.

Backend Selection and CLI Options

OpenCL vs PTX Backends

Select the backend based on your hardware and performance requirements:

OpenCL Backend

• Image: beehivelab/gpullama3.java-nvidia-openjdk-opencl
• Flag: --opencl
• Compatibility: NVIDIA, Intel, AMD GPUs; macOS via OpenCL
• Use case: Heterogeneous environments or non-NVIDIA hardware

PTX Backend

• Image: beehivelab/gpullama3.java-nvidia-openjdk-ptx
• Flag: --ptx
• Compatibility: NVIDIA GPUs with CUDA/PTX drivers
• Use case: Maximum performance on NVIDIA datacenter or consumer GPUs

Understanding LlamaTornadoCli.java and Options.java

When the container executes llama-tornado, it invokes LlamaTornadoCli.java with the parsed Options record. The CLI constructs a complete Java command that includes:

1. Module system flags from TornadoFlags.java (--add-modules jdk.incubator.vector,tornado.runtime)
2. Export directives for internal TornadoVM packages
3. Native library paths pointing to /opt/tornadovm (set via TORNADOVM_HOME)

You can inspect the exact command using the --show-command flag, which is invaluable for debugging container environments.

Docker Compose Configuration

For reproducible multi-service deployments, use Docker Compose with the NVIDIA runtime:

version: "3.9"
services:
  gpullama3:
    image: beehivelab/gpullama3.java-nvidia-openjdk-opencl
    runtime: nvidia
    environment:
      - TORNADOVM_HOME=/opt/tornadovm
      - JAVA_OPTIONS=-Dtornado.device.memory=15GB
    volumes:
      - ./models:/data:ro
    command: >
      /gpullama3/GPULlama3.java/llama-tornado
      --gpu --opencl
      --model /data/beehive-llama-3.2-3b-instruct-fp16.gguf
      --prompt "Explain quantum computing"
      --gpu-memory 15GB

Note that runtime: nvidia is required for Docker versions prior to 20.10; newer versions should use the deploy.resources.reservations.devices syntax or the --gpus flag in the command override.

Debugging and Verification

Inspecting Generated JVM Commands

To verify that the container correctly configures TornadoVM, add the --show-command flag to your execution:

docker run --rm -it --gpus all \
  -v "$PWD":/data \
  beehivelab/gpullama3.java-nvidia-openjdk-opencl \
  /gpullama3/GPULlama3.java/llama-tornado \
  --gpu --opencl \
  --model /data/model.gguf \
  --show-command

The output displays the complete Java invocation including all --add-modules, --add-exports, and -Djava.library.path entries defined in TornadoFlags.java, confirming that the container environment matches TornadoVM's requirements.

Monitoring GPU Utilization

Verify GPU acceleration is active using host-level monitoring tools:

• nvidia-smi: Check GPU memory usage and compute utilization
• nvtop: Real-time process-level GPU monitoring
• TornadoVM debug flags: Add --print-threads or --print-kernel to see GPU kernel execution details in the container logs

If the container runs but GPU utilization remains at 0%, verify that --gpus was passed to Docker and that the correct backend flag (--opencl or --ptx) matches the pulled image variant.

Summary

• Use official images: Choose between beehivelab/gpullama3.java-nvidia-openjdk-opencl (cross-platform) and ...-ptx (NVIDIA-optimized) based on your hardware.
• Mount models externally: Always use volume mounts (-v "$PWD":/data) for GGUF files to keep images lightweight and avoid rebuilds.
• Allocate sufficient GPU memory: Set --gpu-memory according to model size (7GB for 1B-3B, 15GB for 7B, ≥20GB for 8B+).
• Verify TornadoVM initialization: Use --show-command to inspect the JVM flags generated by LlamaTornadoCli.java and TornadoFlags.java.
• Enable GPU runtime: Never forget the --gpus all Docker flag; without it, the container falls back to CPU execution.

Frequently Asked Questions

How do I choose between the OpenCL and PTX Docker images?

Select the OpenCL image (beehivelab/gpullama3.java-nvidia-openjdk-opencl) if you need cross-platform compatibility across NVIDIA, Intel, or AMD GPUs, or if you are running on macOS. Choose the PTX image (beehivelab/gpullama3.java-nvidia-openjdk-ptx) exclusively for NVIDIA GPUs when you require maximum performance, as the PTX backend generates CUDA-optimized machine code specifically for NVIDIA architectures.

What happens if I forget to pass the --gpus flag to Docker?

Without the --gpus flag, the container cannot access the host GPU devices. In this scenario, LlamaTornadoCli.java detects the absence of GPU resources and falls back to CPU execution using the Vector API via jdk.incubator.vector. While functional, this eliminates the performance benefits of TornadoVM's GPU acceleration and may result in significantly slower inference times for large models.

How do I verify that TornadoVM is actually using the GPU inside the container?

Run the container with the --show-command flag to inspect the JVM command constructed by LlamaTornadoCli.java. This output reveals the -Djava.library.path pointing to /opt/tornadovm and the --add-modules directives from TornadoFlags.java. Additionally, use host monitoring tools like nvidia-smi or nvtop to verify GPU memory allocation and compute utilization, or add TornadoVM debug flags like --print-kernel to see GPU kernel execution logs in the container output.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}