# Best Practices for Deploying GPULlama3.java in Docker Containers with GPU Support

> Deploy GPULlama3.java in Docker with GPU. Use official images, the --gpus flag, and read-only model volumes. Optimize with backend selection and sufficient device memory for peak performance.

- Repository: [Beehive lab/gpullama3.java](https://github.com/beehive-lab/gpullama3.java)
- Tags: best-practices
- Published: 2026-02-26

---

**Deploy GPULlama3.java using the official pre-built Docker images with the `--gpus` flag, mount your GGUF models as read-only volumes, and select the correct backend (`--opencl` or `--ptx`) based on your GPU architecture while allocating sufficient device memory via `--gpu-memory`.**

GPULlama3.java is a Java 21-based LLM inference engine that leverages TornadoVM to execute transformer kernels on GPUs. When deploying in production environments, containerization provides reproducible environments and isolates the complex native dependencies required by TornadoVM. This guide covers the best practices for deploying GPULlama3.java in Docker containers with GPU support, based on the official beehive-lab/gpullama3.java repository.

## Understanding the Docker Image Architecture

### Pre-built Images for OpenCL and PTX Backends

The project maintains two distinct production images optimized for different GPU backends. The `beehivelab/gpullama3.java-nvidia-openjdk-opencl` image targets the **OpenCL** backend, which works across NVIDIA, Intel, and AMD GPUs. The `beehivelab/gpullama3.java-nvidia-openjdk-ptx` image targets the **PTX/CUDA** backend, delivering optimal performance on NVIDIA hardware that exposes the PTX driver.

Both images bundle JDK 21, the complete TornadoVM SDK (native libraries and Java modules), the compiled `gpu-llama3` JAR, and the `llama-tornado` CLI script located at `/gpullama3/GPULlama3.java/llama-tornado`.

### Internal Structure and Key Files

The container's runtime behavior is governed by several key source files that define how CLI options translate into JVM execution:

- **[`LlamaTornadoCli.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/LlamaTornadoCli.java)**: The JBang-compatible entry point that parses arguments via the `Options` record and launches inference. It constructs the execution pipeline by loading models through `ModelLoader` and invoking either `runInteractive` or `runInstructOnce`.
- **[`Options.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Options.java)**: An immutable record storing all CLI flags, including the critical `useTornadovm` boolean derived from the `use.tornadovm` system property.
- **[`TornadoFlags.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TornadoFlags.java)**: Contains JBang directives that inject the required `--add-modules`, `--add-exports`, and native library paths (`-Djava.library.path`) necessary for TornadoVM initialization.

## Essential Deployment Configuration

### GPU Runtime Access with --gpus

To expose host GPUs to the container, you must use the Docker `--gpus` flag. The standard configuration `--gpus all` makes every GPU available, while `--gpus device=0` restricts access to a specific device. Without this flag, the container falls back to CPU execution via the Vector API, bypassing GPU acceleration entirely.

### Volume Mounting for Model Files

Never embed GGUF model files inside the image. Instead, mount them as read-only volumes to keep images lightweight and allow runtime model swapping. The standard convention maps the host working directory to `/data` inside the container:

```bash
-v "$PWD":/data:ro

```

Reference models using absolute paths like `/data/beehive-llama-3.2-1b-instruct-fp16.gguf`.

### Memory Allocation with --gpu-memory

TornadoVM requires explicit device memory configuration. The `--gpu-memory` CLI flag sets the `Dtornado.device.memory` property, defaulting to **7 GB**. For production deployments:

- **1B-3B models**: 7-10 GB
- **7B models**: 15 GB
- **8B+ models**: ≥20 GB

Insufficient memory causes `CL_MEM_OBJECT_ALLOCATION_FAILURE` or similar native errors during kernel execution.

## Backend Selection and CLI Options

### OpenCL vs PTX Backends

Select the backend based on your hardware and performance requirements:

**OpenCL Backend**
- **Image**: `beehivelab/gpullama3.java-nvidia-openjdk-opencl`
- **Flag**: `--opencl`
- **Compatibility**: NVIDIA, Intel, AMD GPUs; macOS via OpenCL
- **Use case**: Heterogeneous environments or non-NVIDIA hardware

**PTX Backend**
- **Image**: `beehivelab/gpullama3.java-nvidia-openjdk-ptx`
- **Flag**: `--ptx`
- **Compatibility**: NVIDIA GPUs with CUDA/PTX drivers
- **Use case**: Maximum performance on NVIDIA datacenter or consumer GPUs

### Understanding LlamaTornadoCli.java and Options.java

When the container executes `llama-tornado`, it invokes [`LlamaTornadoCli.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/LlamaTornadoCli.java) with the parsed `Options` record. The CLI constructs a complete Java command that includes:

1. **Module system flags** from [`TornadoFlags.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TornadoFlags.java) (`--add-modules jdk.incubator.vector,tornado.runtime`)
2. **Export directives** for internal TornadoVM packages
3. **Native library paths** pointing to `/opt/tornadovm` (set via `TORNADOVM_HOME`)

You can inspect the exact command using the `--show-command` flag, which is invaluable for debugging container environments.

## Docker Compose Configuration

For reproducible multi-service deployments, use Docker Compose with the NVIDIA runtime:

```yaml
version: "3.9"
services:
  gpullama3:
    image: beehivelab/gpullama3.java-nvidia-openjdk-opencl
    runtime: nvidia
    environment:
      - TORNADOVM_HOME=/opt/tornadovm
      - JAVA_OPTIONS=-Dtornado.device.memory=15GB
    volumes:
      - ./models:/data:ro
    command: >
      /gpullama3/GPULlama3.java/llama-tornado
      --gpu --opencl
      --model /data/beehive-llama-3.2-3b-instruct-fp16.gguf
      --prompt "Explain quantum computing"
      --gpu-memory 15GB

```

Note that `runtime: nvidia` is required for Docker versions prior to 20.10; newer versions should use the `deploy.resources.reservations.devices` syntax or the `--gpus` flag in the command override.

## Debugging and Verification

### Inspecting Generated JVM Commands

To verify that the container correctly configures TornadoVM, add the `--show-command` flag to your execution:

```bash
docker run --rm -it --gpus all \
  -v "$PWD":/data \
  beehivelab/gpullama3.java-nvidia-openjdk-opencl \
  /gpullama3/GPULlama3.java/llama-tornado \
  --gpu --opencl \
  --model /data/model.gguf \
  --show-command

```

The output displays the complete Java invocation including all `--add-modules`, `--add-exports`, and `-Djava.library.path` entries defined in [`TornadoFlags.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TornadoFlags.java), confirming that the container environment matches TornadoVM's requirements.

### Monitoring GPU Utilization

Verify GPU acceleration is active using host-level monitoring tools:

- **`nvidia-smi`**: Check GPU memory usage and compute utilization
- **`nvtop`**: Real-time process-level GPU monitoring
- **TornadoVM debug flags**: Add `--print-threads` or `--print-kernel` to see GPU kernel execution details in the container logs

If the container runs but GPU utilization remains at 0%, verify that `--gpus` was passed to Docker and that the correct backend flag (`--opencl` or `--ptx`) matches the pulled image variant.

## Summary

- **Use official images**: Choose between `beehivelab/gpullama3.java-nvidia-openjdk-opencl` (cross-platform) and `...-ptx` (NVIDIA-optimized) based on your hardware.
- **Mount models externally**: Always use volume mounts (`-v "$PWD":/data`) for GGUF files to keep images lightweight and avoid rebuilds.
- **Allocate sufficient GPU memory**: Set `--gpu-memory` according to model size (7GB for 1B-3B, 15GB for 7B, ≥20GB for 8B+).
- **Verify TornadoVM initialization**: Use `--show-command` to inspect the JVM flags generated by [`LlamaTornadoCli.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/LlamaTornadoCli.java) and [`TornadoFlags.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TornadoFlags.java).
- **Enable GPU runtime**: Never forget the `--gpus all` Docker flag; without it, the container falls back to CPU execution.

## Frequently Asked Questions

### How do I choose between the OpenCL and PTX Docker images?

Select the **OpenCL** image (`beehivelab/gpullama3.java-nvidia-openjdk-opencl`) if you need cross-platform compatibility across NVIDIA, Intel, or AMD GPUs, or if you are running on macOS. Choose the **PTX** image (`beehivelab/gpullama3.java-nvidia-openjdk-ptx`) exclusively for NVIDIA GPUs when you require maximum performance, as the PTX backend generates CUDA-optimized machine code specifically for NVIDIA architectures.

### What happens if I forget to pass the `--gpus` flag to Docker?

Without the `--gpus` flag, the container cannot access the host GPU devices. In this scenario, [`LlamaTornadoCli.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/LlamaTornadoCli.java) detects the absence of GPU resources and falls back to CPU execution using the Vector API via `jdk.incubator.vector`. While functional, this eliminates the performance benefits of TornadoVM's GPU acceleration and may result in significantly slower inference times for large models.

### How do I verify that TornadoVM is actually using the GPU inside the container?

Run the container with the `--show-command` flag to inspect the JVM command constructed by [`LlamaTornadoCli.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/LlamaTornadoCli.java). This output reveals the `-Djava.library.path` pointing to `/opt/tornadovm` and the `--add-modules` directives from [`TornadoFlags.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TornadoFlags.java). Additionally, use host monitoring tools like `nvidia-smi` or `nvtop` to verify GPU memory allocation and compute utilization, or add TornadoVM debug flags like `--print-kernel` to see GPU kernel execution logs in the container output.