# How to Debug GPU Kernel Execution and View Generated CUDA/OpenCL Code in TornadoVM

> Debug GPU kernel execution in TornadoVM and view generated CUDA/OpenCL code. Enable debug system properties to dump kernels and print diagnostic logs for easier troubleshooting.

- Repository: [Beehive lab/gpullama3.java](https://github.com/beehive-lab/gpullama3.java)
- Tags: how-to-guide
- Published: 2026-02-26

---

**Enable the `tornado.debug`, `tornado.fullDebug`, and `tornado.printKernel` system properties when running the gpullama3.java CLI to dump generated GPU kernels to `/tmp/tornado` and print diagnostic execution logs.**

The gpullama3.java repository accelerates large language model inference on GPUs using TornadoVM, a Java-to-GPU compiler. When optimizing kernel performance or troubleshooting device-specific issues, developers need visibility into the generated OpenCL or CUDA code. By leveraging TornadoVM's built-in debugging flags, you can intercept the compiled kernels before they reach the GPU driver and trace execution flow through the Java layers.

## Enabling TornadoVM Debug Flags

TornadoVM exposes debugging facilities through JVM system properties that control kernel dumping and runtime diagnostics. The gpullama3.java project bundles these options into its CLI ([`LlamaTornadoCli.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/LlamaTornadoCli.java)), which forwards flags directly to the JVM.

### Available Debugging Properties

According to the [[`README.md`](https://github.com/beehive-lab/gpullama3.java/blob/main/README.md)](https://github.com/beehive-lab/gpullama3.java/blob/main/README.md#L181) (lines 181-185), the repository supports three primary debugging modes:

- **`tornado.debug`** – Prints basic diagnostic information and the temporary work directory path where kernels are stored.
- **`tornado.fullDebug`** – Dumps the complete device code, including OpenCL C (`.c` or `.cl` files) for OpenCL backends or PTX (`.ptx` files) for CUDA backends.
- **`tornado.printKernel`** – Prints the textual representation of each kernel before compilation, allowing you to verify loop unrolling and memory access patterns.

You can enable these via JVM arguments (`-Dproperty=value`) or through the CLI wrapper's convenience flags (`--debug`, `--full-dump`, `--print-kernel`).

## Locating and Inspecting Generated GPU Code

When debugging is enabled, TornadoVM creates a temporary work directory—defaulting to `${java.io.tmpdir}/tornado` (typically `/tmp/tornado` on Unix systems). The console output identifies the exact path when `tornado.debug` is active:

```text
[TornadoVM] Dumping generated kernel files to /tmp/tornado/...

```

### File Formats by Backend

Inside the dump directory, you will find:

- **[`kernel0.c`](https://github.com/beehive-lab/gpullama3.java/blob/main/kernel0.c)**, **[`kernel1.c`](https://github.com/beehive-lab/gpullama3.java/blob/main/kernel1.c)**, etc. – Generated **OpenCL C** source files when using OpenCL devices.
- **`kernel0.ptx`**, **`kernel1.ptx`**, etc. – Generated **PTX** assembly when targeting NVIDIA CUDA GPUs.

These files contain the final device code after TornadoVM's byte-code-to-GPU translation, including vectorization transformations and memory coalescing optimizations applied by the compiler.

## Architecture and Key Source Files

Understanding the repository structure helps correlate dumped kernels with their originating Java methods.

### CLI Entry Point and Options

The [[`LlamaTornadoCli.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/LlamaTornadoCli.java)](https://github.com/beehive-lab/gpullama3.java/blob/main/LlamaTornadoCli.java) class serves as the entry point, forwarding unrecognized flags to the JVM. The [[`Options.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Options.java)](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/Options.java#L7) record (lines 7-9) holds the `useTornadovm` flag, while debugging flags pass through directly as system properties.

### Kernel Scheduling Infrastructure

The [[`WorkerGridFactory.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/WorkerGridFactory.java)](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/WorkerGridFactory.java#L10) (lines 10-18) generates **WorkerGrid** objects that define kernel launch geometry (work-group sizes and grid dimensions). When inspecting dumped kernels, compare the `__kernel` function signatures against the factory's grid configurations to verify that TornadoVM mapped your Java parallel loops to the correct GPU thread hierarchy.

### Layer Implementations

Specific model layers—such as [[`LogitsQ8_0Layer.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/LogitsQ8_0Layer.java)](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/LogitsQ8_0Layer.java#L30) (lines 30-36)—build **TaskGraphs** and submit them to the TornadoVM runtime. The generated kernel files correspond to these Java methods, allowing you to trace memory-access patterns from the high-level layer code down to the generated OpenCL or CUDA source.

## Step-by-Step Debugging Workflow

Follow this sequence to capture and analyze GPU kernels from the gpullama3.java application:

1. **Run with debug flags**  
   Execute the CLI with `--debug`, `--full-dump`, and `--print-kernel` to enable all diagnostic outputs:
   ```bash
   jbang LlamaTornadoCli.java \
       -m model.gguf \
       -p "Explain quantum computing" \
       --debug --full-dump --print-kernel
   ```

2. **Identify the dump directory**  
   Locate the path printed in the console output (e.g., `/tmp/tornado`).

3. **Inspect generated source**  
   List the directory contents to find your kernels:
   ```bash
   ls /tmp/tornado
   cat /tmp/tornado/kernel0.c
   ```

4. **Correlate with Java source**  
   Open the corresponding layer file (e.g., [`LogitsQ8_0Layer.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/LogitsQ8_0Layer.java)) to map the generated kernel back to its originating Java method.

5. **Validate with external tools** (Optional)  
   Use vendor-specific utilities to analyze the dumped code:
   ```bash
   # For OpenCL kernels

   clcc -c /tmp/tornado/kernel0.c
   
   # For PTX analysis

   nvdisasm /tmp/tornado/kernel0.ptx
   ```

## Practical Code Examples

### Running with JBang and Debug Flags

```bash
jbang LlamaTornadoCli.java \
    -m beehive-llama-3.2-1b-instruct-fp16.gguf \
    -p "Tell me a joke" \
    --debug \
    --full-dump \
    --print-kernel

```

### Direct JVM Execution Without JBang

If running the packaged JAR directly, pass system properties to the Java command:

```bash
java -jar target/gpu-llama3-1.0-SNAPSHOT.jar \
    -m model.gguf \
    -p "Explain quantum computing" \
    -Dtornado.debug=true \
    -Dtornado.fullDebug=true \
    -Dtornado.printKernel=true

```

### Programmatically Checking Debug Status

To verify debugging is enabled within your application code:

```java
if (Boolean.getBoolean("tornado.debug")) {
    String dumpDir = System.getProperty("java.io.tmpdir") + "/tornado";
    System.out.println("[Debug] TornadoVM kernel dumps: " + dumpDir);
}

```

### Using External Profilers

Once kernels are dumped to `/tmp/tornado`, you can profile them with vendor tools:

```bash
cd /tmp/tornado

# Example: Intel VTune for OpenCL analysis

vtune -collect gpu-hotspots -result-dir vtune-result ./application

```

## Summary

- **Enable debugging** using `-Dtornado.debug=true`, `-Dtornado.fullDebug=true`, and `-Dtornado.printKernel=true` (or the equivalent `--debug`, `--full-dump`, `--print-kernel` CLI flags).
- **Locate kernels** in the temporary directory printed at runtime (default: `/tmp/tornado`).
- **Inspect device code** by reading the generated `.c` (OpenCL) or `.ptx` (CUDA) files to verify compiler transformations.
- **Map to Java source** by examining [[`WorkerGridFactory.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/WorkerGridFactory.java)](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/WorkerGridFactory.java) for launch configurations and layer implementations like [[`LogitsQ8_0Layer.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/LogitsQ8_0Layer.java)](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/LogitsQ8_0Layer.java) for kernel logic.
- **Analyze externally** using tools like `nvdisasm` for PTX or standard OpenCL compilers for the generated C kernels.

## Frequently Asked Questions

### What is the difference between `tornado.debug` and `tornado.fullDebug`?

The `tornado.debug` property prints diagnostic startup messages and identifies the temporary directory where kernels are stored, while `tornado.fullDebug` actually writes the complete generated device code (OpenCL C or PTX) to that directory. You typically enable both simultaneously to see where the files are saved and then inspect their contents.

### Where does TornadoVM store the generated CUDA or OpenCL code?

By default, TornadoVM stores generated kernels in `${java.io.tmpdir}/tornado`, which resolves to `/tmp/tornado` on most Linux systems. The exact path is printed to the console when `tornado.debug` is enabled. You can override this location by setting the `tornado.debug.dir` system property.

### Can I use vendor profiling tools with the dumped kernel files?

Yes. The generated `.c` (OpenCL) and `.ptx` (CUDA) files in the dump directory are standard source files that you can analyze with vendor-specific tools. For example, use `nvdisasm` to inspect PTX assembly for CUDA devices, or use Intel VTune and AMD CodeXL to profile the OpenCL C code before it is compiled by the GPU driver.

### How do I map a generated kernel back to the original Java source code?

First, identify the kernel filename (e.g., [`kernel0.c`](https://github.com/beehive-lab/gpullama3.java/blob/main/kernel0.c)) from the dump directory. Then, examine the layer-specific Java files—such as [[`LogitsQ8_0Layer.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/LogitsQ8_0Layer.java)](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/LogitsQ8_0Layer.java#L30) or [[`Qwen3Kernels.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Qwen3Kernels.java)](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/kernels/Qwen3Kernels.java)—to find the TaskGraph submission logic. The [[`WorkerGridFactory.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/WorkerGridFactory.java)](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/WorkerGridFactory.java) defines the grid dimensions that appear as `__global` size qualifiers in the OpenCL code, helping you correlate the generated kernel signature with the Java parallel constructs.