How to Debug GPU Kernel Execution and View Generated CUDA/OpenCL Code in TornadoVM

Enable the tornado.debug, tornado.fullDebug, and tornado.printKernel system properties when running the gpullama3.java CLI to dump generated GPU kernels to /tmp/tornado and print diagnostic execution logs.

The gpullama3.java repository accelerates large language model inference on GPUs using TornadoVM, a Java-to-GPU compiler. When optimizing kernel performance or troubleshooting device-specific issues, developers need visibility into the generated OpenCL or CUDA code. By leveraging TornadoVM's built-in debugging flags, you can intercept the compiled kernels before they reach the GPU driver and trace execution flow through the Java layers.

Enabling TornadoVM Debug Flags

TornadoVM exposes debugging facilities through JVM system properties that control kernel dumping and runtime diagnostics. The gpullama3.java project bundles these options into its CLI (LlamaTornadoCli.java), which forwards flags directly to the JVM.

Available Debugging Properties

According to the [README.md](https://github.com/beehive-lab/gpullama3.java/blob/main/README.md#L181) (lines 181-185), the repository supports three primary debugging modes:

• tornado.debug – Prints basic diagnostic information and the temporary work directory path where kernels are stored.
• tornado.fullDebug – Dumps the complete device code, including OpenCL C (.c or .cl files) for OpenCL backends or PTX (.ptx files) for CUDA backends.
• tornado.printKernel – Prints the textual representation of each kernel before compilation, allowing you to verify loop unrolling and memory access patterns.

You can enable these via JVM arguments (-Dproperty=value) or through the CLI wrapper's convenience flags (--debug, --full-dump, --print-kernel).

Locating and Inspecting Generated GPU Code

When debugging is enabled, TornadoVM creates a temporary work directory—defaulting to ${java.io.tmpdir}/tornado (typically /tmp/tornado on Unix systems). The console output identifies the exact path when tornado.debug is active:

[TornadoVM] Dumping generated kernel files to /tmp/tornado/...

File Formats by Backend

Inside the dump directory, you will find:

• kernel0.c, kernel1.c, etc. – Generated OpenCL C source files when using OpenCL devices.
• kernel0.ptx, kernel1.ptx, etc. – Generated PTX assembly when targeting NVIDIA CUDA GPUs.

These files contain the final device code after TornadoVM's byte-code-to-GPU translation, including vectorization transformations and memory coalescing optimizations applied by the compiler.

Architecture and Key Source Files

Understanding the repository structure helps correlate dumped kernels with their originating Java methods.

CLI Entry Point and Options

The [LlamaTornadoCli.java](https://github.com/beehive-lab/gpullama3.java/blob/main/LlamaTornadoCli.java) class serves as the entry point, forwarding unrecognized flags to the JVM. The [Options.java](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/Options.java#L7) record (lines 7-9) holds the useTornadovm flag, while debugging flags pass through directly as system properties.

Kernel Scheduling Infrastructure

The [WorkerGridFactory.java](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/WorkerGridFactory.java#L10) (lines 10-18) generates WorkerGrid objects that define kernel launch geometry (work-group sizes and grid dimensions). When inspecting dumped kernels, compare the __kernel function signatures against the factory's grid configurations to verify that TornadoVM mapped your Java parallel loops to the correct GPU thread hierarchy.

Layer Implementations

Specific model layers—such as [LogitsQ8_0Layer.java](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/LogitsQ8_0Layer.java#L30) (lines 30-36)—build TaskGraphs and submit them to the TornadoVM runtime. The generated kernel files correspond to these Java methods, allowing you to trace memory-access patterns from the high-level layer code down to the generated OpenCL or CUDA source.

Step-by-Step Debugging Workflow

Follow this sequence to capture and analyze GPU kernels from the gpullama3.java application:

1. Run with debug flags
   Execute the CLI with --debug, --full-dump, and --print-kernel to enable all diagnostic outputs:

   jbang LlamaTornadoCli.java \
       -m model.gguf \
       -p "Explain quantum computing" \
       --debug --full-dump --print-kernel

2. Identify the dump directory
   Locate the path printed in the console output (e.g., /tmp/tornado).

3. Inspect generated source
   List the directory contents to find your kernels:

   ls /tmp/tornado
   cat /tmp/tornado/kernel0.c

4. Correlate with Java source
   Open the corresponding layer file (e.g., LogitsQ8_0Layer.java) to map the generated kernel back to its originating Java method.

5. Validate with external tools (Optional)
   Use vendor-specific utilities to analyze the dumped code:

   # For OpenCL kernels
   
   clcc -c /tmp/tornado/kernel0.c
   
   # For PTX analysis
   
   nvdisasm /tmp/tornado/kernel0.ptx

Practical Code Examples

Running with JBang and Debug Flags

jbang LlamaTornadoCli.java \
    -m beehive-llama-3.2-1b-instruct-fp16.gguf \
    -p "Tell me a joke" \
    --debug \
    --full-dump \
    --print-kernel

Direct JVM Execution Without JBang

If running the packaged JAR directly, pass system properties to the Java command:

java -jar target/gpu-llama3-1.0-SNAPSHOT.jar \
    -m model.gguf \
    -p "Explain quantum computing" \
    -Dtornado.debug=true \
    -Dtornado.fullDebug=true \
    -Dtornado.printKernel=true

Programmatically Checking Debug Status

To verify debugging is enabled within your application code:

if (Boolean.getBoolean("tornado.debug")) {
    String dumpDir = System.getProperty("java.io.tmpdir") + "/tornado";
    System.out.println("[Debug] TornadoVM kernel dumps: " + dumpDir);
}

Using External Profilers

Once kernels are dumped to /tmp/tornado, you can profile them with vendor tools:

cd /tmp/tornado

# Example: Intel VTune for OpenCL analysis

vtune -collect gpu-hotspots -result-dir vtune-result ./application

Summary

• Enable debugging using -Dtornado.debug=true, -Dtornado.fullDebug=true, and -Dtornado.printKernel=true (or the equivalent --debug, --full-dump, --print-kernel CLI flags).
• Locate kernels in the temporary directory printed at runtime (default: /tmp/tornado).
• Inspect device code by reading the generated .c (OpenCL) or .ptx (CUDA) files to verify compiler transformations.
• Map to Java source by examining [WorkerGridFactory.java](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/WorkerGridFactory.java) for launch configurations and layer implementations like [LogitsQ8_0Layer.java](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/LogitsQ8_0Layer.java) for kernel logic.
• Analyze externally using tools like nvdisasm for PTX or standard OpenCL compilers for the generated C kernels.

Frequently Asked Questions

What is the difference between tornado.debug and tornado.fullDebug?

The tornado.debug property prints diagnostic startup messages and identifies the temporary directory where kernels are stored, while tornado.fullDebug actually writes the complete generated device code (OpenCL C or PTX) to that directory. You typically enable both simultaneously to see where the files are saved and then inspect their contents.

Where does TornadoVM store the generated CUDA or OpenCL code?

By default, TornadoVM stores generated kernels in ${java.io.tmpdir}/tornado, which resolves to /tmp/tornado on most Linux systems. The exact path is printed to the console when tornado.debug is enabled. You can override this location by setting the tornado.debug.dir system property.

Can I use vendor profiling tools with the dumped kernel files?

Yes. The generated .c (OpenCL) and .ptx (CUDA) files in the dump directory are standard source files that you can analyze with vendor-specific tools. For example, use nvdisasm to inspect PTX assembly for CUDA devices, or use Intel VTune and AMD CodeXL to profile the OpenCL C code before it is compiled by the GPU driver.

How do I map a generated kernel back to the original Java source code?

First, identify the kernel filename (e.g., kernel0.c) from the dump directory. Then, examine the layer-specific Java files—such as [LogitsQ8_0Layer.java](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/LogitsQ8_0Layer.java#L30) or [Qwen3Kernels.java](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/kernels/Qwen3Kernels.java)—to find the TaskGraph submission logic. The [WorkerGridFactory.java](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/WorkerGridFactory.java) defines the grid dimensions that appear as __global size qualifiers in the OpenCL code, helping you correlate the generated kernel signature with the Java parallel constructs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}