# How to Use the TornadoVM Profiler to Identify Performance Bottlenecks in gpullama3.java Inference

> Learn to use the TornadoVM profiler to find performance bottlenecks in gpullama3.java inference. Analyze CSV reports to pinpoint slow operations and optimize your GPU kernel execution.

- Repository: [Beehive lab/gpullama3.java](https://github.com/beehive-lab/gpullama3.java)
- Tags: how-to-guide
- Published: 2026-02-26

---

**Enable the TornadoVM profiler by setting `-Dtornado.profiler.enabled=true` when running the Llama CLI to generate a CSV report of GPU kernel execution times, then inspect the output to locate slow matmul or memory-bound operations.**

The **gpullama3.java** repository by beehive-lab implements high-performance LLM inference using TornadoVM to execute transformer layers on GPUs. Profiling this inference pipeline requires instrumenting the GPU task graph and analyzing kernel-level metrics to identify whether bottlenecks stem from compute-bound matrix multiplications or memory-bound data transfers.

## Enabling the TornadoVM Profiler

The TornadoVM profiler hooks automatically into the `TaskGraph` built by `InferenceCore.forwardTornadoVM`. You activate it via JVM system properties without modifying any Java source code.

### Required JVM Flags

Pass the following flags to the Java runtime:

```bash
-Dtornado.profiler.enabled=true \
-Dtornado.profiler.output=profiler-report.csv

```

When these flags are present, TornadoVM instruments every kernel in the `TaskGraph` generated by `TornadoVMMasterPlan` and writes a CSV report to the specified path upon program termination.

### CLI Launch Example

Run the `LlamaTornadoCli` entry point with profiling enabled:

```bash
java \
  -Dtornado.profiler.enabled=true \
  -Dtornado.profiler.output=gpuprofile.csv \
  -cp target/gpullama3.jar \
  org.beehive.gpullama3.LlamaTornadoCli \
  --model models/llama-7b.ggmlv3.q8_0.bin \
  --prompt "Explain quantum computing"

```

The profiler captures data for all GPU operations orchestrated through `InferenceCore.forwardTornadoVM` (lines 70-78 in [`src/main/java/org/beehive/gpullama3/inference/InferenceCore.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/inference/InferenceCore.java)), which constructs the `TornadoVMMasterPlan` and submits it to the TornadoVM runtime.

## Understanding the Profiler Output

The TornadoVM profiler outputs a CSV file containing per-kernel execution metrics. This file reveals exactly which operations consume the most GPU time and whether they are limited by compute or memory bandwidth.

### CSV Structure and Key Metrics

The generated CSV contains columns for:

- **task**: Kernel name (e.g., `matmul_q`, `rmsnorm`)
- **device**: Target device identifier (e.g., `GPU`)
- **executionTimeNs**: Wall-clock time in nanoseconds
- **bytesRead**: Total bytes transferred from global memory to the kernel
- **bytesWritten**: Total bytes written back to global memory

A typical row looks like:

```

matmul_q,GPU,12345678,1024,1024
rmsnorm,GPU,234567,2048,0

```

### Interpreting Kernel Execution Times

Sort the CSV by `executionTimeNs` descending to identify the hottest kernels. In LLM inference, you should expect:

1. **Matrix multiplication kernels** (`matmul_q`, `matmul_k`, `matmul_v`, and feed-forward matmuls) to dominate the timeline in large models
2. **Normalization layers** (`rmsnorm`) to appear frequently but with lower individual cost
3. **High `bytesRead`/`bytesWritten` ratios** indicating memory-bound operations, common with quantized weights (Q8_0, FP16)

If `bytesRead` exceeds the theoretical minimum for the algorithm, consider enabling `TornadoRuntime.setMemoryAllocationPolicy` or fusing kernels to reduce traffic.

## Complementary Profiling Techniques

While the TornadoVM profiler captures GPU-side behavior, Java-side overhead in the token loop requires different instrumentation. The repository provides two utilities for this purpose.

### Manual Timing with the Timer Utility

The `Timer` class in [`src/main/java/org/beehive/gpullama3/auxiliary/Timer.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/auxiliary/Timer.java) provides an `AutoCloseable` wrapper for wall-clock measurements. Wrap suspect Java blocks to measure embedding lookups, token decoding, or tensor copies:

```java
try (Timer t = Timer.log("Embedding lookup")) {
    // Token embedding copy performed in forwardTornadoVM
    // ...
}

```

The output prints to standard error:

```

Embedding lookup: 2.3 ms

```

Use this inside `InferenceEngine.generateTokensGPULlama` to isolate latency outside the GPU task graph, such as the sample-and-decode phase.

### LastRunMetrics for End-to-End Benchmarks

For aggregate throughput measurements, the repository uses `LastRunMetrics` ([`src/main/java/org/beehive/gpullama3/auxiliary/LastRunMetrics.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/auxiliary/LastRunMetrics.java)). Call `setMetrics(totalTokens, totalSeconds)` after the generation loop to record overall performance:

```java
long start = System.nanoTime();
List<Integer> out = InferenceEngine.generateTokensGPULlama(...);
long total = System.nanoTime() - start;
int tokens = out.size();
LastRunMetrics.setMetrics(tokens, total / 1_000_000_000.0);
LastRunMetrics.printMetrics();  // prints totalTokens and totalSeconds

```

This utility complements the fine-grained TornadoVM profiler data by confirming whether optimizations actually improved end-to-end token-per-second rates.

## Step-by-Step Bottleneck Analysis Workflow

Follow this workflow to systematically locate and resolve performance issues:

1. **Enable profiling** using the JVM flags shown above and run a representative prompt
2. **Collect the CSV** after the program exits (default: `tornado-profiler-*.csv` in the working directory)
3. **Sort by `executionTimeNs`** to identify the longest-running kernels
4. **Correlate with memory metrics**: High `bytesRead` values alongside high execution times suggest memory bandwidth limits
5. **Add `Timer` blocks** around the token loop in `InferenceEngine.generateTokensGPULlama` if the sum of GPU kernels does not equal total wall-clock time, indicating Java-side overhead
6. **Iterate**: Modify quantization levels, adjust `TaskGraph` fusion in `TornadoVMMasterPlan`, or change batch sizes, then re-run the profiler to verify shifts in the bottleneck

## Code Examples

### Profiling a Custom Inference Loop

To profile specific sections of the token generation loop alongside the automatic GPU profiling:

```java
// Inside InferenceEngine.generateTokensGPULlama
while (pos < actualMaxTokens) {
    // GPU work is automatically profiled by TornadoVM
    FloatArray logits = InferenceCore.forwardTornadoVM(
        model, state, currentToken, pos, tornadoVMPlan
    );

    // Profile Java-side decoding separately
    try (Timer t = Timer.log("Decode + sample")) {
        if (promptIndex < promptTokens.size()) {
            nextToken = promptTokens.get(promptIndex++);
        } else {
            nextToken = sampler.sampleToken(logits);
        }
    }

    generatedTokens.add(nextToken);
    currentToken = nextToken;
    state.latestToken = currentToken;
    pos++;
}

```

### Reading the CSV Report Programmatically

Parse the profiler output to automate regression testing:

```java
Path csv = Paths.get("gpuprofile.csv");
try (Stream<String> lines = Files.lines(csv)) {
    lines.skip(1)  // skip header
         .map(line -> line.split(","))
         .sorted(Comparator.comparingLong(a -> 
             Long.parseLong(a[2])))  // index 2 = executionTimeNs
         .limit(5)  // top 5 slowest
         .forEach(fields -> System.out.printf(
             "Kernel %s on %s took %.2f ms%n",
             fields[0], 
             fields[1], 
             Long.parseLong(fields[2]) / 1_000_000.0
         ));
}

```

## Summary

- **Enable the TornadoVM profiler** by setting `-Dtornado.profiler.enabled=true` to capture per-kernel GPU execution times and memory traffic automatically
- **Analyze the CSV output** to identify whether matrix multiplications or memory transfers dominate the inference pipeline
- **Use `Timer.log`** from [`Timer.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Timer.java) to measure Java-side overhead in the token loop that falls outside the TaskGraph
- **Record aggregate metrics** with `LastRunMetrics` to validate end-to-end throughput improvements after optimization
- **Focus optimization efforts** on kernels with the highest `executionTimeNs` and investigate high `bytesRead` values for memory-bound quantization schemes

## Frequently Asked Questions

### How do I know if the TornadoVM profiler is actually running?

If the profiler is enabled correctly, a CSV file appears in your working directory immediately after the JVM process terminates. The filename defaults to `tornado-profiler-*.csv` or the specific path set by `-Dtornado.profiler.output`. If no file appears, verify that you are running the `forwardTornadoVM` code path (via `LlamaTornadoCli` or `LlamaApp`) rather than the CPU-only fallback, and that the system property is passed to the JVM before the main class.

### What should I do if the CSV shows high execution times for all kernels?

Uniformly high execution times across all kernels typically indicate GPU underutilization or excessive synchronization overhead. Check the `bytesRead` and `bytesWritten` columns for unexpectedly high values, which suggest the model weights are causing memory bandwidth saturation. Consider using a more aggressive quantization format (e.g., Q4_0 instead of Q8_0) or increasing batch size to improve arithmetic intensity. Also verify that `SchedulerDetectionService` has correctly identified your GPU device.

### Can I profile individual transformer layers separately?

The TornadoVM profiler records each task in the `TaskGraph` as defined in [`TornadoVMMasterPlan.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/TornadoVMMasterPlan.java). If `TornadoVMMasterPlan` constructs separate tasks per layer (e.g., `layer_0_matmul`, `layer_1_matmul`), they appear as distinct rows in the CSV. If the plan fuses the entire forward pass into a single task, you will only see aggregate data. To isolate specific layers, modify `TornadoVMMasterPlan` to create intermediate task boundaries, though this may introduce synchronization overhead.

### Why do my manual `Timer` measurements show latency not present in the TornadoVM profiler?

The TornadoVM profiler only instruments code running inside the submitted `TaskGraph`. Java-side operations such as token decoding, sampling, and string formatting occur on the host CPU between task graph executions. If `Timer` shows significant time while the profiler CSV shows minimal GPU activity, the bottleneck lies in the host-side loop within `InferenceEngine.generateTokensGPULlama` rather than in the GPU kernels.