How to Use the TornadoVM Profiler to Identify Performance Bottlenecks in gpullama3.java Inference

Enable the TornadoVM profiler by setting -Dtornado.profiler.enabled=true when running the Llama CLI to generate a CSV report of GPU kernel execution times, then inspect the output to locate slow matmul or memory-bound operations.

The gpullama3.java repository by beehive-lab implements high-performance LLM inference using TornadoVM to execute transformer layers on GPUs. Profiling this inference pipeline requires instrumenting the GPU task graph and analyzing kernel-level metrics to identify whether bottlenecks stem from compute-bound matrix multiplications or memory-bound data transfers.

Enabling the TornadoVM Profiler

The TornadoVM profiler hooks automatically into the TaskGraph built by InferenceCore.forwardTornadoVM. You activate it via JVM system properties without modifying any Java source code.

Required JVM Flags

Pass the following flags to the Java runtime:

-Dtornado.profiler.enabled=true \
-Dtornado.profiler.output=profiler-report.csv

When these flags are present, TornadoVM instruments every kernel in the TaskGraph generated by TornadoVMMasterPlan and writes a CSV report to the specified path upon program termination.

CLI Launch Example

Run the LlamaTornadoCli entry point with profiling enabled:

java \
  -Dtornado.profiler.enabled=true \
  -Dtornado.profiler.output=gpuprofile.csv \
  -cp target/gpullama3.jar \
  org.beehive.gpullama3.LlamaTornadoCli \
  --model models/llama-7b.ggmlv3.q8_0.bin \
  --prompt "Explain quantum computing"

The profiler captures data for all GPU operations orchestrated through InferenceCore.forwardTornadoVM (lines 70-78 in src/main/java/org/beehive/gpullama3/inference/InferenceCore.java), which constructs the TornadoVMMasterPlan and submits it to the TornadoVM runtime.

Understanding the Profiler Output

The TornadoVM profiler outputs a CSV file containing per-kernel execution metrics. This file reveals exactly which operations consume the most GPU time and whether they are limited by compute or memory bandwidth.

CSV Structure and Key Metrics

The generated CSV contains columns for:

• task: Kernel name (e.g., matmul_q, rmsnorm)
• device: Target device identifier (e.g., GPU)
• executionTimeNs: Wall-clock time in nanoseconds
• bytesRead: Total bytes transferred from global memory to the kernel
• bytesWritten: Total bytes written back to global memory

A typical row looks like:

matmul_q,GPU,12345678,1024,1024
rmsnorm,GPU,234567,2048,0

Interpreting Kernel Execution Times

Sort the CSV by executionTimeNs descending to identify the hottest kernels. In LLM inference, you should expect:

1. Matrix multiplication kernels (matmul_q, matmul_k, matmul_v, and feed-forward matmuls) to dominate the timeline in large models
2. Normalization layers (rmsnorm) to appear frequently but with lower individual cost
3. High bytesRead/bytesWritten ratios indicating memory-bound operations, common with quantized weights (Q8_0, FP16)

If bytesRead exceeds the theoretical minimum for the algorithm, consider enabling TornadoRuntime.setMemoryAllocationPolicy or fusing kernels to reduce traffic.

Complementary Profiling Techniques

While the TornadoVM profiler captures GPU-side behavior, Java-side overhead in the token loop requires different instrumentation. The repository provides two utilities for this purpose.

Manual Timing with the Timer Utility

The Timer class in src/main/java/org/beehive/gpullama3/auxiliary/Timer.java provides an AutoCloseable wrapper for wall-clock measurements. Wrap suspect Java blocks to measure embedding lookups, token decoding, or tensor copies:

try (Timer t = Timer.log("Embedding lookup")) {
    // Token embedding copy performed in forwardTornadoVM
    // ...
}

The output prints to standard error:

Embedding lookup: 2.3 ms

Use this inside InferenceEngine.generateTokensGPULlama to isolate latency outside the GPU task graph, such as the sample-and-decode phase.

LastRunMetrics for End-to-End Benchmarks

For aggregate throughput measurements, the repository uses LastRunMetrics (src/main/java/org/beehive/gpullama3/auxiliary/LastRunMetrics.java). Call setMetrics(totalTokens, totalSeconds) after the generation loop to record overall performance:

long start = System.nanoTime();
List<Integer> out = InferenceEngine.generateTokensGPULlama(...);
long total = System.nanoTime() - start;
int tokens = out.size();
LastRunMetrics.setMetrics(tokens, total / 1_000_000_000.0);
LastRunMetrics.printMetrics();  // prints totalTokens and totalSeconds

This utility complements the fine-grained TornadoVM profiler data by confirming whether optimizations actually improved end-to-end token-per-second rates.

Step-by-Step Bottleneck Analysis Workflow

Follow this workflow to systematically locate and resolve performance issues:

1. Enable profiling using the JVM flags shown above and run a representative prompt
2. Collect the CSV after the program exits (default: tornado-profiler-*.csv in the working directory)
3. Sort by executionTimeNs to identify the longest-running kernels
4. Correlate with memory metrics: High bytesRead values alongside high execution times suggest memory bandwidth limits
5. Add Timer blocks around the token loop in InferenceEngine.generateTokensGPULlama if the sum of GPU kernels does not equal total wall-clock time, indicating Java-side overhead
6. Iterate: Modify quantization levels, adjust TaskGraph fusion in TornadoVMMasterPlan, or change batch sizes, then re-run the profiler to verify shifts in the bottleneck

Code Examples

Profiling a Custom Inference Loop

To profile specific sections of the token generation loop alongside the automatic GPU profiling:

// Inside InferenceEngine.generateTokensGPULlama
while (pos < actualMaxTokens) {
    // GPU work is automatically profiled by TornadoVM
    FloatArray logits = InferenceCore.forwardTornadoVM(
        model, state, currentToken, pos, tornadoVMPlan
    );

    // Profile Java-side decoding separately
    try (Timer t = Timer.log("Decode + sample")) {
        if (promptIndex < promptTokens.size()) {
            nextToken = promptTokens.get(promptIndex++);
        } else {
            nextToken = sampler.sampleToken(logits);
        }
    }

    generatedTokens.add(nextToken);
    currentToken = nextToken;
    state.latestToken = currentToken;
    pos++;
}

Reading the CSV Report Programmatically

Parse the profiler output to automate regression testing:

Path csv = Paths.get("gpuprofile.csv");
try (Stream<String> lines = Files.lines(csv)) {
    lines.skip(1)  // skip header
         .map(line -> line.split(","))
         .sorted(Comparator.comparingLong(a -> 
             Long.parseLong(a[2])))  // index 2 = executionTimeNs
         .limit(5)  // top 5 slowest
         .forEach(fields -> System.out.printf(
             "Kernel %s on %s took %.2f ms%n",
             fields[0], 
             fields[1], 
             Long.parseLong(fields[2]) / 1_000_000.0
         ));
}

Summary

• Enable the TornadoVM profiler by setting -Dtornado.profiler.enabled=true to capture per-kernel GPU execution times and memory traffic automatically
• Analyze the CSV output to identify whether matrix multiplications or memory transfers dominate the inference pipeline
• Use Timer.log from Timer.java to measure Java-side overhead in the token loop that falls outside the TaskGraph
• Record aggregate metrics with LastRunMetrics to validate end-to-end throughput improvements after optimization
• Focus optimization efforts on kernels with the highest executionTimeNs and investigate high bytesRead values for memory-bound quantization schemes

Frequently Asked Questions

How do I know if the TornadoVM profiler is actually running?

If the profiler is enabled correctly, a CSV file appears in your working directory immediately after the JVM process terminates. The filename defaults to tornado-profiler-*.csv or the specific path set by -Dtornado.profiler.output. If no file appears, verify that you are running the forwardTornadoVM code path (via LlamaTornadoCli or LlamaApp) rather than the CPU-only fallback, and that the system property is passed to the JVM before the main class.

What should I do if the CSV shows high execution times for all kernels?

Uniformly high execution times across all kernels typically indicate GPU underutilization or excessive synchronization overhead. Check the bytesRead and bytesWritten columns for unexpectedly high values, which suggest the model weights are causing memory bandwidth saturation. Consider using a more aggressive quantization format (e.g., Q4_0 instead of Q8_0) or increasing batch size to improve arithmetic intensity. Also verify that SchedulerDetectionService has correctly identified your GPU device.

Can I profile individual transformer layers separately?

The TornadoVM profiler records each task in the TaskGraph as defined in TornadoVMMasterPlan.java. If TornadoVMMasterPlan constructs separate tasks per layer (e.g., layer_0_matmul, layer_1_matmul), they appear as distinct rows in the CSV. If the plan fuses the entire forward pass into a single task, you will only see aggregate data. To isolate specific layers, modify TornadoVMMasterPlan to create intermediate task boundaries, though this may introduce synchronization overhead.

Why do my manual Timer measurements show latency not present in the TornadoVM profiler?

The TornadoVM profiler only instruments code running inside the submitted TaskGraph. Java-side operations such as token decoding, sampling, and string formatting occur on the host CPU between task graph executions. If Timer shows significant time while the profiler CSV shows minimal GPU activity, the bottleneck lies in the host-side loop within InferenceEngine.generateTokensGPULlama rather than in the GPU kernels.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}