How to Debug JIT Compilation Issues in DeepGEMM Using DG_JIT_DEBUG and DG_PRINT_CONFIGS

Set DG_JIT_DEBUG=1 to print NVCC/NVRTC command lines and runtime loading steps, and set DG_PRINT_CONFIGS=1 to log the selected GEMM configuration for each problem shape.

DeepGEMM is a high-performance FP8 GEMM library that generates CUDA kernels on-the-fly via just-in-time (JIT) compilation. When kernels fail to compile, launch incorrectly, or exhibit unexpected performance, the environment variables DG_JIT_DEBUG and DG_PRINT_CONFIGS provide fine-grained visibility into every stage of the JIT pipeline.

Understanding the JIT Debugging Environment Variables

Both variables are read once per process via the deep_gemm::get_env helper located in csrc/utils/system.hpp (line 79). When enabled, they inject printf-style diagnostics at strategic compilation and runtime points.

DG_JIT_DEBUG for Low-Level Compiler Diagnostics

Setting DG_JIT_DEBUG=1 enables verbose output for the entire compiler toolchain. According to the source in csrc/jit/compiler.hpp (lines 154, 223, 235, and 304), this prints:

• The full NVCC or NVRTC command line used to compile the kernel
• PTX assembler options including --ptxas-options=--verbose,--warn-on-local-memory-usage
• Runtime API loading steps from csrc/jit/kernel_runtime.hpp (lines 42-47)
• cuobjdump disassembly commands when binary inspection is required

DG_PRINT_CONFIGS for GEMM Configuration Verification

Setting DG_PRINT_CONFIGS=1 logs the selected GEMM configuration (layout, storage, pipeline, and launch parameters) for each unique problem shape the first time it is encountered. As implemented in csrc/jit_kernels/heuristics/common.hpp (lines 40-49) and referenced in mega_moe.hpp, this activates when either DG_JIT_DEBUG or DG_PRINT_CONFIGS is set, helping verify that the heuristic selector picks the intended configuration for your specific matrix dimensions.

How to Enable JIT Debugging in Your Code

Enable these diagnostics before importing the deep_gemm module to ensure the environment is read during initialization.

Python Example

import os

# Enable verbose JIT debugging and config printing

os.environ["DG_JIT_DEBUG"] = "1"
os.environ["DG_PRINT_CONFIGS"] = "1"

import deep_gemm as dg

# Example GEMM operation that triggers JIT compilation

A = dg.rand((128, 256), dtype=dg.float16, device='cuda')
B = dg.rand((256, 512), dtype=dg.float16, device='cuda')
C = dg.matmul(A, B)

Bash Example

export DG_JIT_DEBUG=1
export DG_PRINT_CONFIGS=1
python my_deepgemm_benchmark.py

Interpreting the Debug Output

When enabled, DeepGEMM prints structured diagnostics that allow you to pinpoint whether issues originate from compiler flags, configuration selection, or runtime loading.

Compiler Command Line Verification

With DG_JIT_DEBUG=1, the full compiler invocation appears in stdout. For example:

nvcc -std=c++20 -arch=sm_90 -ptxas-options=--verbose,--warn-on-local-memory-usage -lineinfo -cubin kernel.cu -o kernel.cubin

Verify that:

• The -arch flag matches your GPU architecture (e.g., sm_90 for Hopper)
• The -std=c++20 flag is present as required by DeepGEMM
• PTXAS verbosity is enabled to catch register spilling or local memory usage

Configuration Selection Logs

When DG_PRINT_CONFIGS=1 is active, expect output similar to:

[DG JIT] Problem: GemmDesc(m=128, n=512, k=256, dtype=fp16)
[DG JIT] Selected config: layout=LayoutV4R2C8, storage=SM90_BMMA, pipeline=PipelineV3, launch=LaunchB64x64x4

Cross-reference these values against the expected optimal configuration for your matrix shape. If the heuristic selects an unexpected layout or pipeline, you may need to adjust input parameters or file an issue with the specific GemmDesc that triggers the suboptimal selection.

Runtime Loading Diagnostics

From csrc/jit/kernel_runtime.hpp (lines 42-47), DG_JIT_DEBUG=1 enables messages indicating:

• When a compiled kernel is loaded from disk cache versus newly compiled
• The elapsed load time (when DG_JIT_PRINT_LOAD_TIME is also set)
• Any runtime API errors during module loading

Step-by-Step Debugging Workflow

Follow this systematic approach to resolve JIT compilation failures:

1. Enable both flags before importing DeepGEMM:

   os.environ["DG_JIT_DEBUG"] = "1"
   os.environ["DG_PRINT_CONFIGS"] = "1"

2. Run the failing operation and capture full stdout/stderr.

3. Inspect the compiler command line from the output to verify:

   • Correct architecture flags (-arch=sm_90)
   • Presence of required C++ standard (-std=c++20)
   • Valid paths to source files and output directories

4. Verify configuration selection matches your performance expectations for the given matrix shape.

5. If compilation fails, copy the printed NVCC/NVRTC command and execute it manually in a terminal to reproduce the error in isolation and obtain full compiler diagnostics.

6. Check runtime loading messages to confirm the kernel loads successfully after compilation.

Key Source Files for JIT Debugging

File	Role in JIT Debugging	Relevant Lines
csrc/utils/system.hpp	Provides get_env helper and prints external commands when debugging is enabled.	system.hpp#L79-L81
csrc/jit/compiler.hpp	Constructs NVCC/NVRTC command lines; prints them when DG_JIT_DEBUG is set.	compiler.hpp#L154-L160
csrc/jit/kernel_runtime.hpp	Handles kernel loading; emits debug messages and load-time timing.	kernel_runtime.hpp#L42-L47
csrc/jit_kernels/heuristics/common.hpp	Prints selected GemmConfig when debug flags are active.	common.hpp#L40-L49
csrc/jit_kernels/impls/runtime_utils.hpp	Verbose PTX-as and cuobjdump diagnostics.	runtime_utils.hpp#L137-L143
README.md	Documents all environment variables.	README.md#L61-L64

Summary

• Set DG_JIT_DEBUG=1 to expose the full NVCC/NVRTC command line, PTX assembler options, and runtime loading steps.
• Set DG_PRINT_CONFIGS=1 to verify the heuristic selector picks the correct layout, storage, and pipeline for each matrix shape.
• Enable both variables before importing deep_gemm to ensure they are read during the one-time environment initialization in csrc/utils/system.hpp.
• Copy printed compiler commands to reproduce failures in isolation when debugging compilation errors.
• Reference the source files csrc/jit/compiler.hpp, csrc/jit/kernel_runtime.hpp, and csrc/jit_kernels/heuristics/common.hpp to understand exactly what diagnostics are printed at each stage.

Frequently Asked Questions

What is the difference between DG_JIT_DEBUG and DG_PRINT_CONFIGS?

DG_JIT_DEBUG focuses on the compiler and runtime mechanics, printing the NVCC/NVRTC command lines, PTX assembler verbose output, and kernel loading steps from csrc/jit/compiler.hpp and csrc/jit/kernel_runtime.hpp. DG_PRINT_CONFIGS focuses on algorithmic selection, logging the chosen GEMM configuration (layout, storage, pipeline) from csrc/jit_kernels/heuristics/common.hpp to help verify heuristic decisions.

When should I enable these debugging flags?

Enable both flags when you encounter compilation failures, unexpected performance regressions, or numerical inaccuracies that might stem from incorrect kernel selection. Set DG_JIT_DEBUG=1 specifically when you need to inspect compiler flags or reproduce a build failure outside of Python. Set DG_PRINT_CONFIGS=1 when validating that the JIT heuristics select optimal configurations for your specific matrix shapes (M, N, K dimensions).

Can I enable these flags after importing deep_gemm?

No. DeepGEMM reads environment variables once per process during initialization via the get_env helper in csrc/utils/system.hpp (line 79). If you set the variables after importing the module, the JIT compiler and heuristics have already cached the default values (0), and you will not see debug output. Always set DG_JIT_DEBUG and DG_PRINT_CONFIGS using os.environ or shell exports before the import deep_gemm statement.

How do I reproduce a compilation error manually?

When DG_JIT_DEBUG=1 is enabled, DeepGEMM prints the exact NVCC or NVRTC command line executed in csrc/jit/compiler.hpp (around line 154). Copy this printed command—including all flags like -std=c++20, -arch=sm_90, and --ptxas-options=--verbose—and run it directly in your terminal. This eliminates Python wrapping and gives you raw compiler diagnostics, making it easier to identify syntax errors, architecture mismatches, or missing dependencies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}