# How to Debug JIT Compilation Issues in DeepGEMM Using DG_JIT_DEBUG and DG_PRINT_CONFIGS

> Debug DeepGEMM JIT compilation issues by enabling DG_JIT_DEBUG to view NVCC/NVRTC commands and DG_PRINT_CONFIGS to log GEMM configurations. Resolve performance bottlenecks efficiently.

- Repository: [DeepSeek/DeepGEMM](https://github.com/deepseek-ai/DeepGEMM)
- Tags: how-to-guide
- Published: 2026-04-19

---

**Set `DG_JIT_DEBUG=1` to print NVCC/NVRTC command lines and runtime loading steps, and set `DG_PRINT_CONFIGS=1` to log the selected GEMM configuration for each problem shape.**

DeepGEMM is a high-performance FP8 GEMM library that generates CUDA kernels on-the-fly via just-in-time (JIT) compilation. When kernels fail to compile, launch incorrectly, or exhibit unexpected performance, the environment variables `DG_JIT_DEBUG` and `DG_PRINT_CONFIGS` provide fine-grained visibility into every stage of the JIT pipeline.

## Understanding the JIT Debugging Environment Variables

Both variables are read **once per process** via the `deep_gemm::get_env` helper located in [`csrc/utils/system.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/utils/system.hpp) (line 79). When enabled, they inject `printf`-style diagnostics at strategic compilation and runtime points.

### DG_JIT_DEBUG for Low-Level Compiler Diagnostics

Setting `DG_JIT_DEBUG=1` enables verbose output for the entire compiler toolchain. According to the source in [`csrc/jit/compiler.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit/compiler.hpp) (lines 154, 223, 235, and 304), this prints:

- The full **NVCC or NVRTC command line** used to compile the kernel
- **PTX assembler options** including `--ptxas-options=--verbose,--warn-on-local-memory-usage`
- Runtime API loading steps from [`csrc/jit/kernel_runtime.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit/kernel_runtime.hpp) (lines 42-47)
- **`cuobjdump` disassembly** commands when binary inspection is required

### DG_PRINT_CONFIGS for GEMM Configuration Verification

Setting `DG_PRINT_CONFIGS=1` logs the **selected GEMM configuration** (layout, storage, pipeline, and launch parameters) for each unique problem shape the first time it is encountered. As implemented in [`csrc/jit_kernels/heuristics/common.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/heuristics/common.hpp) (lines 40-49) and referenced in [`mega_moe.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/mega_moe.hpp), this activates when either `DG_JIT_DEBUG` **or** `DG_PRINT_CONFIGS` is set, helping verify that the heuristic selector picks the intended configuration for your specific matrix dimensions.

## How to Enable JIT Debugging in Your Code

Enable these diagnostics **before** importing the `deep_gemm` module to ensure the environment is read during initialization.

### Python Example

```python
import os

# Enable verbose JIT debugging and config printing

os.environ["DG_JIT_DEBUG"] = "1"
os.environ["DG_PRINT_CONFIGS"] = "1"

import deep_gemm as dg

# Example GEMM operation that triggers JIT compilation

A = dg.rand((128, 256), dtype=dg.float16, device='cuda')
B = dg.rand((256, 512), dtype=dg.float16, device='cuda')
C = dg.matmul(A, B)

```

### Bash Example

```bash
export DG_JIT_DEBUG=1
export DG_PRINT_CONFIGS=1
python my_deepgemm_benchmark.py

```

## Interpreting the Debug Output

When enabled, DeepGEMM prints structured diagnostics that allow you to pinpoint whether issues originate from compiler flags, configuration selection, or runtime loading.

### Compiler Command Line Verification

With `DG_JIT_DEBUG=1`, the full compiler invocation appears in stdout. For example:

```bash
nvcc -std=c++20 -arch=sm_90 -ptxas-options=--verbose,--warn-on-local-memory-usage -lineinfo -cubin kernel.cu -o kernel.cubin

```

Verify that:
- The `-arch` flag matches your GPU architecture (e.g., `sm_90` for Hopper)
- The `-std=c++20` flag is present as required by DeepGEMM
- PTXAS verbosity is enabled to catch register spilling or local memory usage

### Configuration Selection Logs

When `DG_PRINT_CONFIGS=1` is active, expect output similar to:

```

[DG JIT] Problem: GemmDesc(m=128, n=512, k=256, dtype=fp16)
[DG JIT] Selected config: layout=LayoutV4R2C8, storage=SM90_BMMA, pipeline=PipelineV3, launch=LaunchB64x64x4

```

Cross-reference these values against the expected optimal configuration for your matrix shape. If the heuristic selects an unexpected layout or pipeline, you may need to adjust input parameters or file an issue with the specific `GemmDesc` that triggers the suboptimal selection.

### Runtime Loading Diagnostics

From [`csrc/jit/kernel_runtime.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit/kernel_runtime.hpp) (lines 42-47), `DG_JIT_DEBUG=1` enables messages indicating:
- When a compiled kernel is loaded from disk cache versus newly compiled
- The elapsed load time (when `DG_JIT_PRINT_LOAD_TIME` is also set)
- Any runtime API errors during module loading

## Step-by-Step Debugging Workflow

Follow this systematic approach to resolve JIT compilation failures:

1. **Enable both flags** before importing DeepGEMM:
   ```python
   os.environ["DG_JIT_DEBUG"] = "1"
   os.environ["DG_PRINT_CONFIGS"] = "1"
   ```

2. **Run the failing operation** and capture full stdout/stderr.

3. **Inspect the compiler command line** from the output to verify:
   - Correct architecture flags (`-arch=sm_90`)
   - Presence of required C++ standard (`-std=c++20`)
   - Valid paths to source files and output directories

4. **Verify configuration selection** matches your performance expectations for the given matrix shape.

5. **If compilation fails**, copy the printed NVCC/NVRTC command and execute it manually in a terminal to reproduce the error in isolation and obtain full compiler diagnostics.

6. **Check runtime loading messages** to confirm the kernel loads successfully after compilation.

## Key Source Files for JIT Debugging

| File | Role in JIT Debugging | Relevant Lines |
|------|----------------------|----------------|
| [`csrc/utils/system.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/utils/system.hpp) | Provides `get_env` helper and prints external commands when debugging is enabled. | [system.hpp#L79-L81](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/utils/system.hpp#L79) |
| [`csrc/jit/compiler.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit/compiler.hpp) | Constructs NVCC/NVRTC command lines; prints them when `DG_JIT_DEBUG` is set. | [compiler.hpp#L154-L160](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit/compiler.hpp#L154) |
| [`csrc/jit/kernel_runtime.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit/kernel_runtime.hpp) | Handles kernel loading; emits debug messages and load-time timing. | [kernel_runtime.hpp#L42-L47](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit/kernel_runtime.hpp#L42) |
| [`csrc/jit_kernels/heuristics/common.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/heuristics/common.hpp) | Prints selected `GemmConfig` when debug flags are active. | [common.hpp#L40-L49](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/heuristics/common.hpp#L40) |
| [`csrc/jit_kernels/impls/runtime_utils.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/impls/runtime_utils.hpp) | Verbose PTX-as and cuobjdump diagnostics. | [runtime_utils.hpp#L137-L143](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/impls/runtime_utils.hpp#L137) |
| [`README.md`](https://github.com/deepseek-ai/DeepGEMM/blob/main/README.md) | Documents all environment variables. | [README.md#L61-L64](https://github.com/deepseek-ai/DeepGEMM/blob/main/README.md#L61) |

## Summary

- **Set `DG_JIT_DEBUG=1`** to expose the full NVCC/NVRTC command line, PTX assembler options, and runtime loading steps.
- **Set `DG_PRINT_CONFIGS=1`** to verify the heuristic selector picks the correct layout, storage, and pipeline for each matrix shape.
- **Enable both variables before importing** `deep_gemm` to ensure they are read during the one-time environment initialization in [`csrc/utils/system.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/utils/system.hpp).
- **Copy printed compiler commands** to reproduce failures in isolation when debugging compilation errors.
- **Reference the source files** [`csrc/jit/compiler.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit/compiler.hpp), [`csrc/jit/kernel_runtime.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit/kernel_runtime.hpp), and [`csrc/jit_kernels/heuristics/common.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/heuristics/common.hpp) to understand exactly what diagnostics are printed at each stage.

## Frequently Asked Questions

### What is the difference between DG_JIT_DEBUG and DG_PRINT_CONFIGS?

`DG_JIT_DEBUG` focuses on the **compiler and runtime mechanics**, printing the NVCC/NVRTC command lines, PTX assembler verbose output, and kernel loading steps from [`csrc/jit/compiler.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit/compiler.hpp) and [`csrc/jit/kernel_runtime.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit/kernel_runtime.hpp). `DG_PRINT_CONFIGS` focuses on **algorithmic selection**, logging the chosen GEMM configuration (layout, storage, pipeline) from [`csrc/jit_kernels/heuristics/common.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/heuristics/common.hpp) to help verify heuristic decisions.

### When should I enable these debugging flags?

Enable both flags when you encounter **compilation failures**, **unexpected performance regressions**, or **numerical inaccuracies** that might stem from incorrect kernel selection. Set `DG_JIT_DEBUG=1` specifically when you need to inspect compiler flags or reproduce a build failure outside of Python. Set `DG_PRINT_CONFIGS=1` when validating that the JIT heuristics select optimal configurations for your specific matrix shapes (M, N, K dimensions).

### Can I enable these flags after importing deep_gemm?

No. DeepGEMM reads environment variables **once per process** during initialization via the `get_env` helper in [`csrc/utils/system.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/utils/system.hpp) (line 79). If you set the variables after importing the module, the JIT compiler and heuristics have already cached the default values (`0`), and you will not see debug output. Always set `DG_JIT_DEBUG` and `DG_PRINT_CONFIGS` using `os.environ` or shell exports **before** the `import deep_gemm` statement.

### How do I reproduce a compilation error manually?

When `DG_JIT_DEBUG=1` is enabled, DeepGEMM prints the exact NVCC or NVRTC command line executed in [`csrc/jit/compiler.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit/compiler.hpp) (around line 154). Copy this printed command—including all flags like `-std=c++20`, `-arch=sm_90`, and `--ptxas-options=--verbose`—and run it directly in your terminal. This eliminates Python wrapping and gives you raw compiler diagnostics, making it easier to identify syntax errors, architecture mismatches, or missing dependencies.