DeepGEMM CUDA Graph Compatibility for Masked Grouped GEMMs: Implementation Guide

DeepGEMM achieves CUDA Graph compatibility for masked grouped GEMMs by implementing stateless Triton kernels that perform no host-side allocation, accept all metadata as pre-allocated tensor arguments, and explicitly avoid persistent kernel patterns that break graph capture.

DeepGEMM, developed by DeepSeek AI, provides high-performance grouped GEMM implementations optimized for mixture-of-experts (MoE) architectures. This article examines how DeepGEMM CUDA Graph compatibility for masked grouped GEMMs is achieved through careful kernel design that eliminates dynamic host operations and ensures deterministic launch configurations.

Stateless Kernel Architecture for DeepGEMM CUDA Graph Support

The foundation of CUDA Graph compatibility lies in the stateless implementation of masked grouped GEMM kernels. In deep_gemm/legacy/m_grouped_gemm.py, functions like m_grouped_fp8_gemm_nt_masked and m_grouped_bf16_gemm_nt_masked are designed as pure compute kernels without side effects.

These kernels follow a fixed launch configuration determined solely by problem dimensions (M, N, K) and block sizes. Because they avoid dynamic host-side logic, they integrate seamlessly with torch.cuda.CUDAGraph capture mechanisms.

Eliminating Host-Side Allocation in m_grouped_gemm.py

Graph capture fails when kernels perform memory allocation or host-device synchronization during execution. The DeepGEMM masked kernels explicitly avoid these operations:

• No dynamic memory allocation: Output tensors and temporaries must be pre-allocated before graph capture.
• No host control flow: Kernel execution paths do not depend on host-side conditional logic that could diverge between capture and replay.

This design ensures that the CUDA Graph records only the kernel launch parameters and memory operations, with no unpredictable host-side behavior.

Tensor-Based Metadata Arguments

DeepGEMM passes all variable metadata as tensor arguments rather than host scalars. For masked grouped GEMMs, the masked_m and psum_m tensors are pre-allocated on the device:

# Metadata passed as device tensors, not host scalars

deep_gemm.m_grouped_bf16_gemm_nt_masked(
    a, b, d, 
    masked_m,    # Device tensor: pre-allocated before capture

    max_m_per_group
)

Because these tensors reside in device memory, the CUDA Graph can capture their state without requiring host-side updates during replay.

Avoiding Persistent Kernel Pitfalls in Triton 2.0

DeepGEMM explicitly disables Triton persistent kernels to maintain CUDA Graph compatibility. The source code in deep_gemm/legacy/m_grouped_gemm.py contains the explicit comment:

"For Triton 2.0, persistent kernel will lead to errors"

Persistent kernels, which keep threads resident across multiple loop iterations, can generate PTX code that interferes with CUDA Graph capture mechanisms. By avoiding this pattern, DeepGEMM ensures that the generated GPU code is fully compatible with graph recording and replay.

Practical Implementation: Capturing Masked Grouped GEMMs in CUDA Graphs

The following example demonstrates how to capture a masked grouped GEMM operation within a CUDA Graph using DeepGEMM:

import torch
import deep_gemm

# Pre-allocate all tensors (including masks) before capture

a = torch.randn((M, K), dtype=torch.bfloat16, device='cuda')
b = torch.randn((num_groups, N, K), dtype=torch.bfloat16, device='cuda')
d = torch.empty((M, N), dtype=torch.bfloat16, device='cuda')
masked_m = torch.randint(
    low=1, high=K, size=(num_groups,), 
    device='cuda', dtype=torch.int
)

# Warm-up execution (required before graph capture)

deep_gemm.m_grouped_bf16_gemm_nt_masked(
    a, b, d, masked_m, max_m_per_group
)

# Capture the kernel in a CUDA graph

graph = torch.cuda.CUDAGraph()
stream = torch.cuda.Stream()
with torch.cuda.stream(stream):
    graph.capture_begin()
    deep_gemm.m_grouped_bf16_gemm_nt_masked(
        a, b, d, masked_m, max_m_per_group
    )
    graph.capture_end()

# Replay the captured graph for minimal launch overhead

for _ in range(num_iters):
    graph.replay()

Because the kernel performs no host-side allocation or synchronization, the capture succeeds and subsequent replays achieve the low-overhead execution characteristic of CUDA graphs.

Verification via Test Suite

The DeepGEMM repository validates CUDA Graph compatibility through explicit benchmarking in tests/test_mega_moe.py. The test suite uses the do_bench utility with the backend='cudagraph' parameter to verify that masked grouped GEMM operations can be captured and replayed correctly:

# In tests/test_mega_moe.py

t_baseline = do_bench(
    run_baseline,
    _n_warmup=5,
    _n_repeat=1,
    backend='cudagraph',   # Triggers torch.cuda.CUDAGraph capture

    return_mode='median'
) / 1e3

When the same masked grouped GEMM kernels are used within the fused mega-MoE path (deep_gemm.fp8_fp4_mega_moe), they inherit the same graph compatibility because the entire pipeline—input casting, weight transformation, and the masked GEMM calls—respects the stateless launch model.

Summary

• Stateless kernel design: DeepGEMM implements masked grouped GEMMs in deep_gemm/legacy/m_grouped_gemm.py as stateless Triton kernels that avoid host-side allocation and dynamic control flow.
• Tensor-based metadata: All variable parameters including masked_m and psum_m are passed as pre-allocated device tensors, ensuring compatibility with CUDA Graph capture.
• Avoidance of persistent kernels: The codebase explicitly disables Triton persistent kernels (noting they "will lead to errors" in Triton 2.0) to prevent PTX generation issues that break graph capture.
• Validated implementation: The test suite in tests/test_mega_moe.py verifies functionality using backend='cudagraph', confirming that both standalone and fused mega-MoE paths support CUDA Graph replay.

Frequently Asked Questions

What makes a Triton kernel CUDA Graph compatible?

A Triton kernel is CUDA Graph compatible when it performs no host-side memory allocation, avoids host-device synchronization during execution, and does not rely on host-side control flow that could diverge between graph capture and replay. DeepGEMM achieves this by passing all metadata as device tensors and using fixed launch configurations determined solely by problem dimensions.

Why does DeepGEMM avoid persistent kernels in Triton 2.0?

Persistent kernels keep threads resident across multiple loop iterations, which can generate PTX code that interferes with CUDA Graph capture mechanisms. The DeepGEMM source code explicitly notes that "For Triton 2.0, persistent kernel will lead to errors," so the implementation uses non-persistent kernel patterns to ensure graph compatibility.

Can I use dynamic shapes with DeepGEMM masked GEMMs in CUDA Graphs?

No, CUDA Graphs require static kernel arguments and memory addresses during capture. While the underlying DeepGEMM kernels support variable dimensions via the masked_m tensor, the graph capture itself requires fixed tensor sizes and pre-allocated memory buffers. For dynamic shapes, you must capture separate graphs for each shape configuration or use CUDA Graphs with static upper bounds and masking.

How does the test suite verify CUDA Graph compatibility?

The test suite in tests/test_mega_moe.py uses the do_bench utility with the backend='cudagraph' parameter to trigger torch.cuda.CUDAGraph capture during benchmarking. This validates that the masked grouped GEMM kernels execute correctly when captured and replayed, confirming that the stateless kernel design and tensor-based metadata handling work as intended in production scenarios.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}