What Is Programmatic Dependent Launch (PDL) in DeepGEMM and How to Enable It

Programmatic Dependent Launch (PDL) is a runtime flag in DeepGEMM that allows kernels to dynamically compute launch parameters—such as grid size and shared-memory layout—based on values only known at Python runtime, and you can toggle it using deep_gemm.set_pdl(True).

DeepGEMM, the open-source FP8 GEMM library by DeepSeek, includes a sophisticated just-in-time (JIT) compilation system that supports runtime kernel configuration. PDL enables specific kernels, particularly those handling variable-length sequences like Mega-MoE layers, to adapt their CUDA launch configurations based on actual input dimensions rather than static constants.

Understanding Programmatic Dependent Launch (PDL)

PDL is a boolean flag stored in the C++ DeviceRuntime singleton that controls whether the JIT launch infrastructure may compute launch-time parameters dynamically. By default, PDL is disabled (false), meaning kernels launch with the static arguments supplied by the caller.

When enabled, the system allows kernels to inspect runtime values—such as the actual number of tokens in a batch—and adjust their grid dimensions or shared-memory allocation accordingly. This is particularly valuable for workloads with variable sequence lengths where optimal launch parameters cannot be determined ahead of time.

How PDL Works Under the Hood

The C++ Runtime Flag

The core state lives in csrc/jit/device_runtime.hpp at lines 27-33, where the DeviceRuntime class maintains a boolean member enable_pdl initialized to false. This singleton provides thread-safe getter and setter methods that the Python bindings invoke.

// csrc/jit/device_runtime.hpp (simplified)
class DeviceRuntime {
private:
    bool enable_pdl = false;  // Default: disabled
public:
    void set_pdl(bool value) { enable_pdl = value; }
    bool get_pdl() const { return enable_pdl; }
};

Kernel Launch Integration

During each kernel launch, the JIT runtime reads this flag and injects it into the launch configuration. In csrc/jit/kernel_runtime.hpp at lines 44-48, the system overwrites launch_args.enable_pdl with the current runtime value before constructing the final kernel launch parameters.

// csrc/jit/kernel_runtime.hpp (conceptual)
void launch_kernel(...) {
    LaunchArgs args;
    args.enable_pdl = device_runtime->get_pdl();  // Read current state
    // ... construct launch config based on args
}

Python API Bindings

The Python interface is exposed through deep_gemm/__init__.py at lines 24-26, which imports the setter and getter functions from the compiled extension module. These functions are bound via Pybind11 in csrc/apis/runtime.hpp, creating a direct bridge between Python calls and the C++ singleton state.

# deep_gemm/__init__.py

from deep_gemm._C import set_pdl, get_pdl

Enabling and Using PDL in Practice

You can toggle PDL at any point in your Python program using the provided API. The following example demonstrates checking the default state, enabling PDL for a dynamic kernel launch, and then disabling it.

import torch
import deep_gemm

# 1. Check the default PDL state (should be False)

print("PDL enabled?", deep_gemm.get_pdl())  # → False

# 2. Enable Programmatic Dependent Launch globally

deep_gemm.set_pdl(True)
assert deep_gemm.get_pdl() is True

# 3. Launch a kernel that adapts to runtime parameters

# This example uses a Mega-MoE kernel that selects grid dimensions

# based on the actual token count in the input tensor.

x = torch.randn(1024, 64, dtype=torch.bfloat16, device="cuda")
y = torch.empty_like(x)

# The kernel inspects x.shape[0] at launch time to determine optimal configuration

deep_gemm.fp8_fp4_mega_moe(
    y,
    x,
    torch.randn(64, 64, dtype=torch.bfloat16, device="cuda"),  # Layer 1 weights

    torch.randn(64, 64, dtype=torch.bfloat16, device="cuda"),  # Layer 2 weights

    deep_gemm.testing.utils.get_symm_buffer_for_mega_moe()       # Shared buffer

)

# 4. Disable PDL when static launch parameters are preferred

deep_gemm.set_pdl(False)
print("PDL enabled now?", deep_gemm.get_pdl())  # → False

When to Use Programmatic Dependent Launch

PDL is particularly valuable for dynamic workloads where input dimensions vary between iterations. The primary use case in DeepGEMM is the fp8_fp4_mega_moe kernel, which processes variable-length token sequences in Mixture-of-Experts (MoE) layers.

When PDL is enabled, this kernel can inspect the actual number of tokens in the current batch and adjust its grid size accordingly, potentially improving occupancy and throughput compared to a static launch configured for the maximum possible sequence length.

For kernels with fixed, known dimensions—such as standard GEMM operations with constant matrix sizes—disabling PDL eliminates the runtime check overhead and uses the pre-computed static launch configuration.

Summary

• Programmatic Dependent Launch (PDL) is a runtime flag in DeepGEMM that allows JIT-compiled kernels to compute launch parameters dynamically based on Python runtime values.
• The flag defaults to false (disabled) and is stored in the C++ DeviceRuntime singleton at csrc/jit/device_runtime.hpp.
• Enable PDL in Python using deep_gemm.set_pdl(True) and check its state with deep_gemm.get_pdl(), both exposed via deep_gemm/__init__.py.
• When enabled, the kernel launch path in csrc/jit/kernel_runtime.hpp reads the flag and may adjust grid dimensions, block sizes, or shared-memory layout based on runtime data.
• PDL is particularly useful for variable-length workloads like Mega-MoE kernels, while static configurations may be preferred for fixed-dimension operations to minimize overhead.

Frequently Asked Questions

What is the default state of PDL in DeepGEMM?

By default, Programmatic Dependent Launch is disabled (false). The DeviceRuntime singleton initializes enable_pdl to false in csrc/jit/device_runtime.hpp, meaning kernels use static launch arguments unless you explicitly enable PDL via deep_gemm.set_pdl(True).

How does PDL affect kernel performance?

PDL can improve performance for dynamic workloads by allowing kernels to optimize their launch configuration for the actual input size rather than a worst-case static assumption. However, for fixed-dimension kernels, disabling PDL removes the runtime check overhead in csrc/jit/kernel_runtime.hpp, potentially reducing launch latency.

Can I toggle PDL on and off during a single Python session?

Yes, PDL is designed to be toggled at runtime. You can call deep_gemm.set_pdl(True) before launching dynamic kernels and deep_gemm.set_pdl(False) before static operations. The change takes effect immediately because the JIT launch path in csrc/jit/kernel_runtime.hpp queries the DeviceRuntime singleton during every kernel invocation.

Which DeepGEMM kernels support Programmatic Dependent Launch?

PDL is primarily utilized by dynamic-shape kernels such as fp8_fp4_mega_moe, which handles variable token counts in Mixture-of-Experts layers. According to the implementation in csrc/jit/kernel_runtime.hpp, any JIT-compiled kernel can theoretically access the PDL flag, but the dynamic launch path is specifically beneficial for kernels that need to adapt their grid size or shared-memory allocation based on runtime data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}