DeepGEMM Recipe Parameter: How It Controls Kernel Selection and Tensor Layout

The recipe parameter in DeepGEMM is a tuple-based layout descriptor that determines both how scale-factor tensors are reshaped and which specific CUDA kernel variant is launched for FP8/FP4 matrix multiplication.

In the deepseek-ai/DeepGEMM repository, the recipe parameter bridges high-level Python calls with low-level GPU kernels. It encodes granularity constraints that dictate tensor layout transformations and kernel routing decisions. Understanding this parameter is essential for optimizing mixed-precision GEMM operations on NVIDIA Hopper and Ada architectures.

What Is the Recipe Parameter?

The recipe is a layout-granularity descriptor passed to functions like fp8_fp4_gemm_nt. It tells DeepGEMM how to transform scale-factor (SF) tensors before launching the underlying CUDA kernel.

Structure and Tuple Types

DeepGEMM accepts two tuple formats for the recipe parameter:

Tuple Format	Components	Usage Context
Tuple[int, int, int]	(gran_mn_a, gran_mn_b, gran_k)	When different MN granularities are needed for the A and B side scale-factor tensors
Tuple[int, int]	(gran_mn, gran_k)	When a single recipe applies to both sides (e.g., when is_sfa is None)

• gran_mn controls the MN-major granularity (how many rows are packed together in the scale-factor tensor)
• gran_k controls the K-major granularity (how many columns are packed)

Unpacking Logic in C++

In csrc/apis/layout.hpp, the transform_sf_into_required_layout function unpacks the recipe tuple using std::variant logic:

// csrc/apis/layout.hpp
if (auto p = std::get_if<std::tuple<int, int, int>>(&recipe)) {
    // recipe has three components
    gran_mn = is_sfa.value() ? std::get<0>(*p)   // side A -> first element
                              : std::get<1>(*p); // side B -> second element
    gran_k = std::get<2>(*p);
} else if (auto p = std::get_if<std::tuple<int, int>>(&recipe)) {
    // recipe has two components (same for both sides)
    std::tie(gran_mn, gran_k) = *p;
}

This unpacking determines the physical memory layout of the scale-factor tensors for TMA (Tensor Memory Accelerator) aligned access.

How Recipe Affects Kernel Selection

The recipe parameter does not merely reshape tensors—it acts as a kernel selector that routes execution to specific hand-optimized CUDA implementations.

Mapping Recipes to Kernel Variants

Different recipe values map to distinct kernel families optimized for specific precision and hardware targets:

Recipe Value	Target Architecture	Kernel Type	Precision Context
(1, 128, 128)	SM90 (Hopper/H100)	2-D MN-K kernels	Default FP32 scale-factor kernels
(1, 1, 128)	SM100 (Ada)	1-D-1-D kernels	FP8/FP4 specialized kernels

When you pass (1, 1, 128), the dispatcher in csrc/apis/mega.hpp validates the recipe before selecting the 1-D-1-D kernel implementation:

// csrc/apis/mega.hpp (excerpt)
DG_HOST_ASSERT(std::get<0>(recipe) == 1 && std::get<1>(recipe) == 1);
const int gran_k = std::get<2>(recipe);

If the tuple does not match the expected pattern for the selected kernel family, the dispatcher aborts with DG_HOST_UNREACHABLE.

Default Recipe Derivation

When the recipe parameter is omitted, DeepGEMM automatically selects an optimal configuration based on the detected GPU architecture. The logic resides in csrc/utils/layout.hpp:

// csrc/utils/layout.hpp
if (arch_major == 9) {               // SM90 (H100)
    return {1, 128, 128};
} else if (arch_major == 10) {       // SM100 (Ada)
    return sfb_dtype == torch::kFloat ?
           std::make_tuple(1, 128, 128) :   // legacy FP32 kernels
           std::make_tuple(1, 1, 128);      // 1D‑1D kernels for FP8/FP4
}

This ensures that FP8/FP4 operations on Ada GPUs automatically use the specialized 1-D-1-D kernels, while Hopper GPUs use the 2-D MN-K kernels.

Practical Code Examples

Explicit Recipe for 1D-1D FP8 Kernels

To force the use of 1-D-1D kernels for FP8 matrix multiplication on compatible hardware, pass a three-element tuple with gran_mn set to 1 for both sides:

import torch
import deep_gemm

a = torch.randn(256, 64, dtype=torch.float8_e4m3fn, device='cuda')
b = torch.randn(64, 128, dtype=torch.float8_e4m3fn, device='cuda')
d = torch.empty(256, 128, dtype=torch.float8_e4m3fn, device='cuda')

# Force the 1D-1D kernel (gran_mn_a = 1, gran_mn_b = 1, gran_k = 128)

deep_gemm.fp8_fp4_gemm_nt(a, b, d, recipe=(1, 1, 128))

This call flows through deep_gemm/mega/__init__.py, where the tuple is validated and forwarded to the C++ kernel dispatcher.

Auto-Selecting the Optimal Recipe

For most use cases, omit the recipe parameter to let DeepGEMM select the appropriate kernel based on the GPU architecture and data types:

import torch
import deep_gemm

a = torch.randn(256, 64, dtype=torch.float, device='cuda')
b = torch.randn(64, 128, dtype=torch.float, device='cuda')
d = torch.empty(256, 128, dtype=torch.float, device='cuda')

deep_gemm.fp8_fp4_gemm_nt(a, b, d)   # No recipe argument

The library invokes get_default_recipe from csrc/utils/layout.hpp to determine whether to use (1, 128, 128) for Hopper or (1, 1, 128) for Ada FP8/FP4 operations.

Implementation Details

The recipe parameter is handled across several key source files:

Path	Function	Relevance
csrc/utils/layout.hpp	get_default_recipe	Maps GPU architecture (SM90 vs SM100) to default recipe tuples【/cache/repos/github.com/deepseek-ai/DeepGEMM/main/csrc/utils/layout.hpp#L65-L76】
csrc/apis/layout.hpp	transform_sf_into_required_layout	Unpacks the recipe tuple and applies scale-factor tensor transformations【/cache/repos/github.com/deepseek-ai/DeepGEMM/main/csrc/apis/layout.hpp#L23-L35】
csrc/apis/mega.hpp	Kernel dispatcher	Validates recipe components and selects 2-D or 1-D-1-D kernel variants
deep_gemm/mega/__init__.py	fp8_fp4_gemm_nt	Python entry point that forwards recipe to C++ layer【/cache/repos/github.com/deepseek-ai/DeepGEMM/main/deep_gemm/mega/init.py#L114-L126】
tests/generators.py	Test generators	Provides recipe tuples for unit test matrices【/cache/repos/github.com/deepseek-ai/DeepGEMM/main/tests/generators.py#L55-L62】
tests/test_fp8_fp4.py	Unit tests	Demonstrates both explicit and default recipe handling【/cache/repos/github.com/deepseek-ai/DeepGEMM/main/tests/test_fp8_fp4.py#L32-L41】

Summary

• The recipe parameter is a tuple descriptor (gran_mn_a, gran_mn_b, gran_k) or (gran_mn, gran_k) that defines scale-factor tensor granularity.
• It controls both the physical layout transformation of SF tensors and the kernel selection (2-D MN-K vs 1-D-1D variants).
• Hopper (SM90) defaults to (1, 128, 128) for standard FP32 kernels, while Ada (SM100) uses (1, 1, 128) for FP8/FP4 1-D-1D kernels.
• Invalid recipe tuples trigger assertion failures in the C++ dispatcher via DG_HOST_ASSERT and DG_HOST_UNREACHABLE.

Frequently Asked Questions

What happens if I provide an invalid recipe tuple?

DeepGEMM validates the recipe components in the C++ dispatcher before kernel launch. If the tuple does not match the expected pattern for the selected kernel family—for example, providing (2, 128, 128) when the kernel requires gran_mn to be 1—the code aborts with DG_HOST_UNREACHABLE or fails a DG_HOST_ASSERT check. This ensures that incompatible tensor layouts never reach the GPU kernel.

Can I use the same recipe for different GPU architectures?

No, recipes are architecture-specific. The get_default_recipe function in csrc/utils/layout.hpp maps SM90 (Hopper) to (1, 128, 128) and SM100 (Ada) to either (1, 128, 128) for legacy FP32 or (1, 1, 128) for FP8/FP4. Using a Hopper-optimized recipe on Ada hardware (or vice versa) will either trigger assertion failures or produce incorrect results due to mismatched TMA alignment expectations.

How does the recipe parameter relate to TMA alignment?

The gran_mn and gran_k values in the recipe dictate the physical layout of scale-factor tensors to ensure they meet Tensor Memory Accelerator (TMA) alignment requirements. The transform_sf_into_required_layout function in csrc/apis/layout.hpp uses these granularity values to reshape SF tensors into MN-major or K-major layouts that the TMA hardware can efficiently stream to the tensor cores. Incorrect granularity results in misaligned memory access patterns that degrade performance or cause runtime errors.

Why does FP8/FP4 use (1, 1, 128) instead of (1, 128, 128)?

The (1, 1, 128) recipe selects 1-D-1D kernels that are specifically optimized for FP8 and FP4 precision on SM100 (Ada) architectures. Unlike the 2-D kernels used for FP32, these 1-D-1D variants require both the A and B side scale-factor tensors to have row-major granularity of 1 (gran_mn_a = 1, gran_mn_b = 1). The third value (128) maintains the K-granularity for column alignment. This specialized layout eliminates unnecessary packing overhead for sub-byte quantized types while preserving TMA efficiency.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}