# DeepGEMM Recipe Parameter: How It Controls Kernel Selection and Tensor Layout

> Discover how the DeepGEMM recipe parameter controls CUDA kernel selection and tensor layout for FP8/FP4 matrix multiplication. Optimize your deep learning performance.

- Repository: [DeepSeek/DeepGEMM](https://github.com/deepseek-ai/DeepGEMM)
- Tags: internals
- Published: 2026-04-19

---

**The `recipe` parameter in DeepGEMM is a tuple-based layout descriptor that determines both how scale-factor tensors are reshaped and which specific CUDA kernel variant is launched for FP8/FP4 matrix multiplication.**

In the `deepseek-ai/DeepGEMM` repository, the **recipe parameter** bridges high-level Python calls with low-level GPU kernels. It encodes granularity constraints that dictate tensor layout transformations and kernel routing decisions. Understanding this parameter is essential for optimizing mixed-precision GEMM operations on NVIDIA Hopper and Ada architectures.

## What Is the Recipe Parameter?

The `recipe` is a **layout-granularity descriptor** passed to functions like `fp8_fp4_gemm_nt`. It tells DeepGEMM how to transform scale-factor (SF) tensors before launching the underlying CUDA kernel.

### Structure and Tuple Types

DeepGEMM accepts two tuple formats for the recipe parameter:

| Tuple Format | Components | Usage Context |
|--------------|------------|---------------|
| `Tuple[int, int, int]` | `(gran_mn_a, gran_mn_b, gran_k)` | When different MN granularities are needed for the A and B side scale-factor tensors |
| `Tuple[int, int]` | `(gran_mn, gran_k)` | When a single recipe applies to both sides (e.g., when `is_sfa` is `None`) |

- **`gran_mn`** controls the **MN-major granularity** (how many rows are packed together in the scale-factor tensor)
- **`gran_k`** controls the **K-major granularity** (how many columns are packed)

### Unpacking Logic in C++

In [`csrc/apis/layout.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/apis/layout.hpp), the `transform_sf_into_required_layout` function unpacks the recipe tuple using `std::variant` logic:

```cpp
// csrc/apis/layout.hpp
if (auto p = std::get_if<std::tuple<int, int, int>>(&recipe)) {
    // recipe has three components
    gran_mn = is_sfa.value() ? std::get<0>(*p)   // side A -> first element
                              : std::get<1>(*p); // side B -> second element
    gran_k = std::get<2>(*p);
} else if (auto p = std::get_if<std::tuple<int, int>>(&recipe)) {
    // recipe has two components (same for both sides)
    std::tie(gran_mn, gran_k) = *p;
}

```

This unpacking determines the physical memory layout of the scale-factor tensors for TMA (Tensor Memory Accelerator) aligned access.

## How Recipe Affects Kernel Selection

The recipe parameter does not merely reshape tensors—it acts as a **kernel selector** that routes execution to specific hand-optimized CUDA implementations.

### Mapping Recipes to Kernel Variants

Different recipe values map to distinct kernel families optimized for specific precision and hardware targets:

| Recipe Value | Target Architecture | Kernel Type | Precision Context |
|--------------|---------------------|-------------|-------------------|
| `(1, 128, 128)` | SM90 (Hopper/H100) | **2-D MN-K kernels** | Default FP32 scale-factor kernels |
| `(1, 1, 128)` | SM100 (Ada) | **1-D-1-D kernels** | FP8/FP4 specialized kernels |

When you pass `(1, 1, 128)`, the dispatcher in [`csrc/apis/mega.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/apis/mega.hpp) validates the recipe before selecting the 1-D-1-D kernel implementation:

```cpp
// csrc/apis/mega.hpp (excerpt)
DG_HOST_ASSERT(std::get<0>(recipe) == 1 && std::get<1>(recipe) == 1);
const int gran_k = std::get<2>(recipe);

```

If the tuple does not match the expected pattern for the selected kernel family, the dispatcher aborts with `DG_HOST_UNREACHABLE`.

### Default Recipe Derivation

When the `recipe` parameter is omitted, DeepGEMM automatically selects an optimal configuration based on the detected GPU architecture. The logic resides in [`csrc/utils/layout.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/utils/layout.hpp):

```cpp
// csrc/utils/layout.hpp
if (arch_major == 9) {               // SM90 (H100)
    return {1, 128, 128};
} else if (arch_major == 10) {       // SM100 (Ada)
    return sfb_dtype == torch::kFloat ?
           std::make_tuple(1, 128, 128) :   // legacy FP32 kernels
           std::make_tuple(1, 1, 128);      // 1D‑1D kernels for FP8/FP4
}

```

This ensures that FP8/FP4 operations on Ada GPUs automatically use the specialized 1-D-1-D kernels, while Hopper GPUs use the 2-D MN-K kernels.

## Practical Code Examples

### Explicit Recipe for 1D-1D FP8 Kernels

To force the use of 1-D-1D kernels for FP8 matrix multiplication on compatible hardware, pass a three-element tuple with `gran_mn` set to 1 for both sides:

```python
import torch
import deep_gemm

a = torch.randn(256, 64, dtype=torch.float8_e4m3fn, device='cuda')
b = torch.randn(64, 128, dtype=torch.float8_e4m3fn, device='cuda')
d = torch.empty(256, 128, dtype=torch.float8_e4m3fn, device='cuda')

# Force the 1D-1D kernel (gran_mn_a = 1, gran_mn_b = 1, gran_k = 128)

deep_gemm.fp8_fp4_gemm_nt(a, b, d, recipe=(1, 1, 128))

```

This call flows through [`deep_gemm/mega/__init__.py`](https://github.com/deepseek-ai/DeepGEMM/blob/main/deep_gemm/mega/__init__.py), where the tuple is validated and forwarded to the C++ kernel dispatcher.

### Auto-Selecting the Optimal Recipe

For most use cases, omit the `recipe` parameter to let DeepGEMM select the appropriate kernel based on the GPU architecture and data types:

```python
import torch
import deep_gemm

a = torch.randn(256, 64, dtype=torch.float, device='cuda')
b = torch.randn(64, 128, dtype=torch.float, device='cuda')
d = torch.empty(256, 128, dtype=torch.float, device='cuda')

deep_gemm.fp8_fp4_gemm_nt(a, b, d)   # No recipe argument

```

The library invokes `get_default_recipe` from [`csrc/utils/layout.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/utils/layout.hpp) to determine whether to use `(1, 128, 128)` for Hopper or `(1, 1, 128)` for Ada FP8/FP4 operations.

## Implementation Details

The recipe parameter is handled across several key source files:

| Path | Function | Relevance |
|------|----------|-----------|
| [`csrc/utils/layout.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/utils/layout.hpp) | `get_default_recipe` | Maps GPU architecture (SM90 vs SM100) to default recipe tuples【/cache/repos/github.com/deepseek-ai/DeepGEMM/main/csrc/utils/layout.hpp#L65-L76】 |
| [`csrc/apis/layout.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/apis/layout.hpp) | `transform_sf_into_required_layout` | Unpacks the recipe tuple and applies scale-factor tensor transformations【/cache/repos/github.com/deepseek-ai/DeepGEMM/main/csrc/apis/layout.hpp#L23-L35】 |
| [`csrc/apis/mega.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/apis/mega.hpp) | Kernel dispatcher | Validates recipe components and selects 2-D or 1-D-1-D kernel variants |
| [`deep_gemm/mega/__init__.py`](https://github.com/deepseek-ai/DeepGEMM/blob/main/deep_gemm/mega/__init__.py) | `fp8_fp4_gemm_nt` | Python entry point that forwards recipe to C++ layer【/cache/repos/github.com/deepseek-ai/DeepGEMM/main/deep_gemm/mega/__init__.py#L114-L126】 |
| [`tests/generators.py`](https://github.com/deepseek-ai/DeepGEMM/blob/main/tests/generators.py) | Test generators | Provides recipe tuples for unit test matrices【/cache/repos/github.com/deepseek-ai/DeepGEMM/main/tests/generators.py#L55-L62】 |
| [`tests/test_fp8_fp4.py`](https://github.com/deepseek-ai/DeepGEMM/blob/main/tests/test_fp8_fp4.py) | Unit tests | Demonstrates both explicit and default recipe handling【/cache/repos/github.com/deepseek-ai/DeepGEMM/main/tests/test_fp8_fp4.py#L32-L41】 |

## Summary

- The **recipe parameter** is a tuple descriptor `(gran_mn_a, gran_mn_b, gran_k)` or `(gran_mn, gran_k)` that defines scale-factor tensor granularity.
- It controls **both** the physical layout transformation of SF tensors and the **kernel selection** (2-D MN-K vs 1-D-1D variants).
- **Hopper (SM90)** defaults to `(1, 128, 128)` for standard FP32 kernels, while **Ada (SM100)** uses `(1, 1, 128)` for FP8/FP4 1-D-1D kernels.
- Invalid recipe tuples trigger assertion failures in the C++ dispatcher via `DG_HOST_ASSERT` and `DG_HOST_UNREACHABLE`.

## Frequently Asked Questions

### What happens if I provide an invalid recipe tuple?

DeepGEMM validates the recipe components in the C++ dispatcher before kernel launch. If the tuple does not match the expected pattern for the selected kernel family—for example, providing `(2, 128, 128)` when the kernel requires `gran_mn` to be 1—the code aborts with `DG_HOST_UNREACHABLE` or fails a `DG_HOST_ASSERT` check. This ensures that incompatible tensor layouts never reach the GPU kernel.

### Can I use the same recipe for different GPU architectures?

No, recipes are architecture-specific. The `get_default_recipe` function in [`csrc/utils/layout.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/utils/layout.hpp) maps SM90 (Hopper) to `(1, 128, 128)` and SM100 (Ada) to either `(1, 128, 128)` for legacy FP32 or `(1, 1, 128)` for FP8/FP4. Using a Hopper-optimized recipe on Ada hardware (or vice versa) will either trigger assertion failures or produce incorrect results due to mismatched TMA alignment expectations.

### How does the recipe parameter relate to TMA alignment?

The `gran_mn` and `gran_k` values in the recipe dictate the physical layout of scale-factor tensors to ensure they meet Tensor Memory Accelerator (TMA) alignment requirements. The `transform_sf_into_required_layout` function in [`csrc/apis/layout.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/apis/layout.hpp) uses these granularity values to reshape SF tensors into MN-major or K-major layouts that the TMA hardware can efficiently stream to the tensor cores. Incorrect granularity results in misaligned memory access patterns that degrade performance or cause runtime errors.

### Why does FP8/FP4 use (1, 1, 128) instead of (1, 128, 128)?

The `(1, 1, 128)` recipe selects **1-D-1D kernels** that are specifically optimized for FP8 and FP4 precision on SM100 (Ada) architectures. Unlike the 2-D kernels used for FP32, these 1-D-1D variants require both the A and B side scale-factor tensors to have row-major granularity of 1 (`gran_mn_a = 1`, `gran_mn_b = 1`). The third value (`128`) maintains the K-granularity for column alignment. This specialized layout eliminates unnecessary packing overhead for sub-byte quantized types while preserving TMA efficiency.