# 1D1D vs 1D2D GEMM Kernels in DeepGEMM: Architecture and Usage Guide

> Understand 1D1D vs 1D2D GEMM kernels in DeepGEMM. Learn when to use each for optimal FP8 GEMM performance with standard, grouped, or masked matrices.

- Repository: [DeepSeek/DeepGEMM](https://github.com/deepseek-ai/DeepGEMM)
- Tags: architecture
- Published: 2026-04-19

---

**Use 1D1D kernels for standard dense FP8 GEMM with moderate, balanced matrix dimensions, and 1D2D kernels for grouped or masked GEMM and extremely large or unbalanced shapes where a 2D thread-block grid improves GPU utilization.**

DeepGEMM, the high-performance FP8 GEMM library from DeepSeek-AI, implements two distinct kernel tiling strategies—**1D1D** and **1D2D**—to optimize different matrix multiplication workloads. Understanding the architectural distinction between these kernels enables developers to select the appropriate API and reason about performance characteristics when building inference and training pipelines.

## What Are 1D1D and 1D2D GEMM Kernels?

The names **1D1D** (also called IDID) and **1D2D** (also called ID2D) refer to how the thread-block grid is mapped across the **M** (rows) and **N** (columns) dimensions of the output matrix.

### 1D1D (Kernel1D1D) Single-Dimensional Grid

In [`csrc/jit_kernels/impls/sm90_fp8_gemm_1d1d.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/impls/sm90_fp8_gemm_1d1d.hpp), the **1D1D** kernel implements a purely single-dimensional thread-block grid. Both the **M** and **N** dimensions are tiled using a flat 1D grid where each block processes a contiguous **M×N** tile. This layout minimizes launch overhead and is represented internally by the `Kernel1D1D` enum value in the heuristics engine.

### 1D2D (Kernel1D2D) Hybrid Grid Layout

The **1D2D** kernel, defined in [`csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp), uses a hybrid approach where one dimension (typically **M**) employs 1D tiling while the other dimension (**N**) is split into a 2D thread-block grid. This creates a rectangular grid of blocks (`gridM × gridN`) capable of covering much larger **N** extents than a pure 1D grid. The selector logic in [`csrc/jit_kernels/heuristics/sm90.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/heuristics/sm90.hpp) assigns `Kernel1D2D` when problem shapes require this expanded grid topology.

## Key Architectural Differences

| Aspect | 1D1D (Kernel1D1D) | 1D2D (Kernel1D2D) |
|--------|-------------------|-------------------|
| **Thread-block grid** | 1D grid (single dimension of blocks) | Hybrid 1D × 2D grid (e.g., `gridM × gridN` where `gridN` is further split) |
| **TMA splits** | No TMA splits; requires `swizzle_a_mode == block_k` and `swizzle_b_mode == block_k` (enforced via asserts in both kernels) | Same no-split requirement, but 2D layout enables better handling of very wide **N** without extra splits |
| **Supported GEMM variants** | Simple dense FP8/FP4 GEMM, regular batched GEMM, and basic FP8‑BF16 GEMM | All dense variants **plus** **M‑grouped** and **masked** GEMM, where a per-group layout tensor (`grouped_layout`) is passed (see lines 46‑56 in [`sm90_fp8_gemm_1d2d.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/sm90_fp8_gemm_1d2d.hpp)) |
| **Memory layout** | Requires row-major output (`check_major_type_cd` forces `stride(-1) == 1`) | Same row-major requirement, but can also handle additional layout tensors for grouped or masked execution |
| **Performance sweet-spot** | Small-to-moderate **M** and **N** (generally ≤ ~2 K) where a single-dimensional grid gives low launch overhead and good occupancy | Very large **N** (or **M**) where a pure 1D grid would create too few blocks; the 2D split balances SM utilization and keeps per-block work reasonable |

## When to Use Each Kernel

DeepGEMM’s internal heuristics in [`csrc/jit_kernels/heuristics/sm90.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/heuristics/sm90.hpp) automatically select the appropriate kernel based on problem shape, data type, and GPU architecture. However, understanding the selection criteria helps developers choose the correct API:

| Scenario | Recommended Kernel | Reason |
|----------|-------------------|--------|
| **Standard dense FP8 GEMM** with moderate matrix sizes (e.g., `M, N, K ≤ 2048`) | **1D1D** (`Kernel1D1D`) | Minimal launch overhead, no extra layout tensors needed, and the heuristic picks this as the optimal config. |
| **Very wide or very tall matrices** (e.g., `N` > 4 K while `M` stays modest) | **1D2D** (`Kernel1D2D`) | The 2D tiling creates enough blocks to keep many SMs busy, improving throughput. |
| **M‑grouped GEMM** (`m_grouped_fp8_gemm_contiguous_*` or `*_masked_*`) where each expert has a different token count | **1D2D** | The kernel’s `grouped_layout` argument is used only in the 1D‑2D implementation (see lines 46‑56 in [`sm90_fp8_gemm_1d2d.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/sm90_fp8_gemm_1d2d.hpp)). |
| **Masked GEMM** (dynamic per‑group masks for inference decoding) | **1D2D** | Mask tensor is passed via `grouped_layout` – only the 1D‑2D path supports this layout. |
| **Batched GEMM** where batch size is large but each individual GEMM is small | **1D1D** (if each instance fits the 1‑D grid) or **1D2D** (if *N* of each instance is huge) – the heuristic automatically decides based on per‑instance shape. |
| **SM100 (Ada) GPUs** using the newer [`sm100_fp8_gemm_1d1d.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/sm100_fp8_gemm_1d1d.hpp) implementation – same rule applies: choose 1D1D for modest shapes, 1D2D for large *N*. | Same as SM90. |

## Implementation Details

The kernel implementations are located in the `csrc/jit_kernels/impls/` directory:

- **[`csrc/jit_kernels/impls/sm90_fp8_gemm_1d1d.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/impls/sm90_fp8_gemm_1d1d.hpp)** – Defines the `Kernel1D1D` class and launch logic for single-dimensional grid tiling.
- **[`csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp)** – Defines the `Kernel1D2D` class, supporting grouped and masked variants via the `grouped_layout` parameter (lines 46‑56).
- **[`csrc/jit_kernels/impls/sm100_fp8_gemm_1d1d.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/impls/sm100_fp8_gemm_1d1d.hpp)** and **[`sm100_fp8_gemm_1d2d.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/sm100_fp8_gemm_1d2d.hpp)** – SM100 (Ada) variants following the same architectural pattern.

The heuristic selector resides in **[`csrc/jit_kernels/heuristics/sm90.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/heuristics/sm90.hpp)**, which examines problem shapes and selects `Kernel1D1D` or `Kernel1D2D` accordingly.

## Code Examples

### Standard Dense FP8 GEMM (1D1D)

This example uses the default path for moderate matrix dimensions, mapping to [`sm90_fp8_gemm_1d1d.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/sm90_fp8_gemm_1d1d.hpp):

```python
import torch
import deep_gemm

# M, N, K are modest (≤ 2048)

M, N, K = 1024, 1024, 4096
a = torch.randn(M, K, dtype=torch.float8_e4m3fn, device='cuda')
b = torch.randn(N, K, dtype=torch.float8_e4m3fn, device='cuda').t()
sfa = torch.ones(M, 1, dtype=torch.float32, device='cuda')
sfb = torch.ones(N, 1, dtype=torch.float32, device='cuda')
d = torch.empty(M, N, dtype=torch.bfloat16, device='cuda')

# DeepGEMM automatically selects the 1D1D kernel

deep_gemm.fp8_gemm_nt(a, sfa, b, sfb, None, d,
                     m=M, n=N, k=K,
                     major_a=deep_gemm.UMMA_Major.K,
                     major_b=deep_gemm.UMMA_Major.K,
                     major_sfb=deep_gemm.UMMA_Major.MN)

```

### M-Grouped GEMM (1D2D)

For grouped expert processing where each group has a different token count, use the API that triggers [`sm90_fp8_gemm_1d2d.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/sm90_fp8_gemm_1d2d.hpp) via the `grouped_layout` parameter:

```python
import torch
import deep_gemm

num_experts = 8
total_m = 8192  # Sum of tokens across all experts

N, K = 4096, 8192

# Layout tensor indicating group boundaries

m_indices = torch.arange(num_experts, device='cuda')

a = torch.randn(total_m, K, dtype=torch.float8_e4m3fn, device='cuda')
b = torch.randn(N, K, dtype=torch.float8_e4m3fn, device='cuda').t()
sfa = torch.ones(total_m, 1, dtype=torch.float32, device='cuda')
sfb = torch.ones(N, 1, dtype=torch.float32, device='cuda')
d = torch.empty(total_m, N, dtype=torch.bfloat16, device='cuda')

# Uses 1D2D kernel internally due to grouped_layout

deep_gemm.m_grouped_fp8_gemm_nt_contiguous(a, sfa, b, sfb, d,
                                          m_indices, num_experts,
                                          m=total_m, n=N, k=K,
                                          major_a=deep_gemm.UMMA_Major.K,
                                          major_b=deep_gemm.UMMA_Major.K,
                                          major_sfb=deep_gemm.UMMA_Major.MN,
                                          compiled_dims='mk')

```

### Masked GEMM (1D2D)

For dynamic per-group masking during inference decoding, the mask tensor forces the 1D2D path:

```python

# Define per-expert token counts (including zeros for masked experts)

masked_m = torch.tensor([3, 0, 5, 2, 0, 1, 4, 0], device='cuda', dtype=torch.int32)

deep_gemm.m_grouped_fp8_gemm_nt_masked(a, sfa, b, sfb, d,
                                      masked_m, num_experts,
                                      m=total_m, n=N, k=K,
                                      expected_m=total_m,
                                      major_a=deep_gemm.UMMA_Major.K,
                                      major_b=deep_gemm.UMMA_Major.K,
                                      major_sfb=deep_gemm.UMMA_Major.MN,
                                      compiled_dims='mk')

```

## Summary

- **1D1D kernels** utilize a single-dimensional thread-block grid for both **M** and **N** dimensions, offering minimal launch overhead and optimal performance for standard dense FP8 GEMM with moderate, balanced shapes (typically **M**, **N** ≤ 2K).

- **1D2D kernels** employ a hybrid 1D × 2D grid layout that splits the **N** dimension across a two-dimensional block structure, enabling efficient processing of very large or highly unbalanced matrices while supporting advanced features like **grouped** and **masked** GEMM via the `grouped_layout` parameter.

- The heuristic selector in [`csrc/jit_kernels/heuristics/sm90.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/heuristics/sm90.hpp) automatically chooses between `Kernel1D1D` and `Kernel1D2D` based on problem geometry, though explicit API selection (e.g., `m_grouped_fp8_gemm_nt_contiguous` vs `fp8_gemm_nt`) determines layout capabilities.

## Frequently Asked Questions

### What determines whether DeepGEMM selects a 1D1D or 1D2D kernel automatically?

The selection logic resides in [`csrc/jit_kernels/heuristics/sm90.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/heuristics/sm90.hpp), which examines the problem shape, data type, and GPU architecture. For moderate matrix sizes where **M** and **N** are both small (typically under 2,048), it selects `Kernel1D1D` to minimize launch overhead. For very large **N** or when grouped/masked layouts are detected, it selects `Kernel1D2D` to ensure sufficient thread blocks occupy all SMs.

### Can I force DeepGEMM to use a specific kernel type for benchmarking?

While the library heuristics automatically select the kernel type based on the problem descriptor, you indirectly control the path through the API choice. Calling `deep_gemm.fp8_gemm_nt` for standard dense operations typically routes to `Kernel1D1D`, whereas invoking `deep_gemm.m_grouped_fp8_gemm_nt_contiguous` or `*_masked_*` variants requires the `grouped_layout` parameter, forcing the `Kernel1D2D` path in [`csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp).

### Why does grouped GEMM require the 1D2D kernel architecture?

Grouped GEMM processes batches where each group (expert) has a different token count, requiring a `grouped_layout` tensor to define boundaries. The 1D2D kernel in [`csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp) explicitly supports this tensor via its `grouped_layout` arguments (lines 46‑56), allowing the hybrid 2D grid to efficiently map irregular group sizes across the **N** dimension while maintaining coalesced memory access. The 1D1D kernel lacks these layout parameters and cannot handle per-group masking or indexing.

### Do both kernels support FP4 and BF16 output formats?

Yes, both `Kernel1D1D` and `Kernel1D2D` support FP8 input with FP4 or BF16 output. The architectural distinction lies in grid topology and supported layouts, not data type support. Both implementations require row-major output (`stride(-1) == 1` as enforced by `check_major_type_cd` in the layout utilities) and both utilize TMA (Tensor Memory Access) without splits, requiring `swizzle_a_mode == block_k` and `swizzle_b_mode == block_k` as asserted in both kernel headers.