1D1D vs 1D2D GEMM Kernels in DeepGEMM: Architecture and Usage Guide

Use 1D1D kernels for standard dense FP8 GEMM with moderate, balanced matrix dimensions, and 1D2D kernels for grouped or masked GEMM and extremely large or unbalanced shapes where a 2D thread-block grid improves GPU utilization.

DeepGEMM, the high-performance FP8 GEMM library from DeepSeek-AI, implements two distinct kernel tiling strategies—1D1D and 1D2D—to optimize different matrix multiplication workloads. Understanding the architectural distinction between these kernels enables developers to select the appropriate API and reason about performance characteristics when building inference and training pipelines.

What Are 1D1D and 1D2D GEMM Kernels?

The names 1D1D (also called IDID) and 1D2D (also called ID2D) refer to how the thread-block grid is mapped across the M (rows) and N (columns) dimensions of the output matrix.

1D1D (Kernel1D1D) Single-Dimensional Grid

In csrc/jit_kernels/impls/sm90_fp8_gemm_1d1d.hpp, the 1D1D kernel implements a purely single-dimensional thread-block grid. Both the M and N dimensions are tiled using a flat 1D grid where each block processes a contiguous M×N tile. This layout minimizes launch overhead and is represented internally by the Kernel1D1D enum value in the heuristics engine.

1D2D (Kernel1D2D) Hybrid Grid Layout

The 1D2D kernel, defined in csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp, uses a hybrid approach where one dimension (typically M) employs 1D tiling while the other dimension (N) is split into a 2D thread-block grid. This creates a rectangular grid of blocks (gridM × gridN) capable of covering much larger N extents than a pure 1D grid. The selector logic in csrc/jit_kernels/heuristics/sm90.hpp assigns Kernel1D2D when problem shapes require this expanded grid topology.

Key Architectural Differences

Aspect	1D1D (Kernel1D1D)	1D2D (Kernel1D2D)
Thread-block grid	1D grid (single dimension of blocks)	Hybrid 1D × 2D grid (e.g., gridM × gridN where gridN is further split)
TMA splits	No TMA splits; requires swizzle_a_mode == block_k and swizzle_b_mode == block_k (enforced via asserts in both kernels)	Same no-split requirement, but 2D layout enables better handling of very wide N without extra splits
Supported GEMM variants	Simple dense FP8/FP4 GEMM, regular batched GEMM, and basic FP8‑BF16 GEMM	All dense variants plus M‑grouped and masked GEMM, where a per-group layout tensor (grouped_layout) is passed (see lines 46‑56 in sm90_fp8_gemm_1d2d.hpp)
Memory layout	Requires row-major output (check_major_type_cd forces stride(-1) == 1)	Same row-major requirement, but can also handle additional layout tensors for grouped or masked execution
Performance sweet-spot	Small-to-moderate M and N (generally ≤ ~2 K) where a single-dimensional grid gives low launch overhead and good occupancy	Very large N (or M) where a pure 1D grid would create too few blocks; the 2D split balances SM utilization and keeps per-block work reasonable

When to Use Each Kernel

DeepGEMM’s internal heuristics in csrc/jit_kernels/heuristics/sm90.hpp automatically select the appropriate kernel based on problem shape, data type, and GPU architecture. However, understanding the selection criteria helps developers choose the correct API:

Scenario	Recommended Kernel	Reason
Standard dense FP8 GEMM with moderate matrix sizes (e.g., M, N, K ≤ 2048)	1D1D (Kernel1D1D)	Minimal launch overhead, no extra layout tensors needed, and the heuristic picks this as the optimal config.
Very wide or very tall matrices (e.g., N > 4 K while M stays modest)	1D2D (Kernel1D2D)	The 2D tiling creates enough blocks to keep many SMs busy, improving throughput.
M‑grouped GEMM (m_grouped_fp8_gemm_contiguous_* or *_masked_*) where each expert has a different token count	1D2D	The kernel’s grouped_layout argument is used only in the 1D‑2D implementation (see lines 46‑56 in sm90_fp8_gemm_1d2d.hpp).
Masked GEMM (dynamic per‑group masks for inference decoding)	1D2D	Mask tensor is passed via grouped_layout – only the 1D‑2D path supports this layout.
Batched GEMM where batch size is large but each individual GEMM is small	1D1D (if each instance fits the 1‑D grid) or 1D2D (if N of each instance is huge) – the heuristic automatically decides based on per‑instance shape.
SM100 (Ada) GPUs using the newer sm100_fp8_gemm_1d1d.hpp implementation – same rule applies: choose 1D1D for modest shapes, 1D2D for large N.	Same as SM90.

Implementation Details

The kernel implementations are located in the csrc/jit_kernels/impls/ directory:

• csrc/jit_kernels/impls/sm90_fp8_gemm_1d1d.hpp – Defines the Kernel1D1D class and launch logic for single-dimensional grid tiling.
• csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp – Defines the Kernel1D2D class, supporting grouped and masked variants via the grouped_layout parameter (lines 46‑56).
• csrc/jit_kernels/impls/sm100_fp8_gemm_1d1d.hpp and sm100_fp8_gemm_1d2d.hpp – SM100 (Ada) variants following the same architectural pattern.

The heuristic selector resides in csrc/jit_kernels/heuristics/sm90.hpp, which examines problem shapes and selects Kernel1D1D or Kernel1D2D accordingly.

Code Examples

Standard Dense FP8 GEMM (1D1D)

This example uses the default path for moderate matrix dimensions, mapping to sm90_fp8_gemm_1d1d.hpp:

import torch
import deep_gemm

# M, N, K are modest (≤ 2048)

M, N, K = 1024, 1024, 4096
a = torch.randn(M, K, dtype=torch.float8_e4m3fn, device='cuda')
b = torch.randn(N, K, dtype=torch.float8_e4m3fn, device='cuda').t()
sfa = torch.ones(M, 1, dtype=torch.float32, device='cuda')
sfb = torch.ones(N, 1, dtype=torch.float32, device='cuda')
d = torch.empty(M, N, dtype=torch.bfloat16, device='cuda')

# DeepGEMM automatically selects the 1D1D kernel

deep_gemm.fp8_gemm_nt(a, sfa, b, sfb, None, d,
                     m=M, n=N, k=K,
                     major_a=deep_gemm.UMMA_Major.K,
                     major_b=deep_gemm.UMMA_Major.K,
                     major_sfb=deep_gemm.UMMA_Major.MN)

M-Grouped GEMM (1D2D)

For grouped expert processing where each group has a different token count, use the API that triggers sm90_fp8_gemm_1d2d.hpp via the grouped_layout parameter:

import torch
import deep_gemm

num_experts = 8
total_m = 8192  # Sum of tokens across all experts

N, K = 4096, 8192

# Layout tensor indicating group boundaries

m_indices = torch.arange(num_experts, device='cuda')

a = torch.randn(total_m, K, dtype=torch.float8_e4m3fn, device='cuda')
b = torch.randn(N, K, dtype=torch.float8_e4m3fn, device='cuda').t()
sfa = torch.ones(total_m, 1, dtype=torch.float32, device='cuda')
sfb = torch.ones(N, 1, dtype=torch.float32, device='cuda')
d = torch.empty(total_m, N, dtype=torch.bfloat16, device='cuda')

# Uses 1D2D kernel internally due to grouped_layout

deep_gemm.m_grouped_fp8_gemm_nt_contiguous(a, sfa, b, sfb, d,
                                          m_indices, num_experts,
                                          m=total_m, n=N, k=K,
                                          major_a=deep_gemm.UMMA_Major.K,
                                          major_b=deep_gemm.UMMA_Major.K,
                                          major_sfb=deep_gemm.UMMA_Major.MN,
                                          compiled_dims='mk')

Masked GEMM (1D2D)

For dynamic per-group masking during inference decoding, the mask tensor forces the 1D2D path:

# Define per-expert token counts (including zeros for masked experts)

masked_m = torch.tensor([3, 0, 5, 2, 0, 1, 4, 0], device='cuda', dtype=torch.int32)

deep_gemm.m_grouped_fp8_gemm_nt_masked(a, sfa, b, sfb, d,
                                      masked_m, num_experts,
                                      m=total_m, n=N, k=K,
                                      expected_m=total_m,
                                      major_a=deep_gemm.UMMA_Major.K,
                                      major_b=deep_gemm.UMMA_Major.K,
                                      major_sfb=deep_gemm.UMMA_Major.MN,
                                      compiled_dims='mk')

Summary

• 1D1D kernels utilize a single-dimensional thread-block grid for both M and N dimensions, offering minimal launch overhead and optimal performance for standard dense FP8 GEMM with moderate, balanced shapes (typically M, N ≤ 2K).

• 1D2D kernels employ a hybrid 1D × 2D grid layout that splits the N dimension across a two-dimensional block structure, enabling efficient processing of very large or highly unbalanced matrices while supporting advanced features like grouped and masked GEMM via the grouped_layout parameter.

• The heuristic selector in csrc/jit_kernels/heuristics/sm90.hpp automatically chooses between Kernel1D1D and Kernel1D2D based on problem geometry, though explicit API selection (e.g., m_grouped_fp8_gemm_nt_contiguous vs fp8_gemm_nt) determines layout capabilities.

Frequently Asked Questions

What determines whether DeepGEMM selects a 1D1D or 1D2D kernel automatically?

The selection logic resides in csrc/jit_kernels/heuristics/sm90.hpp, which examines the problem shape, data type, and GPU architecture. For moderate matrix sizes where M and N are both small (typically under 2,048), it selects Kernel1D1D to minimize launch overhead. For very large N or when grouped/masked layouts are detected, it selects Kernel1D2D to ensure sufficient thread blocks occupy all SMs.

Can I force DeepGEMM to use a specific kernel type for benchmarking?

While the library heuristics automatically select the kernel type based on the problem descriptor, you indirectly control the path through the API choice. Calling deep_gemm.fp8_gemm_nt for standard dense operations typically routes to Kernel1D1D, whereas invoking deep_gemm.m_grouped_fp8_gemm_nt_contiguous or *_masked_* variants requires the grouped_layout parameter, forcing the Kernel1D2D path in csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp.

Why does grouped GEMM require the 1D2D kernel architecture?

Grouped GEMM processes batches where each group (expert) has a different token count, requiring a grouped_layout tensor to define boundaries. The 1D2D kernel in csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp explicitly supports this tensor via its grouped_layout arguments (lines 46‑56), allowing the hybrid 2D grid to efficiently map irregular group sizes across the N dimension while maintaining coalesced memory access. The 1D1D kernel lacks these layout parameters and cannot handle per-group masking or indexing.

Do both kernels support FP4 and BF16 output formats?

Yes, both Kernel1D1D and Kernel1D2D support FP8 input with FP4 or BF16 output. The architectural distinction lies in grid topology and supported layouts, not data type support. Both implementations require row-major output (stride(-1) == 1 as enforced by check_major_type_cd in the layout utilities) and both utilize TMA (Tensor Memory Access) without splits, requiring swizzle_a_mode == block_k and swizzle_b_mode == block_k as asserted in both kernel headers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}