Contiguous and Masked Grouped GEMM Layouts in DeepGEMM: When to Use Each

Use the contiguous grouped layout in DeepGEMM when all groups share the same M dimension for uniform batched workloads, and use the masked grouped layout when groups have variable sizes or you need to dynamically skip empty groups without reshaping the output tensor.

The deepseek-ai/DeepGEMM library accelerates grouped matrix multiplication (GEMM) for FP8 and BF16 inference through custom CUDA kernels. When invoking grouped GEMM operations, you must specify how the per-group output is organized using either the contiguous grouped layout or the masked grouped layout. Understanding these layouts is essential for optimizing throughput in transformer inference and training workloads.

What Are Grouped GEMM Layouts?

Grouped GEMM kernels process multiple independent matrix multiplications (groups) in a single launch. Unlike standard batched GEMM, grouped GEMM allows each group to have distinct dimensions or to be conditionally skipped. DeepGEMM provides two distinct layout abstractions to describe how these groups map to the output tensor: the contiguous layout for dense, uniform grouping and the masked layout for sparse or ragged grouping.

Contiguous Grouped Layout

The contiguous grouped layout represents group metadata as a dense, contiguous tensor storing per-group offsets or partial-sum (psum) accumulations. This layout assumes every group in the batch has an identical M dimension.

Structure and Validation

In csrc/apis/gemm.hpp, the contiguous layout API validates the tensor using strict contiguity and shape assertions:

DG_HOST_ASSERT(grouped_layout.is_contiguous());
if (use_psum_layout) {
    const auto [num_groups_] = get_shape<1>(grouped_layout);
    DG_HOST_ASSERT(num_groups == num_groups_);
} else {
    const auto [m__] = get_shape<1>(grouped_layout);
    DG_HOST_ASSERT(m == m__);
}

For standard operations, the layout must be a 2-D tensor of shape [M, num_groups]. When using partial-sum reduction (use_psum_layout=true), it becomes a 1-D int tensor of length num_groups.

When to Use Contiguous Layout

Choose the contiguous grouped layout when:

• All groups share the same M dimension (e.g., fixed-length sequences in batched inference).
• You require the fastest execution path with minimal metadata overhead.
• You are using partial-sum (psum) reduction across groups.

Masked Grouped Layout

The masked grouped layout uses an integer mask tensor to indicate which groups are active. While the actual per-group data remains in a dense tensor d, the mask allows the kernel to skip groups with zero mask entries, supporting variable-sized or sparse batches.

Structure and Validation

The masked layout validation occurs in csrc/apis/gemm.hpp within the m_grouped_*_masked functions:

DG_HOST_ASSERT(masked_m.is_contiguous());
DG_HOST_ASSERT(masked_m.scalar_type() == torch::kInt);
const auto num_groups___ = static_cast<int>(masked_m.numel());
DG_HOST_ASSERT(num_groups == num_groups___);

The mask must be a contiguous 1-D int32 tensor with length equal to num_groups. Non-zero values indicate active groups; zero values cause the kernel to skip processing for that group index.

When to Use Masked Layout

Select the masked grouped layout when:

• Groups have heterogeneous M dimensions (e.g., ragged batches in transformer inference).
• You need to dynamically skip empty or pruned groups without reshaping the output tensor d.
• You are implementing conditional execution or dynamic load balancing across groups.

Source Code Implementation

The layout abstractions are defined in the following key files:

• csrc/apis/gemm.hpp: Contains the public API entry points m_grouped_*_contiguous and m_grouped_*_masked, including all validation logic for tensor contiguity, shape, and data type.
• csrc/utils/layout.hpp: Implements underlying layout checks such as is_contiguous() used by the API validators.
• csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp: Houses the low-level JIT kernels that consume these layouts (e.g., sm90_m_grouped_fp8_gemm_masked_1d2d), executing the actual grouped GEMM operations based on the provided layout type.

Practical Examples

Below are complete Python examples demonstrating both layout types using the deep_gemm module.

Contiguous Grouped GEMM

Use this when all groups have uniform size:

import torch
import deep_gemm as dg

M, N, K = 64, 128, 128
num_groups = 4

# Input tensors

A = torch.randn(M, K, dtype=torch.float, device='cuda')
B = torch.randn(num_groups, N, K, dtype=torch.float, device='cuda')
D = torch.empty(M, N, dtype=torch.float, device='cuda')

# Contiguous 1-D layout (psum style) or 2-D layout [M, num_groups]

grouped_layout = torch.arange(num_groups, device='cuda', dtype=torch.int)

dg.m_grouped_fp8_fp4_gemm_nt_contiguous(
    (A, torch.empty_like(A)),  # Scaling factors for A (dummy here)

    (B, torch.empty_like(B)),  # Scaling factors for B

    D,
    grouped_layout,
    recipe=None,
    compiled_dims="nk"
)

Masked Grouped GEMM

Use this to skip specific groups dynamically:

import torch
import deep_gemm as dg

M, N, K = 64, 128, 128
num_groups = 4

A = torch.randn(M, K, dtype=torch.float, device='cuda')
B = torch.randn(num_groups, N, K, dtype=torch.float, device='cuda')
D = torch.empty(M, N, dtype=torch.float, device='cuda')

# Mask: skip group 2 (index 2) by setting it to 0

masked_m = torch.tensor([1, 1, 0, 1], device='cuda', dtype=torch.int)

dg.m_grouped_fp8_fp4_gemm_nt_masked(
    (A, torch.empty_like(A)),
    (B, torch.empty_like(B)),
    D,
    masked_m,
    expected_m=M,  # Required for validation

    compiled_dims="nk"
)

In the first example, grouped_layout is contiguous and tells the kernel that all four groups are present. In the second example, masked_m is also contiguous but acts as a mask; the kernel processes only the groups whose mask entry is non-zero.

Summary

• Contiguous grouped layout stores per-group offsets in a dense, contiguous tensor and requires all groups to share the same M dimension. It is validated in csrc/apis/gemm.hpp via is_contiguous() and shape assertions. Use this for uniform batched workloads and maximum performance.

• Masked grouped layout uses an integer mask tensor to indicate active groups, allowing variable M dimensions and dynamic group skipping. It is validated in csrc/apis/gemm.hpp by checking is_contiguous(), scalar_type() == torch::kInt, and mask length. Use this for ragged batches, dynamic pruning, or conditional execution.

• Both layouts require contiguous memory but serve different sparsity patterns: contiguous for dense regular grouping, masked for sparse or irregular grouping.

Frequently Asked Questions

What is the difference between contiguous and masked grouped GEMM layouts in DeepGEMM?

The contiguous grouped layout represents group metadata as a dense tensor where every group has the same M dimension, storing per-group offsets or partial-sum accumulations. The masked grouped layout represents groups as an integer mask where non-zero values indicate active groups, allowing the kernel to skip inactive groups and support variable M dimensions without reshaping the output tensor.

When should I use the masked grouped layout over the contiguous layout?

Use the masked grouped layout when your input groups have heterogeneous sizes (ragged batches), when you need to dynamically skip empty or pruned groups during inference, or when implementing conditional execution where only a subset of groups contains valid data. Use the contiguous layout only when all groups share identical M dimensions and you require the fastest execution path.

How does DeepGEMM validate layout tensors in the source code?

In csrc/apis/gemm.hpp, DeepGEMM validates contiguous layouts by asserting grouped_layout.is_contiguous() and checking that the tensor shape matches [M, num_groups] (or [num_groups] for psum layouts). For masked layouts, the code asserts masked_m.is_contiguous(), verifies masked_m.scalar_type() == torch::kInt, and confirms the mask length equals num_groups.

Can I use partial-sum (psum) reduction with the masked grouped layout?

No, the partial-sum (psum) layout is specifically designed for the contiguous grouped layout as a 1-D integer tensor of length num_groups. The masked layout uses a different signaling mechanism (integer mask) and does not support psum reduction in the same way. If your workload requires partial-sum accumulation across groups, use the contiguous layout with use_psum_layout=true.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}