How to Optimize Memory Layout for BF16 vs FP8 Tensors in DeepGEMM

Use deep_gemm.set_mk_alignment_for_contiguous_layout for BF16 tensors to ensure row-major alignment, and convert FP8 tensors to UE8M0 format with get_mn_major_tma_aligned_packed_ue8m0_tensor for TMA-compatible memory access on NVIDIA Hopper and Blackwell GPUs.

DeepGEMM is a high-performance CUDA library for mixed-precision GEMM operations that requires distinct memory layout strategies for different data types. To optimize memory layout for BF16 vs FP8 tensors in DeepGEMM, you must understand how the library handles contiguous row-major alignment for BF16 versus TMA-aligned packed UE8M0 formatting for FP8. This distinction ensures optimal bandwidth utilization and kernel compatibility across SM90 (Hopper) and SM100 (Blackwell) architectures.

BF16 Memory Layout: Contiguous Row-Major with Group Alignment

BF16 tensors in DeepGEMM follow a straightforward contiguous layout that prioritizes memory coalescing through dimension padding.

Alignment Requirements

BF16 tensors require group-level M/K alignment with a default value of 128. The M dimension must be padded to this alignment boundary to ensure that GPU warps can issue full coalesced loads. As implemented in csrc/utils/layout.hpp, this alignment prevents strided memory access penalties and maximizes cache line utilization.

Implementation Details

The alignment configuration propagates through the Python API in deep_gemm/utils/layout.py to the C++ implementation in csrc/apis/layout.hpp. The function set_mk_alignment_for_contiguous_layout allows runtime tuning of the alignment value, which is stored in the heuristic runtime configuration.

import deep_gemm
import torch

# Configure alignment (default is 128, but tune for specific kernels)

deep_gemm.set_mk_alignment_for_contiguous_layout(256)

# Create BF16 tensor and align the M dimension

M, K = 1024, 4096
a = torch.randn((M, K), dtype=torch.bfloat16, device='cuda')
a_aligned = deep_gemm.utils.align(a, axis=0)  # Pads M to alignment boundary

The align helper in deep_gemm/utils/math.py internally queries get_mk_alignment_for_contiguous_layout to determine the required padding.

FP8 Memory Layout: TMA-Aligned UE8M0 Packed Format

FP8 tensors utilize a specialized layout designed for the Tensor Memory Access (TMA) engine available on SM90 and SM100 GPUs.

The UE8M0 Packed Format

FP8 tensors are cast to FP8 and then packed into UE8M0 (uint8) representation. This packed format eliminates the half-byte padding that naive FP8 implementations would introduce. The packing operation is handled by per_token_cast_to_fp8 in deep_gemm/utils/math.py, which returns both the packed tensor and a scale-factor tensor.

TMA Alignment Requirements

For TMA compatibility, the packed tensor must be padded to a TMA-aligned size (multiple of 32 bytes) on the M dimension. Optionally, the K dimension can be padded via the gran_k parameter. These alignment constraints enable the TMA engine to stream data directly into shared memory without intermediate copies.

The Python API exposes this through get_mn_major_tma_aligned_packed_ue8m0_tensor and get_k_grouped_mn_major_tma_aligned_packed_ue8m0_tensor, with C++ implementations in csrc/apis/layout.hpp and underlying kernel support in csrc/jit_kernels/impls/sm100_fp8_gemm_1d1d.hpp.

import torch
import deep_gemm

# Convert BF16 activation to FP8 and pack into UE8M0

x = torch.randn((4096, 2048), dtype=torch.bfloat16, device='cuda')
x_fp8, sf = deep_gemm.per_token_cast_to_fp8(
    x,
    use_ue8m0=True,      # Pack into uint8 UE8M0 format

    gran_k=32            # Group K in chunks of 32

)

# Align for TMA loads (MN-major layout)

sf_aligned = deep_gemm.get_mn_major_tma_aligned_packed_ue8m0_tensor(sf)

K-Grouped Layout for MoE Workloads

For Mixture-of-Experts (MoE) configurations where different experts have varying K dimensions, DeepGEMM provides grouped packing:

# Example: 4 experts with different K sizes

ks = [1024, 2048, 512, 768]
ks_tensor = torch.tensor(ks, device='cuda', dtype=torch.int)

# Concatenated input along K dimension

x = torch.randn((sum(ks), 4096), dtype=torch.bfloat16, device='cuda')
x_fp8, sf = deep_gemm.per_channel_cast_to_fp8(x, use_ue8m0=True, gran_k=32)

# Pack with per-expert K grouping

packed_sf = deep_gemm.get_k_grouped_mn_major_tma_aligned_packed_ue8m0_tensor(
    sf, ks_tensor, ks, gran_k=32
)

The layout verification for grouped tensors is handled by check_grouped_ab_fp8_fp4 in csrc/utils/layout.hpp.

Key Differences Between BF16 and FP8 Layouts

Aspect	BF16 (Contiguous)	FP8 (UE8M0 Packed)
Element Size	2 bytes (bfloat16)	1 byte (FP8) or 0.5 bytes (packed FP4)
Layout Strategy	Row-major with padded M dimension	MN-major with TMA alignment on M
Alignment	Group-level M/K alignment (default 128)	TMA-aligned size (multiple of 32 bytes)
Stride Pattern	stride(-2)=1, stride(-1) padded to alignment	stride(-2)=1 (MN-major), stride(-1) TMA-aligned
Kernel Entry	deep_gemm.bf16_gemm_*	deep_gemm.fp8_* or deep_gemm.fp8_fp4_*
GPU Architecture	SM90/100	SM90/100 with TMA engine optimization

Runtime Switching Between BF16 and FP8

You can implement runtime dispatching to select the optimal layout based on your precision requirements:

import torch
import deep_gemm

def optimized_matmul(a: torch.Tensor, b: torch.Tensor, use_fp8: bool = False):
    """
    Perform GEMM with automatic layout optimization for BF16 or FP8.
    """
    if use_fp8:
        # Convert to FP8 UE8M0 and align for TMA

        a_fp8, a_sf = deep_gemm.per_token_cast_to_fp8(a, use_ue8m0=True, gran_k=32)
        b_fp8, b_sf = deep_gemm.per_token_cast_to_fp8(b, use_ue8m0=True, gran_k=32)
        
        a_aligned = deep_gemm.get_mn_major_tma_aligned_packed_ue8m0_tensor(a_sf)
        b_aligned = deep_gemm.get_mn_major_tma_aligned_packed_ue8m0_tensor(b_sf)
        
        return deep_gemm.fp8_gemm_nt(a_aligned, b_aligned)
    else:
        # Ensure BF16 contiguous alignment

        a_aligned = deep_gemm.utils.align(a, axis=0)
        b_aligned = deep_gemm.utils.align(b, axis=0)
        
        return deep_gemm.bf16_gemm_nt(a_aligned, b_aligned)

The bf16_gemm_nt and fp8_gemm_nt functions are exposed in deep_gemm/__init__.py and delegate to the respective C++ kernels in csrc/jit_kernels/impls/sm100_bf16_gemm.hpp and csrc/jit_kernels/impls/sm100_fp8_fp4_gemm_1d1d.hpp.

Summary

• BF16 tensors require contiguous row-major layout with M dimension padded to the group alignment (default 128) using set_mk_alignment_for_contiguous_layout and deep_gemm.utils.align.

• FP8 tensors must be cast to UE8M0 packed format and aligned to TMA boundaries (32-byte multiples) using get_mn_major_tma_aligned_packed_ue8m0_tensor for standard layouts or get_k_grouped_mn_major_tma_aligned_packed_ue8m0_tensor for MoE workloads.

• Architecture-specific constraints: SM90 (Hopper) requires K-major FP8 tensors in some configurations, while SM100 (Blackwell) supports both K-major and MN-major layouts. The Python API automatically handles these constraints via csrc/utils/layout.hpp.

• Performance impact: Proper layout optimization eliminates strided memory access penalties, ensures full cache line utilization, and enables direct TMA streaming for FP8, resulting in significantly higher memory bandwidth efficiency compared to naive layouts.

Frequently Asked Questions

What is the default alignment value for BF16 tensors in DeepGEMM?

The default alignment for BF16 tensors is 128, which corresponds to the group-level M/K alignment. You can verify or modify this value using deep_gemm.set_mk_alignment_for_contiguous_layout() in deep_gemm/utils/layout.py, which propagates to the C++ runtime heuristics in csrc/apis/layout.hpp.

Why does FP8 require TMA alignment while BF16 does not?

FP8 tensors utilize the UE8M0 packed format to achieve maximum memory bandwidth efficiency on SM90 and SM100 GPUs. The Tensor Memory Access (TMA) engine requires 32-byte aligned boundaries to stream data directly into shared memory without intermediate copies. BF16 tensors use traditional contiguous layouts that rely on coalesced warp loads rather than TMA hardware, requiring only standard cache line alignment (128 bytes).

Can I use the same tensor for both BF16 and FP8 kernels without reformatting?

No, you cannot reuse the same memory layout between BF16 and FP8 kernels. BF16 kernels expect contiguous row-major tensors with M-dimension padding to the group alignment (via deep_gemm.utils.align). FP8 kernels require MN-major TMA-aligned UE8M0 packed tensors created via get_mn_major_tma_aligned_packed_ue8m0_tensor. Attempting to pass a BF16 layout to an FP8 kernel will trigger layout verification errors in csrc/utils/layout.hpp (functions like check_ab_fp8_fp4).

How do I handle variable K dimensions in MoE (Mixture of Experts) workloads?

For MoE configurations where different experts have varying K dimensions, use the K-grouped layout helpers. Instead of get_mn_major_tma_aligned_packed_ue8m0_tensor, call get_k_grouped_mn_major_tma_aligned_packed_ue8m0_tensor() with the ks_tensor parameter containing the per-expert K sizes. This function, implemented in csrc/apis/layout.hpp, internally verifies grouped layouts via check_grouped_ab_fp8_fp4 and ensures each expert subgroup meets TMA alignment requirements independently.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}