# How to Optimize Memory Layout for BF16 vs FP8 Tensors in DeepGEMM

> Optimize BF16 and FP8 tensor memory layouts in DeepGEMM. Learn techniques for row-major alignment and TMA-compatible access on NVIDIA GPUs to boost performance.

- Repository: [DeepSeek/DeepGEMM](https://github.com/deepseek-ai/DeepGEMM)
- Tags: performance
- Published: 2026-04-19

---

**Use `deep_gemm.set_mk_alignment_for_contiguous_layout` for BF16 tensors to ensure row-major alignment, and convert FP8 tensors to UE8M0 format with `get_mn_major_tma_aligned_packed_ue8m0_tensor` for TMA-compatible memory access on NVIDIA Hopper and Blackwell GPUs.**

DeepGEMM is a high-performance CUDA library for mixed-precision GEMM operations that requires distinct memory layout strategies for different data types. To optimize memory layout for BF16 vs FP8 tensors in DeepGEMM, you must understand how the library handles **contiguous row-major alignment** for BF16 versus **TMA-aligned packed UE8M0** formatting for FP8. This distinction ensures optimal bandwidth utilization and kernel compatibility across SM90 (Hopper) and SM100 (Blackwell) architectures.

## BF16 Memory Layout: Contiguous Row-Major with Group Alignment

BF16 tensors in DeepGEMM follow a straightforward contiguous layout that prioritizes memory coalescing through dimension padding.

### Alignment Requirements

BF16 tensors require **group-level M/K alignment** with a default value of 128. The M dimension must be padded to this alignment boundary to ensure that GPU warps can issue full coalesced loads. As implemented in [`csrc/utils/layout.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/utils/layout.hpp), this alignment prevents strided memory access penalties and maximizes cache line utilization.

### Implementation Details

The alignment configuration propagates through the Python API in [`deep_gemm/utils/layout.py`](https://github.com/deepseek-ai/DeepGEMM/blob/main/deep_gemm/utils/layout.py) to the C++ implementation in [`csrc/apis/layout.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/apis/layout.hpp). The function `set_mk_alignment_for_contiguous_layout` allows runtime tuning of the alignment value, which is stored in the heuristic runtime configuration.

```python
import deep_gemm
import torch

# Configure alignment (default is 128, but tune for specific kernels)

deep_gemm.set_mk_alignment_for_contiguous_layout(256)

# Create BF16 tensor and align the M dimension

M, K = 1024, 4096
a = torch.randn((M, K), dtype=torch.bfloat16, device='cuda')
a_aligned = deep_gemm.utils.align(a, axis=0)  # Pads M to alignment boundary

```

The `align` helper in [`deep_gemm/utils/math.py`](https://github.com/deepseek-ai/DeepGEMM/blob/main/deep_gemm/utils/math.py) internally queries `get_mk_alignment_for_contiguous_layout` to determine the required padding.

## FP8 Memory Layout: TMA-Aligned UE8M0 Packed Format

FP8 tensors utilize a specialized layout designed for the Tensor Memory Access (TMA) engine available on SM90 and SM100 GPUs.

### The UE8M0 Packed Format

FP8 tensors are cast to FP8 and then packed into **UE8M0** (`uint8`) representation. This packed format eliminates the half-byte padding that naive FP8 implementations would introduce. The packing operation is handled by `per_token_cast_to_fp8` in [`deep_gemm/utils/math.py`](https://github.com/deepseek-ai/DeepGEMM/blob/main/deep_gemm/utils/math.py), which returns both the packed tensor and a scale-factor tensor.

### TMA Alignment Requirements

For TMA compatibility, the packed tensor must be padded to a **TMA-aligned size** (multiple of 32 bytes) on the M dimension. Optionally, the K dimension can be padded via the `gran_k` parameter. These alignment constraints enable the TMA engine to stream data directly into shared memory without intermediate copies.

The Python API exposes this through `get_mn_major_tma_aligned_packed_ue8m0_tensor` and `get_k_grouped_mn_major_tma_aligned_packed_ue8m0_tensor`, with C++ implementations in [`csrc/apis/layout.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/apis/layout.hpp) and underlying kernel support in [`csrc/jit_kernels/impls/sm100_fp8_gemm_1d1d.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/impls/sm100_fp8_gemm_1d1d.hpp).

```python
import torch
import deep_gemm

# Convert BF16 activation to FP8 and pack into UE8M0

x = torch.randn((4096, 2048), dtype=torch.bfloat16, device='cuda')
x_fp8, sf = deep_gemm.per_token_cast_to_fp8(
    x,
    use_ue8m0=True,      # Pack into uint8 UE8M0 format

    gran_k=32            # Group K in chunks of 32

)

# Align for TMA loads (MN-major layout)

sf_aligned = deep_gemm.get_mn_major_tma_aligned_packed_ue8m0_tensor(sf)

```

### K-Grouped Layout for MoE Workloads

For Mixture-of-Experts (MoE) configurations where different experts have varying K dimensions, DeepGEMM provides grouped packing:

```python

# Example: 4 experts with different K sizes

ks = [1024, 2048, 512, 768]
ks_tensor = torch.tensor(ks, device='cuda', dtype=torch.int)

# Concatenated input along K dimension

x = torch.randn((sum(ks), 4096), dtype=torch.bfloat16, device='cuda')
x_fp8, sf = deep_gemm.per_channel_cast_to_fp8(x, use_ue8m0=True, gran_k=32)

# Pack with per-expert K grouping

packed_sf = deep_gemm.get_k_grouped_mn_major_tma_aligned_packed_ue8m0_tensor(
    sf, ks_tensor, ks, gran_k=32
)

```

The layout verification for grouped tensors is handled by `check_grouped_ab_fp8_fp4` in [`csrc/utils/layout.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/utils/layout.hpp).

## Key Differences Between BF16 and FP8 Layouts

| Aspect | BF16 (Contiguous) | FP8 (UE8M0 Packed) |
|--------|-------------------|-------------------|
| **Element Size** | 2 bytes (bfloat16) | 1 byte (FP8) or 0.5 bytes (packed FP4) |
| **Layout Strategy** | Row-major with padded M dimension | MN-major with TMA alignment on M |
| **Alignment** | Group-level M/K alignment (default 128) | TMA-aligned size (multiple of 32 bytes) |
| **Stride Pattern** | `stride(-2)=1`, `stride(-1)` padded to alignment | `stride(-2)=1` (MN-major), `stride(-1)` TMA-aligned |
| **Kernel Entry** | `deep_gemm.bf16_gemm_*` | `deep_gemm.fp8_*` or `deep_gemm.fp8_fp4_*` |
| **GPU Architecture** | SM90/100 | SM90/100 with TMA engine optimization |

## Runtime Switching Between BF16 and FP8

You can implement runtime dispatching to select the optimal layout based on your precision requirements:

```python
import torch
import deep_gemm

def optimized_matmul(a: torch.Tensor, b: torch.Tensor, use_fp8: bool = False):
    """
    Perform GEMM with automatic layout optimization for BF16 or FP8.
    """
    if use_fp8:
        # Convert to FP8 UE8M0 and align for TMA

        a_fp8, a_sf = deep_gemm.per_token_cast_to_fp8(a, use_ue8m0=True, gran_k=32)
        b_fp8, b_sf = deep_gemm.per_token_cast_to_fp8(b, use_ue8m0=True, gran_k=32)
        
        a_aligned = deep_gemm.get_mn_major_tma_aligned_packed_ue8m0_tensor(a_sf)
        b_aligned = deep_gemm.get_mn_major_tma_aligned_packed_ue8m0_tensor(b_sf)
        
        return deep_gemm.fp8_gemm_nt(a_aligned, b_aligned)
    else:
        # Ensure BF16 contiguous alignment

        a_aligned = deep_gemm.utils.align(a, axis=0)
        b_aligned = deep_gemm.utils.align(b, axis=0)
        
        return deep_gemm.bf16_gemm_nt(a_aligned, b_aligned)

```

The `bf16_gemm_nt` and `fp8_gemm_nt` functions are exposed in [`deep_gemm/__init__.py`](https://github.com/deepseek-ai/DeepGEMM/blob/main/deep_gemm/__init__.py) and delegate to the respective C++ kernels in [`csrc/jit_kernels/impls/sm100_bf16_gemm.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/impls/sm100_bf16_gemm.hpp) and [`csrc/jit_kernels/impls/sm100_fp8_fp4_gemm_1d1d.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/jit_kernels/impls/sm100_fp8_fp4_gemm_1d1d.hpp).

## Summary

- **BF16 tensors** require **contiguous row-major layout** with M dimension padded to the group alignment (default 128) using `set_mk_alignment_for_contiguous_layout` and `deep_gemm.utils.align`.

- **FP8 tensors** must be cast to **UE8M0 packed format** and aligned to **TMA boundaries** (32-byte multiples) using `get_mn_major_tma_aligned_packed_ue8m0_tensor` for standard layouts or `get_k_grouped_mn_major_tma_aligned_packed_ue8m0_tensor` for MoE workloads.

- **Architecture-specific constraints**: SM90 (Hopper) requires K-major FP8 tensors in some configurations, while SM100 (Blackwell) supports both K-major and MN-major layouts. The Python API automatically handles these constraints via [`csrc/utils/layout.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/utils/layout.hpp).

- **Performance impact**: Proper layout optimization eliminates strided memory access penalties, ensures full cache line utilization, and enables direct TMA streaming for FP8, resulting in significantly higher memory bandwidth efficiency compared to naive layouts.

## Frequently Asked Questions

### What is the default alignment value for BF16 tensors in DeepGEMM?

The default alignment for BF16 tensors is **128**, which corresponds to the group-level M/K alignment. You can verify or modify this value using `deep_gemm.set_mk_alignment_for_contiguous_layout()` in [`deep_gemm/utils/layout.py`](https://github.com/deepseek-ai/DeepGEMM/blob/main/deep_gemm/utils/layout.py), which propagates to the C++ runtime heuristics in [`csrc/apis/layout.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/apis/layout.hpp).

### Why does FP8 require TMA alignment while BF16 does not?

FP8 tensors utilize the **UE8M0 packed format** to achieve maximum memory bandwidth efficiency on SM90 and SM100 GPUs. The **Tensor Memory Access (TMA)** engine requires 32-byte aligned boundaries to stream data directly into shared memory without intermediate copies. BF16 tensors use traditional contiguous layouts that rely on coalesced warp loads rather than TMA hardware, requiring only standard cache line alignment (128 bytes).

### Can I use the same tensor for both BF16 and FP8 kernels without reformatting?

No, you cannot reuse the same memory layout between BF16 and FP8 kernels. BF16 kernels expect **contiguous row-major** tensors with M-dimension padding to the group alignment (via `deep_gemm.utils.align`). FP8 kernels require **MN-major TMA-aligned** UE8M0 packed tensors created via `get_mn_major_tma_aligned_packed_ue8m0_tensor`. Attempting to pass a BF16 layout to an FP8 kernel will trigger layout verification errors in [`csrc/utils/layout.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/utils/layout.hpp) (functions like `check_ab_fp8_fp4`).

### How do I handle variable K dimensions in MoE (Mixture of Experts) workloads?

For MoE configurations where different experts have varying K dimensions, use the **K-grouped layout helpers**. Instead of `get_mn_major_tma_aligned_packed_ue8m0_tensor`, call `get_k_grouped_mn_major_tma_aligned_packed_ue8m0_tensor()` with the `ks_tensor` parameter containing the per-expert K sizes. This function, implemented in [`csrc/apis/layout.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/apis/layout.hpp), internally verifies grouped layouts via `check_grouped_ab_fp8_fp4` and ensures each expert subgroup meets TMA alignment requirements independently.