UE8M0 Format in DeepGEMM: How SM100 Accelerates FP8 GEMM with Packed Scaling Factors

DeepGEMM uses the UE8M0 format to pack four 8-bit scaling factors into a single 32-bit integer, enabling SM100 GPUs to consume scaling factors directly via tensor cores without runtime conversion overhead.

The UE8M0 format is a specialized alternate floating-point representation defined by NVIDIA’s CUDA PTX specification. DeepGEMM leverages this format exclusively for SM100-class GPUs (also referenced as SMIOO in the codebase) to optimize the handling of scaling factors in high-performance FP8 GEMM operations. Unlike the FP32 scaling factors used on SM90, UE8M0 allows the hardware to read and apply scaling directly, eliminating costly conversion steps during kernel execution.

What Is the UE8M0 Format?

The UE8M0 format is an 8-bit floating-point encoding that stores a single-precision value using 1 sign bit, 8 exponent bits, and 0 mantissa bits. This design prioritizes dynamic range over precision, making it ideal for scaling factors where the exact value matters less than the order of magnitude.

Bit Layout and Packing

Four UE8M0 values are packed into a single 32-bit integer (torch.int), allowing vectorized memory access. The packing follows the pattern:

• Bit 0-7: First UE8M0 value
• Bit 8-15: Second UE8M0 value
• Bit 16-23: Third UE8M0 value
• Bit 24-31: Fourth UE8M0 value

This 4-way packing aligns with SM100’s memory transaction granularity, ensuring coalesced loads from global memory into the tensor core registers.

Hardware Support on SM100

SM100 (Blackwell architecture) introduces native UE8M0 support in its tensor core instruction set. The hardware can decode UE8M0 scaling factors and apply them to FP8 matrix multiplications without intermediate conversion to FP32 or FP16. This direct consumption path reduces register pressure and instruction count compared to SM90, which requires FP32 inputs and software-side conversion.

Why DeepGEMM Uses UE8M0 for SM100 Scaling Factors

DeepGEMM’s SM100 kernels require scaling factors in the packed UE8M0 format to maximize hardware utilization. The library abstracts this complexity, but understanding the rationale explains the strict layout requirements.

Performance Benefits Over FP32

Using UE8M0 instead of FP32 for scaling factors provides three key advantages on SM100:

1. Zero conversion overhead: The tensor core consumes UE8M0 directly, eliminating the FP32 → FP8 conversion instructions required on SM90.
2. Higher memory bandwidth: Packing four values into 32 bits reduces memory traffic by 4× compared to FP32 scalars.
3. Improved TMA efficiency: The 4-way packed layout matches the Tensor Memory Access (TMA) unit’s preferred transaction size, enabling asynchronous scaling factor loads that overlap with computation.

Memory Layout Requirements

SM100 kernels in DeepGEMM require the scaling factor tensor to follow a specific MN-major (column-major) layout with TMA alignment:

Architecture	Scaling Factor Data Type	Layout Requirement
SM90	FP32 (torch.float32)	Column-major TMA-aligned
SM100	Packed UE8M0 (torch.int)	MN-major TMA-aligned, packed 4-way

The MN-major (column-major) layout ensures that adjacent threads access contiguous memory addresses when loading scaling factors for the K dimension. The TMA alignment requirement (typically 128-byte boundaries) allows the hardware to use the Tensor Memory Access unit for asynchronous data movement.

How DeepGEMM Converts and Packs UE8M0 Scaling Factors

DeepGEMM handles the UE8M0 conversion internally through a series of preprocessing steps that transform user-provided FP32 tensors into the hardware-specific packed format.

Input Validation and TMA Alignment

The library first validates the input scaling factor tensor in get_mn_major_tma_aligned_packed_ue8m0_tensor. It checks that the tensor is FP32 (scalar_type == torch.kFloat) and determines whether it is 2-D [M, K] or 3-D [num_groups, M, K]. The function then computes the TMA-aligned size for the M dimension, typically rounding up to the nearest multiple of 64 or 128 to meet hardware alignment constraints.

FP32 to UE8M0 Conversion Logic

The core conversion logic extracts the exponent bits from the FP32 representation. Since UE8M0 uses the same exponent encoding as FP32 but omits the mantissa, the conversion requires a right-shift of 23 bits on the FP32 bit-pattern to discard the 23 mantissa bits. This operation is implemented in csrc/jit_kernels/impls/smxx_layout.hpp between lines 49 and 78.

The conversion packs four consecutive UE8M0 values into a single int32, producing a tensor with shape [num_groups, M_aligned, K//4] and dtype torch.int.

Fast CUDA Path vs. PyTorch Fallback

DeepGEMM provides two code paths for the conversion:

1. Fast CUDA Kernel (transpose_and_pack_fp32_into_ue8m0): Used when the input tensor is contiguous and meets layout requirements. This kernel fuses the transpose, alignment padding, and packing operations into a single CUDA kernel for maximum throughput.

2. PyTorch Fallback (get_mn_major_tma_aligned_packed_ue8m0_tensor_torch): Implemented in lines 49-78 of smxx_layout.hpp, this path uses PyTorch operations for non-contiguous inputs or when the CUDA kernel cannot be used. It provides flexibility at the cost of some performance.

MN-Major Layout Construction

The final layout places the packed UE8M0 data in MN-major (column-major) order with TMA alignment. The stride pattern is [packed_sf_k * aligned_M, 1, aligned_M], as implemented in get_mn_major_tma_aligned_packed_ue8m0_tensor at lines 74-80 of smxx_layout.hpp. This layout ensures that the TMA unit can load scaling factors efficiently during the GEMM computation.

Kernel Consumption on SM100

Once converted, the packed UE8M0 scaling factors are consumed directly by SM100 GEMM kernels without further conversion.

TMA Load Requirements

SM100 kernels such as fp8_gemm_* and fp8_fp4_mega_moe read the scaling factor tensor using the Tensor Memory Access (TMA) unit. The TMA load instructions expect the data to be in the packed UE8M0 format with MN-major layout, allowing the hardware to decompress and apply the scaling factors directly to the FP8 matrix elements during the multiply-accumulate operations.

Alignment Constraints

The scheduler enforces strict alignment requirements for the scaling factor tensor. In deep_gemm/include/deep_gemm/common/scheduler.cuh at line 35, the constant SF_K_ALIGNMENT = 512u indicates that the K dimension must be aligned to 512 bits (64 bytes), which corresponds to 4 UE8M0 values per 32-bit word. This alignment ensures that TMA transfers are coalesced and that the tensor cores receive data in the optimal format, as seen in the storage to l2_sf_buffer in deep_gemm/include/deep_gemm/impls/sm100_fp8_fp4_mega_moe.cuh at line 1059.

Practical Code Example

The following example demonstrates how to prepare scaling factors for SM100 kernels in DeepGEMM. The library handles the UE8M0 conversion internally, but understanding the data flow helps debug alignment issues.

import torch
import deep_gemm

# 1. Create a regular FP32 scaling factor tensor

# Shape: [num_groups, M, K] for grouped GEMM or [M, K] for standard

sf_fp32 = torch.randn(1, 1024, 256, dtype=torch.float32, device='cuda')

# 2. Convert to the packed UE8M0 layout required by SM100

# This uses the fast CUDA path when the tensor is contiguous

sf_ue8m0 = deep_gemm.get_mn_major_tma_aligned_packed_ue8m0_tensor(sf_fp32)

# The output is torch.int32 with shape [group, M_aligned, K//4]

print(f"Packed shape: {sf_ue8m0.shape}, dtype: {sf_ue8m0.dtype}")

# 3. Use in an SM100 GEMM kernel

a = torch.randn(1024, 256, dtype=torch.float8_e4m3fn, device='cuda')
b = torch.randn(256, 512, dtype=torch.float8_e4m3fn, device='cuda')
c = torch.empty(1024, 512, dtype=torch.bfloat16, device='cuda')

# The kernel reads sf_ue8m0 directly via TMA without conversion

deep_gemm.fp8_gemm_nt(c, a, b, sf_ue8m0)  # SM100 only

For debugging or when working with non-contiguous tensors, use the PyTorch fallback:

# Fallback path when CUDA kernel constraints aren't met

sf_ue8m0_torch = deep_gemm.get_mn_major_tma_aligned_packed_ue8m0_tensor_torch(sf_fp32)

Summary

• UE8M0 is an 8-bit floating-point format (1 sign, 8 exponent, 0 mantissa) that packs four values into a 32-bit integer, reducing memory bandwidth by 4× compared to FP32.
• SM100 GPUs consume UE8M0 scaling factors directly via tensor cores, eliminating the conversion overhead required on SM90 and improving TMA load efficiency.
• DeepGEMM converts user-provided FP32 tensors to packed UE8M0 internally using get_mn_major_tma_aligned_packed_ue8m0_tensor, which applies a 23-bit right-shift to extract exponents and arranges data in an MN-major, TMA-aligned layout.
• Alignment requirements are strict: the K dimension must align to 512 bits (64 bytes), and the layout must be column-major (MN-major) to ensure coalesced TMA transfers as defined in SF_K_ALIGNMENT = 512u in scheduler.cuh.

Frequently Asked Questions

What does UE8M0 stand for in the context of DeepGEMM?

UE8M0 stands for Unsigned Exponent 8-bit Mantissa 0-bit, though DeepGEMM uses it with a sign bit included (1 sign, 8 exponent, 0 mantissa). This format is defined in the NVIDIA CUDA PTX specification as an alternate floating-point format that stores only the exponent and sign, effectively representing a scale factor as a power-of-two value.

Why does SM100 require UE8M0 instead of FP32 for scaling factors?

SM100 tensor cores can directly consume UE8M0 values without intermediate conversion, whereas SM90 requires FP32 inputs that must be converted to FP8 internally. By using UE8M0, DeepGEMM eliminates conversion instructions inside the kernel, reduces memory bandwidth by packing four values per 32-bit word, and enables more efficient TMA (Tensor Memory Access) loads that align with the hardware's 128-bit transaction granularity.

How do I prepare a scaling factor tensor for SM100 kernels in DeepGEMM?

You do not need to manually convert FP32 values to UE8M0. Instead, pass your FP32 scaling factor tensor to deep_gemm.get_mn_major_tma_aligned_packed_ue8m0_tensor(), which returns a torch.int32 tensor with shape [group, M_aligned, K//4]. This function handles the bit-shifting (right-shift by 23), 4-way packing, and MN-major TMA alignment automatically. If your tensor is non-contiguous, use the fallback get_mn_major_tma_aligned_packed_ue8m0_tensor_torch() instead.

What happens if the scaling factor tensor is not aligned to 512 bits on SM100?

DeepGEMM enforces strict alignment requirements defined by SF_K_ALIGNMENT = 512u in deep_gemm/include/deep_gemm/common/scheduler.cuh. If the K dimension is not aligned to 512 bits (64 bytes), the TMA (Tensor Memory Access) unit cannot perform efficient coalesced loads, leading to hardware exceptions or severe performance degradation. The preprocessing functions in csrc/jit_kernels/impls/smxx_layout.hpp automatically pad the M dimension and ensure the K dimension meets this alignment before the kernel launches.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}