SM90 vs SM100 Architecture Support in DeepGEMM: Hopper vs Ada Differences

SM90 (Hopper) supports only NT layout with FP32 scaling factors, while SM100 (Ada) supports all four GEMM layouts (NT, TN, NN, TT) with packed UE8M0 scaling factors and requires CUDA 12.9+.

DeepGEMM, the open-source FP8 GEMM library by DeepSeek, provides optimized matrix multiplication kernels for NVIDIA GPUs with distinct code paths for different compute architectures. Understanding the SM90 vs SM100 architecture support in DeepGEMM is essential for maximizing performance on Hopper (H100) versus Ada (RTX 40-series) hardware.

Architectural Overview

DeepGEMM implements separate kernel families targeting specific NVIDIA GPU generations, with fundamental differences in memory layout support, scaling factor formats, and thread-level optimizations.

SM90 (Hopper) Support

SM90 targets NVIDIA Hopper architecture GPUs with compute capability 9.0, such as the H100. According to the README.md, SM90 requires CUDA 12.3 or higher. The implementation resides primarily in deep_gemm/include/deep_gemm/common/sm90_utils.cuh, which defines SM90-specific utilities like SM90_U32x2_LDSM_N for low-level memory operations.

SM100 (Ada) Support

SM100 targets NVIDIA Ada Lovelace architecture GPUs with compute capability 10.0, including RTX 40-series cards. The README specifies CUDA 12.9 or higher for SM100 support. The implementation uses deep_gemm/include/deep_gemm/common/sm100_utils.cuh, containing structures like SM100_MMA_F16BF16_SS for mixed-precision arithmetic.

Key Differences in DeepGEMM Implementation

CUDA Version Requirements

The two architectures have distinct minimum CUDA toolkit requirements. As documented in README.md lines 29-36, SM90 requires CUDA 12.3 or higher, while SM100 requires CUDA 12.9 or higher. This reflects the different PTX instruction sets and driver features available on each generation.

Memory Layout Constraints

The most significant functional difference lies in supported GEMM layouts. As stated in README.md lines 65-66, the SM90 implementation supports only the NT layout (row-major A, column-major B). In contrast, the SM100 implementation supports all four classic GEMM layouts: NT, TN, NN, and TT.

This restriction in SM90 stems from Hopper's TMA (Tensor Memory Accelerator) constraints and the specific kernels implemented in deep_gemm/include/deep_gemm/scheduler/gemm.cuh, which notes SM90-specific scheduling limitations.

Scaling Factor Formats

The two architectures require different scaling factor (SF) formats for FP8 quantization. According to README.md lines 67-70:

• SM90: Requires FP32 scaling factors
• SM100: Requires packed UE8M0 format (4× UE8M0 packed into a single torch.int)

This difference affects how users must prepare input tensors. The SM100 path utilizes specialized packing utilities found in the SM100 kernel sources to handle the UE8M0 format efficiently.

TMA and Multicast Optimization

The scheduler implementations reveal architectural differences in TMA multicast handling. In deep_gemm/include/deep_gemm/scheduler/gemm.cuh lines 108-112, comments indicate that SM90 can dynamically disable TMA multicast based on workload characteristics.

Conversely, SM100 uses a fixed multicast configuration as implemented in deep_gemm/include/deep_gemm/common/sm100_utils.cuh, with SM100-specific TMA utilities that handle the packed UE8M0 data paths.

Kernel Architecture and Alignment

The alignment requirements differ significantly between the two:

• SM90: Uses an SF-aligned stride of 128 for float scaling factors, as defined in deep_gemm/include/deep_gemm/scheduler/gemm.cuh lines 35-36
• SM100: Uses an SF-aligned stride of 512 for packed UE8M0 formats

Additionally, SM90 kernels rely on single-CTA execution paths, while SM100 kernels can exploit 2-CTA pipelines for higher throughput, as evidenced by the dual-CTA scheduling logic in the SM100 utils.

Code Examples

The deep_gemm/__init__.py module provides a unified Python API that automatically dispatches to the appropriate kernel based on the runtime GPU's compute capability.

SM90 (Hopper) Usage

When running on an H100 (SM90), DeepGEMM automatically selects kernels that require NT layout and FP32 scaling factors:

import torch
import deep_gemm as dg

# SM90 requires NT layout: A row-major, B column-major

A = torch.randn(128, 256, dtype=torch.float16, device="cuda")
B = torch.randn(256, 128, dtype=torch.float16, device="cuda").t()

# Scaling factor must be FP32

sf = torch.randn(256, dtype=torch.float32, device="cuda")

# Automatically dispatches to SM90 kernel on H100

C = dg.fp8_gemm_nt(A, B, sf)

SM100 (Ada) Usage

On RTX 40-series GPUs (SM100), the library supports all layouts and requires packed UE8M0 scaling factors:

import torch
import deep_gemm as dg

# SM100 supports TN layout: A column-major, B row-major

A_tn = torch.randn(256, 128, dtype=torch.float16, device="cuda").t()
B_tn = torch.randn(128, 256, dtype=torch.float16, device="cuda")

# Scaling factor must be packed UE8M0 format

sf_float = torch.randn(128, dtype=torch.float32, device="cuda")
sf_packed = dg.pack_ue8m0(sf_float)

# Dispatches to SM100 kernel with TN layout support

C_tn = dg.fp8_gemm_tn(A_tn, B_tn, sf_packed)

Summary

• SM90 (Hopper) targets compute capability 9.0 (H100) and requires CUDA 12.3+, supporting only NT layout with FP32 scaling factors and 128-byte alignment.
• SM100 (Ada) targets compute capability 10.0 (RTX 40-series) and requires CUDA 12.9+, supporting all four layouts (NT, TN, NN, TT) with packed UE8M0 scaling factors and 512-byte alignment.
• Kernel architecture differs in TMA multicast handling (dynamic vs fixed) and CTA execution (single-CTA for SM90, 2-CTA pipelines for SM100).
• Implementation files are located in deep_gemm/include/deep_gemm/common/sm90_utils.cuh and deep_gemm/include/deep_gemm/common/sm100_utils.cuh, with scheduling logic in deep_gemm/include/deep_gemm/scheduler/gemm.cuh.

Frequently Asked Questions

What CUDA version do I need for SM90 vs SM100 in DeepGEMM?

For SM90 (Hopper) GPUs like the H100, you need CUDA 12.3 or higher. For SM100 (Ada) GPUs like the RTX 4090, you need CUDA 12.9 or higher. These requirements are documented in the README.md and reflect the different PTX instruction sets required for each architecture's TMA and MMA instructions.

Why does SM90 only support NT layout while SM100 supports all layouts?

The SM90 implementation is optimized specifically for Hopper's TMA (Tensor Memory Accelerator) constraints and uses specialized kernels that only handle the NT (row-major A, column-major B) layout efficiently. In contrast, the SM100 implementation in deep_gemm/include/deep_gemm/common/sm100_utils.cuh contains more flexible MMA utilities that support all four standard GEMM layouts (NT, TN, NN, TT), leveraging the broader capabilities of the Ada architecture.

What is the difference between FP32 and packed UE8M0 scaling factors?

SM90 requires FP32 scaling factors (32-bit floating point) for FP8 quantization, using a 128-byte alignment stride. SM100 requires packed UE8M0 format, which packs four 8-bit unsigned exponent values (UE8M0) into a single 32-bit integer, requiring 512-byte alignment. The packed format allows SM100 to process scaling factors more efficiently in its 2-CTA pipeline architecture, as implemented in the SM100 kernel utilities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}