fp8_gemm vs fp8_fp4_gemm in DeepGEMM: Architecture, Features, and Migration Guide

fp8_gemm is a legacy compatibility alias that forwards to fp8_fp4_gemm, the core implementation that supports both FP8 and the memory-efficient FP4 (UE8M0) format on NVIDIA Hopper and Blackwell GPUs.

In the DeepGEMM library, these two function families represent different layers of the API stack. While both execute high-performance matrix multiplication on quantized tensors, only fp8_fp4_gemm exposes the full feature set including packed FP4 scaling recipes and architecture-specific kernel selection.

What is fp8_gemm?

fp8_gemm exists as a thin backward-compatibility wrapper that provides the original FP8-only API surface. According to the source code in csrc/apis/gemm.hpp, these symbols are explicitly aliased to their fp8_fp4_gemm counterparts at line 659:

// csrc/apis/gemm.hpp (line 659)
m.attr("fp8_gemm_nt") = m.attr("fp8_fp4_gemm_nt");
m.attr("fp8_gemm_nn") = m.attr("fp8_fp4_gemm_nn");
// ... additional variants

When you call fp8_gemm_nt, the library immediately delegates to fp8_fp4_gemm_nt without additional logic. This design preserves existing codebases while funneling all execution through a unified implementation path.

What is fp8_fp4_gemm?

fp8_fp4_gemm is the core computational implementation located at lines 50-100 of csrc/apis/gemm.hpp. Unlike the legacy alias, this function handles both traditional FP8 quantization and the newer FP4 (packed UE8M0) format that reduces memory bandwidth on NVIDIA Hopper and Blackwell architectures.

The implementation performs three critical operations:

1. Shape validation for tiled matrix dimensions
2. Scaling factor layout transformation based on the recipe parameters
3. Kernel dispatch to architecture-specific JIT-compiled kernels

// csrc/apis/gemm.hpp (simplified excerpt)
static void fp8_fp4_gemm_nt(
    const std::pair<torch::Tensor, torch::Tensor>& a,
    const std::pair<torch::Tensor, torch::Tensor>& b,
    const torch::Tensor& d,
    const std::optional<torch::Tensor>& c,
    std::optional<std::tuple<int, int, int>> recipe,
    std::optional<std::tuple<int, int>> recipe_a,
    std::optional<std::tuple<int, int>> recipe_b,
    const std::string& compiled_dims,
    const bool& disable_ue8m0_cast) {
    
    // Architecture-specific dispatch
    if (arch_major == 9 && sfa.scalar_type() == torch::kFloat) {
        sm90_fp8_gemm_1d1d(...);
    } else if (arch_major == 10 && sfa.scalar_type() == torch::kInt) {
        sm100_fp8_fp4_gemm_1d1d(...);
    }
}

The function selects between:

• sm90_fp8_gemm_1d1d from csrc/jit_kernels/impls/sm90_fp8_gemm_1d1d.hpp for Hopper (SM90) FP8-only mode
• sm100_fp8_fp4_gemm_1d1d from csrc/jit_kernels/impls/sm100_fp8_fp4_gemm_1d1d.hpp for Blackwell (SM100) with packed FP4 support

Key Differences Between fp8_gemm and fp8_fp4_gemm

Feature	fp8_gemm (Legacy)	fp8_fp4_gemm (Current)
Implementation Layer	Alias/wrapper in csrc/apis/gemm.hpp line 659	Core implementation in csrc/apis/gemm.hpp lines 50-100
Format Support	FP8 only (historically)	FP8 and packed FP4 (UE8M0)
GPU Architecture	SM90 (Hopper) via delegation	SM90 and SM100 (Blackwell) with native kernel selection
API Parameters	Standard tensor inputs	Additional recipe, recipe_a, recipe_b, and disable_ue8m0_cast flags
Memory Bandwidth	Standard FP8 traffic	Reduced traffic when using FP4 mode
Deprecation Status	Legacy alias for backward compatibility	Recommended API for new development

Code Examples: Calling Both APIs

The following examples demonstrate the functional equivalence of the legacy alias and the modern API, while highlighting the FP4-specific features available only in fp8_fp4_gemm.

import torch
import deep_gemm as dg

# Common setup

M, N, K = 1024, 1024, 1024
a = torch.rand((M, K), dtype=torch.float16).cuda()
b = torch.rand((N, K), dtype=torch.float16).cuda()
c = torch.empty((M, N), dtype=torch.float16).cuda()

# -------------------------------------------------

# Legacy API: fp8_gemm_nt (backward compatibility)

# -------------------------------------------------

# Scaling factors for FP8 (float32)

sfa_fp8 = torch.ones(M, dtype=torch.float32).cuda()
sfb_fp8 = torch.ones(N, dtype=torch.float32).cuda()

# This call forwards internally to fp8_fp4_gemm_nt

dg.fp8_gemm_nt((a, sfa_fp8), (b, sfb_fp8), c)
print("Legacy API completed")

# -------------------------------------------------

# Modern API: fp8_fp4_gemm_nt (full features)

# -------------------------------------------------

# Case 1: Pure FP8 mode (same as above, but explicit)

dg.fp8_fp4_gemm_nt((a, sfa_fp8), (b, sfb_fp8), c)

# Case 2: FP4 packed mode (SM100/Blackwell only)

# Scaling factors packed as UE8M0 (uint8)

sfa_fp4 = torch.randint(0, 256, (M,), dtype=torch.uint8).cuda()
sfb_fp4 = torch.randint(0, 256, (N,), dtype=torch.uint8).cuda()

# Recipe describes how scaling factors are packed

recipe = (1, 1, 128)  # (gran_k_a, gran_k_b, gran_n)

dg.fp8_fp4_gemm_nt(
    (a, sfa_fp4), 
    (b, sfb_fp4), 
    c,
    recipe=recipe,
    compiled_dims="MKN"
)
print("Modern FP4 API completed")

When targeting NVIDIA Hopper (SM90), both APIs route to sm90_fp8_gemm_1d1d from csrc/jit_kernels/impls/sm90_fp8_gemm_1d1d.hpp. On Blackwell (SM100), the modern API can dispatch to sm100_fp8_fp4_gemm_1d1d from csrc/jit_kernels/impls/sm100_fp8_fp4_gemm_1d1d.hpp when FP4 scaling is detected.

Migration Guide: When to Use Which

Use fp8_fp4_gemm for all new development. This API provides complete access to DeepGEMM's capabilities, including the packed FP4 format that reduces memory bandwidth by storing scaling factors as 8-bit UE8M0 integers rather than 32-bit floats.

Continue using fp8_gemm only for legacy code maintenance. The function remains functional but offers no performance advantage. According to the binding code in csrc/apis/gemm.hpp, these symbols are explicitly aliased:

m.attr("fp8_gemm_nt") = m.attr("fp8_fp4_gemm_nt");

Enable FP4 mode when:

• Running on SM100 (Blackwell) or newer architectures
• Memory bandwidth is the bottleneck in your GEMM operations
• You can represent scaling factors as packed UE8M0 format

Stick to FP8 mode when:

• Running on SM90 (Hopper) architecture
• You require floating-point scaling factors for numerical precision
• You have not yet implemented UE8M0 packing for your tensors

Summary

• fp8_gemm is a legacy alias maintained for backward compatibility that forwards all calls to fp8_fp4_gemm without additional logic.
• fp8_fp4_gemm is the core implementation in csrc/apis/gemm.hpp (lines 50-100) that handles both FP8 and the packed FP4 (UE8M0) format.
• The modern API accepts specialized parameters including recipe, recipe_a, recipe_b, and disable_ue8m0_cast to control FP4 packing behavior.
• Architecture-specific kernels are selected automatically: sm90_fp8_gemm_1d1d for Hopper (SM90) and sm100_fp8_fp4_gemm_1d1d for Blackwell (SM100).
• New code should use fp8_fp4_gemm directly to access full functionality and future-proof against removal of the legacy aliases.

Frequently Asked Questions

Is fp8_gemm deprecated in DeepGEMM?

While not officially removed, fp8_gemm functions are implemented as explicit aliases to their fp8_fp4_gemm counterparts in csrc/apis/gemm.hpp (line 659). The DeepGEMM source code marks these as compatibility bindings, and new development should use the fp8_fp4_gemm variants to ensure access to all current and future features.

Can I use fp8_fp4_gemm on older GPUs like A100?

No. The fp8_fp4_gemm implementation specifically checks for architecture compatibility in csrc/apis/gemm.hpp. For SM90 (Hopper, H100), it dispatches to sm90_fp8_gemm_1d1d. For SM100 (Blackwell), it uses sm100_fp8_fp4_gemm_1d1d. The A100 (SM80) lacks native FP8 hardware support, and DeepGEMM does not provide kernels for this architecture.

What is the FP4 (UE8M0) format and when should I use it?

FP4 refers to a packed 4-bit quantization format using UE8M0 (unsigned 8-bit magnitude, 0-bit exponent) storage for scaling factors. In fp8_fp4_gemm, enabling FP4 mode (by passing int scaling factors and a valid recipe) reduces memory bandwidth by packing scale factors into 8-bit integers rather than 32-bit floats. Use this mode on SM100 (Blackwell) GPUs when your workload is memory-bound and you can quantize your scaling factors to the UE8M0 representation.

Do I need to change my code if I'm currently using fp8_gemm?

No immediate changes are required for functionality, as fp8_gemm calls will continue to work through the alias mechanism. However, to access FP4 capabilities and ensure forward compatibility, you should migrate your calls to fp8_fp4_gemm. The function signatures are compatible for FP8-only usage, requiring only the function name change in most cases. If you wish to enable FP4 mode, you will need to update your scaling factor tensors from float32 to packed uint8 format and provide the appropriate recipe tuple.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}