How to Handle NT/NN/TN/TT Transpose Conventions in DeepGEMM

DeepGEMM requires explicit tensor transposition for non-default layouts, with NT being the native row-major/column-major format that requires no manual preparation, while NN, TN, and TT layouts require calling .T.contiguous() on inputs before invoking the corresponding kernel variants.

The deepseek-ai/DeepGEMM repository provides just-in-time compiled GPU kernels for FP8 and FP4 matrix multiplication. Understanding how to handle the NT/NN/TN/TT transpose conventions is critical for correctly invoking these GEMM APIs, as the library does not perform implicit transpositions to maintain a lightweight kernels-only JIT path.

Understanding the Four Layout Conventions

DeepGEMM follows standard BLAS naming conventions where two letters describe the storage format of matrices A and B:

• First letter: Storage of matrix A (row-major or column-major)
• Second letter: Storage of matrix B (row-major or column-major)

N denotes non-transposed (row-major for A, column-major for B in DeepGEMM's convention), while T denotes transposed (column-major for A, row-major for B).

DeepGEMM's Default NT Layout

The NT layout is the native format for all DeepGEMM kernels. In this configuration:

• Matrix A is stored in row-major (non-transposed) order
• Matrix B is stored in column-major (transposed) order
• The kernel computes D = C + A @ Bᵀ

For NT operations, you can pass tensors directly without manual transposition:

import torch
import deep_gemm

A = torch.randn(1024, 2048, device='cuda')
B = torch.randn(4096, 2048, device='cuda')  # column-major (transposed) storage

D = torch.empty_like(A @ B.T)

deep_gemm.fp8_gemm_nt(A, B, D)

Handling NN, TN, and TT Layouts on SM 100

While SM 90 GPUs only support the NT layout, SM 100 (Blackwell) GPUs support all four transpose conventions. However, DeepGEMM does not perform implicit transpositions internally. You must explicitly transpose tensors before calling the corresponding API (*_nn, *_tn, *_tt).

NN Layout (Both Operands Transposed)

For the NN layout, both A and B must be transposed from their native NT orientation:

A_nn = A.T.contiguous()  # row-major → column-major

B_nn = B.T.contiguous()  # column-major → row-major

D_nn = torch.empty_like(A_nn @ B_nn.T)

deep_gemm.fp8_gemm_nn(A_nn, B_nn, D_nn)

TN Layout (Only A Transposed)

For the TN layout, only matrix A requires transposition:

A_tn = A.T.contiguous()
D_tn = torch.empty_like(A_tn @ B.T)

deep_gemm.fp8_gemm_tn(A_tn, B, D_tn)

TT Layout (Both Operands + Scaling Factor Transposed)

The TT layout requires transposing both input matrices and the scaling-factor tensor. For FP4 operations, use the transpose_packed_fp4 utility:

from deep_gemm.utils.math import transpose_packed_fp4

A_tt = A.T.contiguous()
B_tt = B.T.contiguous()

# sf is a packed FP4 scaling-factor tensor from per_token_cast_to_fp4

sf_tt = transpose_packed_fp4(sf)

D_tt = torch.empty_like(A_tt @ B_tt.T)
deep_gemm.fp4_gemm_tt(A_tt, B_tt, D_tt, sf=sf_tt)

Scaling Factor Transposition Requirements

The scaling-factor tensor (used for FP8/FP4 quantization) must always be provided in a TMA-aligned and transposed layout, regardless of the main GEMM layout. This constraint applies to both SM 90 and SM 100 architectures.

The transpose_packed_fp4 function in deep_gemm/utils/math.py handles the specific transposition logic for packed FP4 scaling factors when preparing TT layout operations.

Key Implementation Files

File	Purpose	Relevant Content
deep_gemm/__init__.py	API surface	Exposes fp8_gemm_nt, fp8_gemm_nn, fp8_gemm_tn, fp8_gemm_tt and FP4 variants【28†L28-L53】
tests/test_fp8_fp4.py	Usage examples	Demonstrates conditional transposition based on layout strings before kernel calls【25†L25-L27】【38†L38-L40】
tests/generators.py	Layout definitions	Defines MajorTypeAB enum mapping K-major (NT) vs MN-major (transposed) layouts【31†L31-L38】
deep_gemm/utils/math.py	Transposition utilities	Implements transpose_packed_fp4 for scaling-factor transposition【104†L104-L118】
README.md	Architecture constraints	Documents NT-only support on SM 90 vs full layout support on SM 100【65†L65-L66】

Summary

• NT is the default: DeepGEMM natively computes D = C + A @ Bᵀ with row-major A and column-major B; no manual transposition required.
• SM 90 limitation: Only NT kernels are available on Hopper architecture; other layouts require SM 100 (Blackwell).
• Explicit transposition required: For NN, TN, and TT layouts, you must call .T.contiguous() on the appropriate tensors before invoking *_nn, *_tn, or *_tt kernels.
• Scaling factor constraints: Quantization scaling factors must always be TMA-aligned and transposed; use transpose_packed_fp4 for FP4 TT layouts.
• Kernel naming convention: The suffix (_nt, _nn, _tn, _tt) explicitly indicates the expected input layout, matching the BLAS convention.

Frequently Asked Questions

Does DeepGEMM automatically transpose matrices for NN or TT layouts?

No. DeepGEMM does not perform implicit transpositions. You must explicitly transpose tensors using .T.contiguous() before calling the corresponding kernel (e.g., fp8_gemm_nn or fp8_gemm_tt). This design keeps the JIT compilation path lightweight and avoids runtime overhead.

Why do scaling factors need to be transposed separately from the main GEMM inputs?

The scaling-factor tensor follows its own layout constraint: it must always be TMA-aligned and transposed regardless of whether the main GEMM uses NT, NN, TN, or TT. This is a hardware requirement for the Tensor Memory Accelerator on NVIDIA GPUs. For FP4 operations in TT layouts, use the transpose_packed_fp4 utility in deep_gemm/utils/math.py to prepare the scaling factor correctly.

Can I use NN, TN, or TT layouts on older GPUs like Hopper (SM 90)?

No. The NN, TN, and TT kernels are only available on SM 100 (Blackwell) GPUs. SM 90 (Hopper) only supports the NT layout. According to the README and the kernel dispatch logic in deep_gemm/__init__.py, attempting to use non-NT layouts on SM 90 will result in compilation or runtime errors because the corresponding kernel variants are not compiled for that architecture.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}