How to Set Up Symmetric Memory Buffers for Multi-Process Mega MoE Execution in DeepGEMM

Use PyTorch's private _symmetric_memory API to allocate a shared CUDA buffer via symm_mem.empty and symm_mem.rendezvous, then slice it into input views for the fused fp8_fp4_mega_moe kernel.

DeepGEMM's Mega MoE kernel fuses expert dispatch, dual FP8×FP4 GEMMs, SwiGLU activation, and EP combine into a single high-throughput operation. To enable overlapped NVLink communication with tensor-core computation, every GPU rank must access identical memory layouts for inputs, routing metadata, and intermediate activations through symmetric memory buffers for multi-process Mega MoE execution.

Understanding Symmetric Memory in Mega MoE

Why Symmetric Buffers Are Required

The Mega MoE kernel overlaps NVLink communication with tensor-core compute. Because every rank performs simultaneous communication and computation, all processes must agree on the exact memory addresses for shared routing metadata (topk_idx, topk_weights) and intermediate activations. In deep_gemm/mega/__init__.py, the SymmBuffer class implements this contract by allocating a single symmetric memory region that is visible to every rank in the process group.

The Role of NVLink Overlap

According to the DeepGEMM source code, the kernel uses symmetric buffers to pipeline expert dispatch and combine operations across NVLink. Without symmetric memory, each rank would require explicit NCCL collectives to exchange routing information, which would stall the tensor cores. The symmetric buffer eliminates these stalls by allowing zero-copy access to shared metadata.

Allocating Symmetric Memory Buffers

Calculating Buffer Size with C++ Helpers

Before allocating GPU memory, you must compute the exact byte size required for the entire Mega MoE communication pattern. In csrc/apis/mega.hpp, the library exposes a C++ helper that calculates buffer sizes based on the MoE geometry.

# deep_gemm/mega/__init__.py

num_bytes, slice_input_buffers = _C.get_symm_buffer_size_for_mega_moe(
    group.size(), num_experts,
    num_max_tokens_per_rank, num_topk,
    hidden, intermediate_hidden,
    use_fp8_dispatch, activation
)

This function returns both the total allocation size and a callable slice_input_buffers that will later map the raw buffer to specific tensor views.

Creating the Shared Buffer Object

With the size calculated, you allocate the symmetric memory using PyTorch's private symm_mem API and register it with the process group via rendezvous.

# deep_gemm/mega/__init__.py

buffer = symm_mem.empty(num_bytes, dtype=torch.int8, device='cuda')
handle = symm_mem.rendezvous(buffer, group=group)   # <-- makes it visible to every rank

buffer.zero_()
group.barrier()
torch.cuda.synchronize()

The symm_mem.empty call allocates a CUDA-device buffer in a special symmetric memory region. The symm_mem.rendezvous call registers this buffer with the supplied torch.distributed.ProcessGroup, ensuring every rank can address the same physical memory via the returned handle.

Slicing Buffers into Tensor Views

After rendezvous, the raw byte buffer must be partitioned into typed tensors for inputs, scaling factors, and routing metadata. The slice_input_buffers callable returned earlier performs this mapping based on the C++-calculated offsets.

# deep_gemm/mega/__init__.py

(self.x, self.x_sf,
 self.topk_idx, self.topk_weights,
 self.l1_acts, self.l1_acts_sf,
 self.l2_acts, self.l2_acts_sf) = slice_input_buffers(self.buffer)

This unpacking creates typed views (x, x_sf, topk_idx, etc.) that point to specific offsets within the shared symmetric buffer. All ranks see identical layouts for these views, enabling the kernel to perform zero-copy routing lookups.

Public API for Buffer Creation

To simplify setup, DeepGEMM exposes a convenience function that encapsulates the entire allocation workflow, including alignment calculations.

# deep_gemm/mega/__init__.py

def get_symm_buffer_for_mega_moe(group, num_experts,
                                num_max_tokens_per_rank, num_topk,
                                hidden, intermediate_hidden,
                                use_fp8_dispatch=True, activation='swiglu'):
    # Align token count to the block‑M size required by the kernel

    block_m = _C.get_block_m_for_mega_moe(num_ranks, num_experts,
                                         num_max_tokens_per_rank, num_topk)
    num_max_tokens_per_rank = align(num_max_tokens_per_rank, block_m)
    return SymmBuffer(group, num_experts,
                      num_max_tokens_per_rank, num_topk,
                      hidden, intermediate_hidden,
                      use_fp8_dispatch, activation)

This helper automatically aligns the num_max_tokens_per_rank to the kernel's required block_m size using _C.get_block_m_for_mega_moe, ensuring the buffer satisfies hardware alignment constraints.

Integrating into the Mega MoE Workflow

Setting up symmetric memory buffers for multi-process Mega MoE execution follows a four-step pattern:

1. Create the buffer once per training job before the first kernel launch using get_symm_buffer_for_mega_moe.
2. Copy per-rank inputs (x, x_sf, topk_idx, topk_weights) into the sliced views of the symmetric buffer.
3. Launch the fused kernel deep_gemm.fp8_fp4_mega_moe, passing the same SymmBuffer instance to every rank.
4. Synchronize implicitly; the kernel handles internal barriers, leaving the output tensor y ready locally.

The buffer is lightweight (typically a few hundred megabytes for standard MoE configurations) and resides entirely on the GPU, eliminating extra NCCL collectives for routing metadata exchange.

Debug Mode for Buffer Validation

To detect out-of-bounds writes during development, DeepGEMM provides a debug mode controlled by the environment variable DG_COMM_KERNEL_DEBUG. When set to 1, the library zero-fills the entire symmetric buffer before each kernel call, making corrupted data immediately visible.

export DG_COMM_KERNEL_DEBUG=1

This validation occurs in deep_gemm/mega/__init__.py and is documented in the project's README.

Summary

• Symmetric memory buffers are required for Mega MoE execution because they allow all ranks to share routing metadata and intermediate activations without NCCL collectives.
• Allocation workflow: Calculate size with _C.get_symm_buffer_size_for_mega_moe, allocate with symm_mem.empty, and register with symm_mem.rendezvous.
• Slicing: Use the returned slice_input_buffers callable to map the raw byte buffer to typed tensors (x, topk_idx, etc.).
• Public helper: get_symm_buffer_for_mega_moe handles alignment and encapsulates the entire setup.
• Debug: Set DG_COMM_KERNEL_DEBUG=1 to zero-fill buffers and detect memory corruption.

Frequently Asked Questions

What is the difference between symmetric memory and regular CUDA memory?

Symmetric memory is allocated through PyTorch's private _symmetric_memory subsystem and registered across a process group via symm_mem.rendezvous. While regular CUDA memory is private to each rank, symmetric memory maps to the same physical GPU memory pages visible to all ranks in the group, enabling zero-copy access to shared routing metadata.

How does symm_mem.rendezvous ensure visibility across all ranks?

The symm_mem.rendezvous function, called in deep_gemm/mega/__init__.py, registers the allocated buffer with the provided torch.distributed.ProcessGroup. This establishes a common handle that all ranks use to address the same physical memory region, effectively creating a symmetric memory region accessible via NVLink without additional copies.

Can I reuse the same symmetric buffer for multiple Mega MoE layers?

Yes. The SymmBuffer object is designed to be allocated once per training job and reused across multiple forward passes and layers. Because the buffer is sized according to num_max_tokens_per_rank and the MoE geometry, as long as these dimensions remain constant, you can copy new inputs into the sliced views and launch the kernel repeatedly without reallocation.

What alignment requirements exist for tokens per rank?

The Mega MoE kernel requires the number of tokens per rank to be aligned to a specific block_m size determined by the GPU architecture and MoE configuration. The helper get_symm_buffer_for_mega_moe automatically aligns num_max_tokens_per_rank using _C.get_block_m_for_mega_moe and an align utility function before allocating the buffer, ensuring hardware constraints are satisfied.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}