# How to Set Up Symmetric Memory Buffers for Multi-Process Mega MoE Execution in DeepGEMM

> Learn to set up symmetric memory buffers for multi-process Mega MoE execution in DeepGEMM. Use PyTorch's private API to allocate shared CUDA buffers and create input views for the fused kernel.

- Repository: [DeepSeek/DeepGEMM](https://github.com/deepseek-ai/DeepGEMM)
- Tags: how-to-guide
- Published: 2026-04-19

---

**Use PyTorch's private `_symmetric_memory` API to allocate a shared CUDA buffer via `symm_mem.empty` and `symm_mem.rendezvous`, then slice it into input views for the fused `fp8_fp4_mega_moe` kernel.**

DeepGEMM's Mega MoE kernel fuses expert dispatch, dual FP8×FP4 GEMMs, SwiGLU activation, and EP combine into a single high-throughput operation. To enable overlapped NVLink communication with tensor-core computation, every GPU rank must access identical memory layouts for inputs, routing metadata, and intermediate activations through **symmetric memory buffers for multi-process Mega MoE execution**.

## Understanding Symmetric Memory in Mega MoE

### Why Symmetric Buffers Are Required

The Mega MoE kernel overlaps NVLink communication with tensor-core compute. Because every rank performs simultaneous communication and computation, all processes must agree on the exact memory addresses for shared routing metadata (`topk_idx`, `topk_weights`) and intermediate activations. In [`deep_gemm/mega/__init__.py`](https://github.com/deepseek-ai/DeepGEMM/blob/main/deep_gemm/mega/__init__.py), the `SymmBuffer` class implements this contract by allocating a single symmetric memory region that is visible to every rank in the process group.

### The Role of NVLink Overlap

According to the DeepGEMM source code, the kernel uses symmetric buffers to pipeline expert dispatch and combine operations across NVLink. Without symmetric memory, each rank would require explicit NCCL collectives to exchange routing information, which would stall the tensor cores. The symmetric buffer eliminates these stalls by allowing zero-copy access to shared metadata.

## Allocating Symmetric Memory Buffers

### Calculating Buffer Size with C++ Helpers

Before allocating GPU memory, you must compute the exact byte size required for the entire Mega MoE communication pattern. In [`csrc/apis/mega.hpp`](https://github.com/deepseek-ai/DeepGEMM/blob/main/csrc/apis/mega.hpp), the library exposes a C++ helper that calculates buffer sizes based on the MoE geometry.

```python

# deep_gemm/mega/__init__.py

num_bytes, slice_input_buffers = _C.get_symm_buffer_size_for_mega_moe(
    group.size(), num_experts,
    num_max_tokens_per_rank, num_topk,
    hidden, intermediate_hidden,
    use_fp8_dispatch, activation
)

```

This function returns both the total allocation size and a callable `slice_input_buffers` that will later map the raw buffer to specific tensor views.

### Creating the Shared Buffer Object

With the size calculated, you allocate the symmetric memory using PyTorch's private `symm_mem` API and register it with the process group via `rendezvous`.

```python

# deep_gemm/mega/__init__.py

buffer = symm_mem.empty(num_bytes, dtype=torch.int8, device='cuda')
handle = symm_mem.rendezvous(buffer, group=group)   # <-- makes it visible to every rank

buffer.zero_()
group.barrier()
torch.cuda.synchronize()

```

The `symm_mem.empty` call allocates a CUDA-device buffer in a special symmetric memory region. The `symm_mem.rendezvous` call registers this buffer with the supplied `torch.distributed.ProcessGroup`, ensuring every rank can address the same physical memory via the returned handle.

## Slicing Buffers into Tensor Views

After rendezvous, the raw byte buffer must be partitioned into typed tensors for inputs, scaling factors, and routing metadata. The `slice_input_buffers` callable returned earlier performs this mapping based on the C++-calculated offsets.

```python

# deep_gemm/mega/__init__.py

(self.x, self.x_sf,
 self.topk_idx, self.topk_weights,
 self.l1_acts, self.l1_acts_sf,
 self.l2_acts, self.l2_acts_sf) = slice_input_buffers(self.buffer)

```

This unpacking creates typed views (`x`, `x_sf`, `topk_idx`, etc.) that point to specific offsets within the shared symmetric buffer. All ranks see identical layouts for these views, enabling the kernel to perform zero-copy routing lookups.

## Public API for Buffer Creation

To simplify setup, DeepGEMM exposes a convenience function that encapsulates the entire allocation workflow, including alignment calculations.

```python

# deep_gemm/mega/__init__.py

def get_symm_buffer_for_mega_moe(group, num_experts,
                                num_max_tokens_per_rank, num_topk,
                                hidden, intermediate_hidden,
                                use_fp8_dispatch=True, activation='swiglu'):
    # Align token count to the block‑M size required by the kernel

    block_m = _C.get_block_m_for_mega_moe(num_ranks, num_experts,
                                         num_max_tokens_per_rank, num_topk)
    num_max_tokens_per_rank = align(num_max_tokens_per_rank, block_m)
    return SymmBuffer(group, num_experts,
                      num_max_tokens_per_rank, num_topk,
                      hidden, intermediate_hidden,
                      use_fp8_dispatch, activation)

```

This helper automatically aligns the `num_max_tokens_per_rank` to the kernel's required `block_m` size using `_C.get_block_m_for_mega_moe`, ensuring the buffer satisfies hardware alignment constraints.

## Integrating into the Mega MoE Workflow

Setting up symmetric memory buffers for multi-process Mega MoE execution follows a four-step pattern:

1. **Create the buffer once** per training job before the first kernel launch using `get_symm_buffer_for_mega_moe`.
2. **Copy per-rank inputs** (`x`, `x_sf`, `topk_idx`, `topk_weights`) into the sliced views of the symmetric buffer.
3. **Launch the fused kernel** `deep_gemm.fp8_fp4_mega_moe`, passing the same `SymmBuffer` instance to every rank.
4. **Synchronize implicitly**; the kernel handles internal barriers, leaving the output tensor `y` ready locally.

The buffer is lightweight (typically a few hundred megabytes for standard MoE configurations) and resides entirely on the GPU, eliminating extra NCCL collectives for routing metadata exchange.

## Debug Mode for Buffer Validation

To detect out-of-bounds writes during development, DeepGEMM provides a debug mode controlled by the environment variable `DG_COMM_KERNEL_DEBUG`. When set to `1`, the library zero-fills the entire symmetric buffer before each kernel call, making corrupted data immediately visible.

```bash
export DG_COMM_KERNEL_DEBUG=1

```

This validation occurs in [`deep_gemm/mega/__init__.py`](https://github.com/deepseek-ai/DeepGEMM/blob/main/deep_gemm/mega/__init__.py) and is documented in the project's README.

## Summary

- **Symmetric memory buffers** are required for Mega MoE execution because they allow all ranks to share routing metadata and intermediate activations without NCCL collectives.
- **Allocation workflow**: Calculate size with `_C.get_symm_buffer_size_for_mega_moe`, allocate with `symm_mem.empty`, and register with `symm_mem.rendezvous`.
- **Slicing**: Use the returned `slice_input_buffers` callable to map the raw byte buffer to typed tensors (`x`, `topk_idx`, etc.).
- **Public helper**: `get_symm_buffer_for_mega_moe` handles alignment and encapsulates the entire setup.
- **Debug**: Set `DG_COMM_KERNEL_DEBUG=1` to zero-fill buffers and detect memory corruption.

## Frequently Asked Questions

### What is the difference between symmetric memory and regular CUDA memory?

Symmetric memory is allocated through PyTorch's private `_symmetric_memory` subsystem and registered across a process group via `symm_mem.rendezvous`. While regular CUDA memory is private to each rank, symmetric memory maps to the same physical GPU memory pages visible to all ranks in the group, enabling zero-copy access to shared routing metadata.

### How does symm_mem.rendezvous ensure visibility across all ranks?

The `symm_mem.rendezvous` function, called in [`deep_gemm/mega/__init__.py`](https://github.com/deepseek-ai/DeepGEMM/blob/main/deep_gemm/mega/__init__.py), registers the allocated buffer with the provided `torch.distributed.ProcessGroup`. This establishes a common handle that all ranks use to address the same physical memory region, effectively creating a symmetric memory region accessible via NVLink without additional copies.

### Can I reuse the same symmetric buffer for multiple Mega MoE layers?

Yes. The `SymmBuffer` object is designed to be allocated once per training job and reused across multiple forward passes and layers. Because the buffer is sized according to `num_max_tokens_per_rank` and the MoE geometry, as long as these dimensions remain constant, you can copy new inputs into the sliced views and launch the kernel repeatedly without reallocation.

### What alignment requirements exist for tokens per rank?

The Mega MoE kernel requires the number of tokens per rank to be aligned to a specific `block_m` size determined by the GPU architecture and MoE configuration. The helper `get_symm_buffer_for_mega_moe` automatically aligns `num_max_tokens_per_rank` using `_C.get_block_m_for_mega_moe` and an `align` utility function before allocating the buffer, ensuring hardware constraints are satisfied.