# DeepEP Intranode and Internode Kernels: Architecture and Performance Differences

> Explore DeepEP intranode vs internode kernels. Understand their distinct NVLink and RDMA communication patterns and synchronization strategies for efficient distributed computing.

- Repository: [DeepSeek/DeepEP](https://github.com/deepseek-ai/DeepEP)
- Tags: architecture
- Published: 2026-04-25

---

**DeepEP intranode kernels communicate exclusively via NVLink within a single node using simple barriers, while internode kernels orchestrate RDMA traffic across nodes with NVLink forwarding inside nodes, requiring additional metadata exchange and two-level NVSHMEM synchronization.**

DeepEP is an open-source communication library developed by deepseek-ai/DeepEP that optimizes Mixture-of-Experts (MoE) token routing for distributed training. The library implements two distinct kernel families—intranode and internode—to handle different scales of GPU topology. Understanding the differences between DeepEP intranode and internode kernels is essential for optimizing multi-node training performance and debugging communication bottlenecks.

## Communication Topology and Target Use Cases

The primary distinction lies in the communication fabric each kernel family targets.

**Intranode kernels** operate exclusively within a single physical machine where all GPUs share a high-bandwidth NVLink fabric. These kernels are found in `csrc/kernels/intranode.cu` under the `deep_ep::intranode` namespace and are optimized for scenarios where the process group fits entirely within one node (e.g., an 8-GPU server).

**Internode kernels**, defined in `csrc/kernels/internode.cu` within the `deep_ep::internode` namespace, handle RDMA-based all-to-all communication across different nodes. Within each node, these kernels still leverage NVLink as a "low-latency" forwarding path, but must additionally coordinate with remote nodes via InfiniBand or similar RDMA networks.

## Metadata Handling and Buffer Structures

The complexity of metadata management differs significantly between the two implementations.

### Intranode Metadata Flow

Intranode kernels require minimal metadata: only token counts per rank and per-expert prefix sums. The `notify_dispatch` kernel writes the per-rank matrix directly into a shared buffer accessible by all GPUs in the node via NVLink.

### Internode Metadata and SourceMeta

Internode kernels introduce a **`SourceMeta`** struct (defined at lines 17-34 in `csrc/kernels/internode.cu`) that encodes the source RDMA rank and a bitmap of 8 NVLink peers. This kernel must exchange both RDMA and NVLink metadata, clean two separate buffers, and compute two distinct prefix-sum matrices: `rdma_channel_prefix_matrix` and `gbl_channel_prefix_matrix`.

The memory layout reflects this complexity: intranode uses one contiguous buffer per rank (`kNumRanks × kNumRanks` int counters + per-expert data), while internode maintains two separate layouts—the **RDMA buffer** (`rdma_buffer_ptr`) storing `SourceMeta` per token, and the **NVL buffer** (`buffer_ptrs[nvl_rank]`) holding the standard layout after forwarding.

## Synchronization Mechanisms

Barrier synchronization represents a major architectural divergence between the kernel families.

**Intranode barriers** utilize a simple intra-node barrier (`barrier_block<kNumRanks,…>`) executed once per kernel launch. All participants synchronize through shared memory operations within the PCIe/NVLink domain.

**Internode barriers** implement a more complex two-level scheme:

1. An intra-node NVLink barrier for the first warp
2. An inter-node **NVSHMEM** barrier (`nvshmem_sync`) for the second warp
3. Additional per-step barriers for RDMA metadata exchange

This dual-barrier approach accounts for the heterogeneous latency characteristics of NVLink (microseconds) versus RDMA network paths.

## Communication Primitives and Dispatch Flow

The actual data movement primitives differ substantially between the kernel families.

### Intranode Communication

Intranode kernels rely exclusively on NVLink-based store/load operations (`st_relaxed_sys_global`, `ld_volatile_global`) and intra-node atomic primitives. The dispatch flow follows a simple pattern: *Notify → Prefix-sum → Send → Receive*, with all data remaining within the node's NVLink fabric.

### Internode Communication

Internode kernels utilize **NVSHMEM/IBGDA** calls including:

- `nvshmemi_ibgda_put_nbi_warp` for non-blocking puts
- `nvshmemi_ibgda_quiet` for completion fencing
- `nvshmemi_ibgda_amo_nonfetch_add` for atomic operations

The dispatch flow adds complexity: *Notify (RDMA meta) → RDMA put → NVL forwarder → NVL receive → Prefix-sum*. The forwarder stage copies data from the RDMA buffer to the local NVLink buffer before reusing the standard intranode dispatch path.

## Kernel Launch Configuration

The launch configurations reflect the topology differences:

- **Intranode**: `SETUP_LAUNCH_CONFIG(1 + num_ranks, …)`—one SM for the control path and one SM per rank
- **Internode**: `SETUP_LAUNCH_CONFIG(1 + num_rdma_ranks, …)`—one SM for control and one SM per RDMA rank (where each RDMA rank groups 8 NVL peers)

This configuration acknowledges that internode communication is bottlenecked by the number of RDMA network interfaces rather than individual GPUs.

## Low-Latency Mode

Internode kernels support an optional `low_latency_mode` flag exposed as a template parameter `kLowLatencyMode`. When enabled, this mode collapses RDMA and NVLink into a single address space, eliminating the forwarder stage.

In this mode, the kernel uses `translate_dst_rdma_rank<true>` to map a remote RDMA rank plus NVL lane to a unique global rank, effectively treating the entire cluster as a flat NVLink domain from the perspective of address translation. This reduces latency by avoiding the explicit copy between RDMA and NVL buffers.

## Python API and Automatic Selection

Despite these internal differences, the Python API remains identical for both paths.

```python
import torch
import deep_ep

# Setup works for both intranode and internode

group = torch.distributed.init_process_group(backend="nccl")
buffer = deep_ep.Buffer(
    group,
    int(2e9),                # NVL buffer size (bytes)

    num_rdma_bytes=0,        # 0 triggers intranode; >0 triggers internode

    low_latency_mode=False
)

# Dispatch operation

recv_x, recv_topk_idx, recv_topk_weights, recv_counts, handle, event = \
    buffer.dispatch(
        x=x,
        num_tokens_per_rank=num_tokens_per_rank,
        is_token_in_rank=is_token_in_rank,
        num_tokens_per_expert=num_tokens_per_expert,
        topk_idx=topk_idx,
        topk_weights=topk_weights,
        config=deep_ep.Config(num_sms, nvl_chunk, nvl_buf),
        async_finish=False
    )

# Combine operation

combined_x, combined_weights, event = buffer.combine(
    x=recv_x,
    handle=handle,
    topk_weights=recv_topk_weights,
    config=deep_ep.Config(num_sms, nvl_chunk, nvl_buf),
    async_finish=False
)

```

The library detects topology at runtime via the `num_rdma_bytes` parameter and `num_ranks` relative to GPUs per node. When `num_rdma_bytes` is non-zero or the process group spans multiple nodes, `buffer.dispatch` launches the internode kernels; otherwise, it selects the intranode path.

## Performance Characteristics

**Intranode** latency is dominated by NVLink bandwidth and intra-node barrier cost, typically measured in microseconds. The uniform memory model allows for predictable performance without network jitter.

**Internode** latency includes additional RDMA round-trip costs, extra metadata-exchange overhead, and potential NIC-to-GPU copy costs. The "low-latency" mode mitigates this by packing RDMA and NVL communication into a single address space, but performance remains bound by InfiniBand latency and congestion.

## Summary

- **Intranode kernels** in `csrc/kernels/intranode.cu` handle NVLink-only communication within single nodes using simple barriers and contiguous buffers.
- **Internode kernels** in `csrc/kernels/internode.cu` manage RDMA traffic across nodes with NVLink forwarding, requiring `SourceMeta` structs, dual barriers, and separate RDMA/NVL buffers.
- **Synchronization** differs from single `barrier_block` calls to two-level NVLink+NVSHMEM barriers.
- **Python API** abstracts both paths through `deep_ep.Buffer` with automatic kernel selection based on `num_rdma_bytes` and topology detection.
- **Low-latency mode** collapses the RDMA/NVL boundary for internode kernels, eliminating the forwarder stage when hardware permits.

## Frequently Asked Questions

### When does DeepEP automatically select internode kernels over intranode kernels?

DeepEP selects internode kernels when the process group spans multiple physical nodes or when `num_rdma_bytes` is set to a non-zero value during `Buffer` initialization. Internally, the library compares the total number of ranks against the number of GPUs per node; if the rank count exceeds local GPU capacity, the internode path in `csrc/kernels/internode.cu` is activated to handle cross-node RDMA communication.

### What is the purpose of the SourceMeta struct in internode kernels?

The `SourceMeta` struct, defined at lines 17-34 in `csrc/kernels/internode.cu`, encodes the source RDMA rank and a bitmap representing which of the 8 NVLink peers within a node should receive the token. This metadata enables the internode dispatch kernel to route tokens correctly across the RDMA network and then distribute them to the appropriate local GPU via NVLink, managing the dual-buffer architecture required for cross-node communication.

### How does the low-latency mode affect internode kernel performance?

Low-latency mode, controlled by the template parameter `kLowLatencyMode`, eliminates the forwarder stage that copies data between RDMA and NVLink buffers. By collapsing these address spaces and using `translate_dst_rdma_rank<true>` for direct addressing, the kernel reduces latency overhead. This mode is most effective when the network topology allows direct RDMA-to-GPU access without intermediate copies, though it requires specific hardware support for optimal performance.

### What are the key file paths for understanding DeepEP kernel implementations?

The core implementations reside in `csrc/kernels/intranode.cu` for single-node NVLink operations and `csrc/kernels/internode.cu` for multi-node RDMA orchestration. The Python wrappers that handle automatic kernel selection are located in [`deep_ep/buffer.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/buffer.py), while [`deep_ep/__init__.py`](https://github.com/deepseek-ai/DeepEP/blob/main/deep_ep/__init__.py) exposes the public API including `deep_ep.Config` and `deep_ep.Buffer`.