How to Implement Custom Expert Routing Strategies in the DeepSeek-V3 Gate Module

You can customize expert routing in DeepSeek-V3 by modifying ModelArgs hyperparameters, subclassing the Gate class in inference/model.py, or replacing the linear projection with alternative scoring mechanisms like MLPs or hash-based routing.

The Gate module in DeepSeek-V3 serves as the routing core for the Mixture-of-Experts (MoE) layer, determining which experts process each token. Located in inference/model.py at lines 535-599, this component scores routed experts, selects top-k candidates, and returns routing weights and indices. Understanding how to implement custom expert routing strategies in this module allows researchers to experiment with novel MoE architectures without modifying the broader model infrastructure.

Understanding the Gate Module Architecture

Core Components in inference/model.py

The Gate class is defined between lines 535 and 599 in inference/model.py. During initialization, it reads hyper-parameters from the ModelArgs dataclass (lines 19-54) and creates the learnable routing matrix weight along with an optional bias parameter.

Key attributes include:

• weight: The linear projection matrix mapping hidden states to expert scores.
• bias: Optional bias added to raw scores before selection.
• score_func: Either "softmax" or "sigmoid", determining how raw logits are normalized.
• route_scale: A scalar multiplier applied to final routing weights.

The Forward Pass Flow

The forward method (lines 566-599) executes the routing logic in six distinct stages:

1. Linear projection: Computes raw scores via linear(x, self.weight).
2. Scoring function: Applies softmax or sigmoid to obtain normalized probabilities.
3. Bias addition: Adds learned bias if initialized.
4. Group-wise selection: When n_groups > 1, reshapes scores to [batch, groups, experts_per_group], selects top groups (topk_groups), and masks others.
5. Top-k expert selection: Picks the final topk experts per token using torch.topk.
6. Weight normalization: Normalizes sigmoid scores, scales by route_scale, and returns (weights, indices).

Method 1: Configuring Routing via ModelArgs

The simplest way to alter routing behavior is to modify the hyper-parameters passed to ModelArgs during model construction. These values automatically propagate into Gate.__init__ and Gate.forward.

from inference.model import ModelArgs, MoE

args = ModelArgs(
    n_routed_experts=64,
    n_activated_experts=4,      # Reduce from default for sparser routing

    n_expert_groups=2,            # Enable hierarchical group selection

    n_limited_groups=1,           # Select only 1 group before top-k

    score_func="sigmoid",         # Switch from softmax to sigmoid scoring

    route_scale=0.75,             # Dampen routing weight magnitudes

)
model = MoE(args)

# Verify the configuration

print(f"Gate score function: {model.gate.score_func}")
print(f"Top-k experts: {model.gate.topk}")

This approach requires no subclassing and is ideal for ablation studies on routing density and scoring functions.

Method 2: Subclassing Gate for Advanced Customization

When linear projection proves insufficient, subclass Gate to inject complex scoring mechanisms while preserving the original grouping and top-k selection logic.

Implementing an MLP-Based Scorer

Replace the single linear layer with a multi-layer perceptron to capture non-linear relationships between hidden states and expert affinity:

import torch
import torch.nn as nn
from inference.model import Gate, ModelArgs, MoE

class MLPGate(Gate):
    def __init__(self, args: ModelArgs):
        super().__init__(args)
        # Replace linear projection with three-layer MLP

        self.scorer = nn.Sequential(
            nn.Linear(args.dim, args.dim // 2),
            nn.ReLU(),
            nn.Linear(args.dim // 2, args.n_routed_experts),
        )
        # Re-initialize bias to match new output dimension

        self.bias = nn.Parameter(torch.zeros(args.n_routed_experts))

    def forward(self, x: torch.Tensor):
        # Compute scores via MLP

        scores = self.scorer(x)
        
        # Apply scoring function

        if self.score_func == "softmax":
            scores = scores.softmax(dim=-1, dtype=torch.float32)
        else:
            scores = scores.sigmoid()
        
        original_scores = scores
        if self.bias is not None:
            scores = scores + self.bias
        
        # Re-use grouping and top-k logic from base implementation

        if self.n_groups > 1:
            scores = scores.view(x.size(0), self.n_groups, -1)
            if self.bias is None:
                group_scores = scores.amax(dim=-1)
            else:
                group_scores = scores.topk(2, dim=-1)[0].sum(dim=-1)
            indices = group_scores.topk(self.topk_groups, dim=-1)[1]
            mask = scores.new_ones(x.size(0), self.n_groups, dtype=bool).scatter_(1, indices, False)
            scores = scores.masked_fill_(mask.unsqueeze(-1), float("-inf")).flatten(1)
        
        # Final top-k selection and normalization

        indices = torch.topk(scores, self.topk, dim=-1)[1]
        weights = original_scores.gather(1, indices)
        if self.score_func == "sigmoid":
            weights = weights / weights.sum(dim=-1, keepdim=True)
        weights = weights * self.route_scale
        return weights.type_as(x), indices

# Usage

args = ModelArgs()
moe = MoE(args)
moe.gate = MLPGate(args)  # Plug in custom gate

x = torch.randn(12, args.dim)
out = moe(x)
print(out.shape)  # torch.Size([12, args.dim])

Adding Deterministic Hash Routing

For experimental routing that bypasses learned parameters entirely, implement a hash-based router:

import torch
import torch.nn as nn
from inference.model import MoE, ModelArgs

class HashGate(nn.Module):
    """
    Deterministic hash-based router that assigns tokens to experts
    based on token ID hash values rather than learned scores.
    """
    def __init__(self, args: ModelArgs):
        super().__init__()
        self.topk = args.n_activated_experts
        self.n_experts = args.n_routed_experts

    def forward(self, x: torch.Tensor):
        # Assume first column contains token identifiers

        token_ids = x[:, 0].long()
        expert_ids = torch.arange(self.n_experts, device=x.device)
        
        # Compute XOR hash between token and each expert

        hashes = (token_ids.unsqueeze(1) ^ expert_ids).float()
        
        # Select experts with lowest hash values (deterministic)

        indices = hashes.topk(self.topk, largest=False).indices
        weights = torch.full_like(indices, 1.0 / self.topk, dtype=x.dtype)
        return weights, indices

# Integration

args = ModelArgs()
moe = MoE(args)
moe.gate = HashGate(args)

x = torch.randn(5, args.dim)
out = moe(x)
print(out.shape)

Method 3: Full Gate Replacement

When the routing algorithm diverges completely from the original top-k paradigm (e.g., reinforcement learning-based routing or k-means clustering), create a standalone module adhering to the interface contract forward(x) -> (weights, indices):

class ReinforcementLearningGate(nn.Module):
    def __init__(self, args: ModelArgs):
        super().__init__()
        self.policy_network = nn.Sequential(
            nn.Linear(args.dim, args.dim),
            nn.ReLU(),
            nn.Linear(args.dim, args.n_routed_experts)
        )
        self.topk = args.n_activated_experts
        
    def forward(self, x):
        logits = self.policy_network(x)
        # Sample or select based on policy

        weights, indices = torch.topk(torch.softmax(logits, dim=-1), self.topk)
        return weights, indices

The MoE class (lines 636-694 in inference/model.py) consumes the Gate output via self.gate(x), dispatching tokens to experts based solely on the returned indices and weights. This decoupling ensures that any object satisfying the return signature integrates seamlessly with the existing expert dispatch infrastructure.

Summary

• The Gate module in inference/model.py (lines 535-599) controls all routing decisions in DeepSeek-V3's MoE layer through a linear projection, scoring function, and top-k selection.
• Hyper-parameter tuning via ModelArgs offers the fastest customization path, adjusting n_activated_experts, score_func, and group routing settings without code changes.
• Subclassing Gate preserves the original grouping and normalization logic while allowing custom scoring mechanisms, such as MLP-based scorers or hash-based deterministic routing.
• Full Gate replacement is possible when implementing radically different algorithms (e.g., RL-based or clustering approaches), provided the new module returns (weights, indices) tensors.
• The MoE class (lines 636-694) remains agnostic to the specific routing implementation, consuming only the Gate's output tuple.

Frequently Asked Questions

What is the default routing strategy in DeepSeek-V3?

The default strategy uses a learned linear projection followed by either softmax or sigmoid scoring, then selects the top-k experts per token. When n_expert_groups is greater than 1, the Gate first selects the best groups using either max-scoring or top-2 aggregation before choosing final experts within those groups, as implemented in inference/model.py lines 566-599.

Can I use reinforcement learning to train the Gate module?

Yes. You can implement a custom nn.Module that replaces the standard Gate and includes a policy network trained via RL. Ensure your forward method returns a tuple of (weights, indices) compatible with the MoE class dispatch mechanism. The existing infrastructure in inference/model.py supports this because MoE.forward only depends on the Gate's output signature, not its internal training procedure.

How does group-wise routing affect performance?

Group-wise routing reduces computational overhead by first selecting a subset of expert groups (controlled by n_limited_groups) before performing the final top-k selection within those groups. This hierarchical approach limits the number of experts evaluated during the second stage, improving inference speed at the potential cost of routing accuracy if the group selection is too restrictive.

Where are the Gate weights stored in the checkpoint?

The Gate's linear projection weights and bias are mapped to specific tensor names during checkpoint conversion. You can verify the exact naming convention in inference/convert.py (lines 20-25), which defines the mapping between the original training checkpoint keys and the inference model's parameter names used by the Gate module.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}