DeepSeek-R1 Architecture: Understanding the MoE (Mixture of Experts) Setup

DeepSeek-R1 activates only 37 billion parameters during inference from its 671 billion total parameter count by routing each token through a sparse subset of expert networks, achieving full-scale model performance with fractional compute costs.

DeepSeek-R1 is an open-source reasoning model developed by DeepSeek-AI that leverages a sparse Mixture-of-Experts (MoE) architecture derived from the DeepSeek-V3 family. According to the deepseek-ai/DeepSeek-R1 repository, this design enables the model to distribute specialized computation across isolated expert sub-networks while maintaining inference efficiency through dynamic token routing.

How the DeepSeek-R1 MoE Architecture Works

The MoE architecture partitions a massive 671 billion parameter transformer into discrete functional components coordinated by a routing mechanism. As documented in README.md#L108, the system achieves 37 billion activated parameters per token—approximately 5% of the total capacity—dramatically reducing computational overhead while preserving model capability.

Core Components

The routing system comprises three primary elements:

Router: For every input token, this lightweight network computes score vectors and selects a small subset (typically 2) of experts that will actually process the token.

Experts: Each expert constitutes a full transformer block or layer group. Only the selected experts activate for a given token, ensuring computational costs remain proportional to the activated parameter count rather than the full 671B model size.

Aggregation: The outputs from selected experts combine through weighted summation before reintegration into the main transformer stream, producing the final token representation.

Why DeepSeek-R1 Uses Mixture of Experts

The MoE paradigm addresses two critical scalability challenges for modern reasoning models:

Scalability: Adding experts increases total model capacity without proportional inference cost increases, because the router maintains sparse activation patterns regardless of expert pool size.

Specialization: Individual experts develop competency in distinct domains such as mathematical reasoning, code generation, or long-context analysis. This specialization directly enables the advanced chain-of-thought capabilities reported for DeepSeek-R1.

This architectural foundation is inherited from DeepSeek-V3-Base, as noted in README.md#L78-L79.

Training Pipeline for the MoE System

DeepSeek-R1's expert routing behavior emerges through a specialized four-stage training regimen:

1. Pre-training: The base 671B parameter model trains on standard language corpora with initialized MoE routing mechanisms.

2. RL Stage 1: Reinforcement learning exposes the router to chain-of-thought tasks, rewarding the model for discovering efficient expert utilization patterns during reasoning.

3. SFT Seed: Supervised fine-tuning with high-quality reasoning data stabilizes MoE behavior and prevents routing collapse before final optimization.

4. RL Stage 2: Additional reinforcement learning refines routing decisions to align with human preferences and improve reasoning consistency across expert boundaries.

Deploying DeepSeek-R1 with MoE Inference

The repository provides deployment configurations for production frameworks capable of executing the sparse MoE computation graph.

Serving with vLLM

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
    --tensor-parallel-size 2 \
    --max-model-len 32768 \
    --enforce-eager

The --tensor-parallel-size flag distributes active expert calculations across GPUs, while --max-model-len accommodates the 128K context window supported by the MoE architecture.

Serving with SGLang

python3 -m sglang.launch_server \
    --model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
    --trust-remote-code \
    --tp 2

The --trust-remote-code parameter is required because the MoE routing logic executes through the model's custom implementation files rather than standard transformer architectures.

Prompting Recommendations

For optimal expert utilization during reasoning tasks, the repository recommends this prompt structure:

Please reason step by step, and put your final answer within \boxed{}.

Summary

• DeepSeek-R1 implements a 671B parameter MoE architecture that activates only 37B parameters per token (approximately 5% sparsity) according to README.md#L108.
• A router network dynamically selects 2 experts per token from the available pool, enabling computational specialization without full model activation.
• The training pipeline combines pre-training, two RL stages, and supervised fine-tuning to optimize routing decisions specifically for chain-of-thought reasoning.
• DeepSeek-V3-Base provides the foundational MoE implementation inherited by DeepSeek-R1 as referenced in README.md#L78-L79.
• Production deployment requires tensor parallelism across GPUs to distribute the activated expert computations efficiently.

Frequently Asked Questions

How many parameters are active during DeepSeek-R1 inference?

According to README.md#L108, DeepSeek-R1 activates approximately 37 billion parameters during inference from its total 671 billion parameter count. This represents roughly 5% of the total model capacity, enabling the performance characteristics of a dense 671B model with the computational cost equivalent to a 37B dense model.

What determines which experts process a specific token?

The router network computes a lightweight scoring vector for every input token and typically selects the top-2 experts with the highest affinity scores. This dynamic routing occurs at each MoE layer, allowing tokens to access specialized expert knowledge based on their semantic content and contextual requirements within the reasoning chain.

How does DeepSeek-R1's MoE training differ from standard transformer training?

Unlike dense transformers that update all parameters simultaneously, DeepSeek-R1's pipeline specifically optimizes routing efficiency through dedicated reinforcement learning stages. The first RL stage encourages the model to discover effective expert utilization patterns during reasoning tasks, while the second RL stage refines these decisions for human preference alignment and reasoning consistency across expert boundaries.

Can I run the full DeepSeek-R1 MoE model on consumer hardware?

The full 671B parameter model requires substantial multi-GPU server infrastructure with tensor parallelism to distribute the 37B active parameters across devices. However, the DeepSeek-R1-Distill variants retain the same MoE routing logic while significantly reducing memory requirements, making them deployable on multi-GPU setups using vLLM or SGLang as shown in the examples above.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}