How Many Activated Parameters Does DeepSeek-R1 Use in MoE?

DeepSeek-R1 activates approximately 37 billion parameters during inference through its Mixture-of-Experts (MoE) architecture.

DeepSeek-R1, developed by DeepSeek-AI, implements a sparse Mixture-of-Experts (MoE) architecture that selectively activates a subset of its total parameters during forward passes. According to the source code repository at deepseek-ai/DeepSeek-R1, both the Zero and full R1 variants utilize this MoE design to balance computational efficiency with model capacity. The specific count of activated parameters is documented in the repository's model configuration table.

Understanding DeepSeek-R1's MoE Architecture

DeepSeek-R1 employs a Mixture-of-Experts (MoE) architecture, a sparse neural network design that routes each input token to a specific subset of specialized "expert" networks rather than activating the full parameter set. This approach allows the model to scale total capacity while maintaining reasonable inference costs.

Both the DeepSeek-R1-Zero and the full DeepSeek-R1 variants implement this MoE structure. According to the repository's documentation at README.md#L109, the model summary table explicitly lists the activated parameter count in the # Activated Params column. For the MoE models, this value is 37B (37 billion parameters), representing the subset of total weights engaged during a single forward pass.

Locating the Activated Parameter Count in the Repository

The definitive source for architecture specifications resides in the repository's primary documentation file.

README.md Model Summary Table

In README.md, the maintainers provide a comprehensive model summary table that delineates critical architecture statistics. The # Activated Params column specifies 37B for the MoE configuration, confirming that DeepSeek-R1 activates approximately 37 billion parameters during inference. This sparse activation pattern distinguishes MoE models from dense architectures where every parameter participates in every computation.

Verifying Activated Parameters Programmatically

You can programmatically verify the activated parameter count using the Hugging Face Hub API to inspect the model's metadata.

from huggingface_hub import ModelInfo, model_info

# Load model metadata from the hub

info: ModelInfo = model_info("deepseek-ai/DeepSeek-R1")

# The "model_card_data" field often contains architecture details.

# Here we just print the description where activated params are mentioned.

print(info.model_card_data.get("model_description", "No description found"))

Executing this script retrieves the model card metadata, which includes the architecture description referencing the 37 billion activated parameters. This programmatic approach allows automated validation of model specifications without manual repository inspection.

Key Source Files for Architecture Details

Several files in the deepseek-ai/DeepSeek-R1 repository provide authoritative information about the MoE implementation:

• README.md: Contains the model-summary table with the # Activated Params entry (37B) for the MoE architecture at line 109.

• DeepSeek_R1.pdf: The original research paper detailing the MoE design, routing mechanisms, and parameter allocation strategies.

• LICENSE: Governs the terms for using and redistributing the model weights and inference code.

These files collectively confirm that DeepSeek-R1's inference process activates roughly 37 billion parameters from its larger total parameter pool.

Summary

• DeepSeek-R1 utilizes a Mixture-of-Experts (MoE) architecture that activates 37 billion parameters during inference.
• Both the R1-Zero and full R1 variants share this 37B activated parameter configuration.
• The README.md file at line 109 explicitly documents this value in the model summary table.
• You can verify this specification programmatically using the Hugging Face Hub API to query model metadata.
• The architecture balances computational efficiency by engaging only a sparse subset of experts per token.

Frequently Asked Questions

What is the total parameter count of DeepSeek-R1?

While DeepSeek-R1 activates 37 billion parameters during inference, the total parameter count is significantly larger. The MoE architecture maintains a sparse set of expert networks, allowing the model to possess substantial total capacity while only engaging 37B parameters per forward pass.

Does DeepSeek-R1 Zero use the same activated parameter count?

Yes, both DeepSeek-R1-Zero and the full DeepSeek-R1 variant employ the same MoE configuration with 37 billion activated parameters. The repository documentation confirms this consistency across both model versions in the architecture summary table.

How does MoE architecture reduce computational costs?

MoE architectures reduce costs by implementing conditional computation, where only specific expert networks activate for each input token. Instead of utilizing all model parameters for every prediction, DeepSeek-R1 routes tokens to specialized experts, limiting active computation to the 37 billion parameter subset while maintaining access to broader knowledge across the full expert pool.

Where can I find the official paper detailing the MoE design?

The complete technical specifications and architectural rationale are documented in DeepSeek_R1.pdf, located in the repository root. This paper explains the routing algorithms, expert allocation strategies, and the specific implementation details that result in the 37 billion activated parameter configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}