Implementing Mixed Expert (MoE) Layers in Transformers: A Complete Guide to the LLMs-from-Scratch Implementation

Implementing Mixed Expert (MoE) layers in transformers involves routing each token to a subset of feed-forward experts using a gating network, which reduces computational cost while maintaining model capacity—implemented in the LLMs-from-scratch repository through the MoEFeedForward class in pkg/llms_from_scratch/qwen3.py and a pedagogical version in ch04/07_moe/gpt_with_kv_moe.py.

The LLMs-from-scratch repository by Sebastian Raschka provides production-ready and educational implementations of transformer architectures, including a complete Mixture-of-Experts (MoE) mechanism that allows models to scale parameter count without proportional increases in inference cost. This article examines how the repository implements MoE layers, from the core gating logic to memory-efficient expert routing.

Architecture Overview of MoE Layers

Mixed Expert (MoE) layers replace the single dense feed-forward network (FFN) in transformer blocks with multiple parallel expert sub-networks. Each token is routed to only a few experts (typically 2), keeping compute costs low while the total parameter count scales with the number of experts.

The repository provides two implementations for implementing Mixed Expert (MoE) layers in transformers:

1. pkg/llms_from_scratch/qwen3.py – Production-grade MoE implementation matching the Qwen-3 architecture
2. ch04/07_moe/gpt_with_kv_moe.py – Educational step-by-step implementation used in Chapter 4

The MoE Feed-Forward Layer Implementation

Production Implementation in qwen3.py

In pkg/llms_from_scratch/qwen3.py, the MoEFeedForward class (lines 13–31 and 62–64) implements sparse expert routing with the following components:

# https://github.com/rasbt/LLMs-from-scratch/blob/main/pkg/llms_from_scratch/qwen3.py#L13-L31

# https://github.com/rasbt/LLMs-from-scratch/blob/main/pkg/llms_from_scratch/qwen3.py#L62-L64

The class stores:

• self.gate – A linear layer that produces routing scores for each expert
• num_experts parallel linear layers (fc1, fc2, fc3) representing the expert networks

In the forward method, the gating network produces scores, torch.topk selects the top-k experts per token, and softmax normalization generates routing probabilities. Tokens are grouped by expert, processed in parallel, and aggregated using weighted sums based on the routing probabilities.

Educational Implementation in gpt_with_kv_moe.py

For learning purposes, ch04/07_moe/gpt_with_kv_moe.py provides an identical architectural pattern with explicit implementation details:

# https://github.com/rasbt/LLMs-from-scratch/blob/main/ch04/07_moe/gpt_with_kv_moe.py#L59-L66

# https://github.com/rasbt/LLMs-from-scratch/blob/main/ch04/07_moe/gpt_with_kv_moe.py#L86-L99

This version uses nn.ModuleList containers for expert layers and implements the same routing, selection, and weighted aggregation logic, making it ideal for understanding the algorithm before moving to the optimized production code.

Core Algorithmic Steps for MoE Routing

The implementation follows a strict pipeline for token routing:

Gate computation – scores = self.gate(x) produces unnormalized scores for each expert per token (qwen3.py:L29-L30)

Top-k selection – topk_scores, topk_indices = torch.topk(scores, self.num_experts_per_tok, dim=-1) selects the most relevant experts (qwen3.py:L30-L31)

Routing probabilities – topk_probs = torch.softmax(topk_scores, dim=-1) normalizes selected scores into probabilities (qwen3.py:L31-L32)

Group tokens by expert – unique_experts = torch.unique(topk_indices_flat) iterates only over actually selected experts, maximizing efficiency (qwen3.py:L40-L43)

Expert forward pass – For each selected expert: hidden = silu(fc1[e](inp)) * fc2[e](inp) followed by out = fc3[e](hidden) (qwen3.py:L54-L56)

Weighted aggregation – out_flat.index_add_(0, selected_idx, expert_out * selected_probs.unsqueeze(-1)) combines expert contributions using routing probabilities (qwen3.py:L61-L62)

Reshape – Returns tensor to shape (batch, seq_len, emb_dim) (qwen3.py:L63)

Integrating MoE into the Transformer Block

The TransformerBlock class (lines 66–80 in qwen3.py) conditionally instantiates either FeedForward (standard) or MoEFeedForward (Mixture-of-Experts) based on the presence of cfg["num_experts"]:

# https://github.com/rasbt/LLMs-from-scratch/blob/main/pkg/llms_from_scratch/qwen3.py#L66-L80

This design maintains interface compatibility—the MoE block receives hidden representations of shape (batch, seq_len, emb_dim) and returns identical shapes, enabling seamless drop-in replacement without modifying attention mechanisms or layer normalization.

Memory Efficiency and Active Parameters

A critical advantage of implementing Mixed Expert (MoE) layers in transformers is the reduction in active parameters per token. Since only num_experts_per_tok experts are evaluated per token, the compute cost scales with the selected subset rather than the total expert count.

The repository includes ch04/07_moe/memory_estimator_moe.py to calculate both total and active parameter counts:

# https://github.com/rasbt/LLMs-from-scratch/blob/main/ch04/07_moe/memory_estimator_moe.py

This tool helps researchers quantify the memory savings when scaling the number of experts, ensuring efficient hardware utilization during training and inference.

Practical Code Examples

Instantiating a Small MoE Transformer

Configure a Qwen-3 model with MoE layers using the production implementation:

import torch
from pkg.llms_from_scratch.qwen3 import Qwen3Model

# Minimal MoE configuration (2-layer, 4 experts, 2 experts per token)

cfg = {
    "vocab_size": 1024,
    "context_length": 512,
    "emb_dim": 256,
    "n_heads": 8,
    "n_layers": 2,
    "hidden_dim": 512,
    "head_dim": 32,
    "qk_norm": True,
    "n_kv_groups": 2,
    "rope_base": 1_000_000.0,
    "dtype": torch.bfloat16,
    "num_experts": 4,           # Enable MoE

    "num_experts_per_tok": 2,
    "moe_intermediate_size": 768,
}
model = Qwen3Model(cfg)

# Dummy token IDs (batch=1, seq_len=10)

tokens = torch.randint(0, cfg["vocab_size"], (1, 10))
logits = model(tokens)          # shape: (1, 10, vocab_size)

print(logits.shape)

Direct Use of the MoE Feed-Forward Block

Test the MoEFeedForward class independently:

import torch
from pkg.llms_from_scratch.qwen3 import MoEFeedForward

cfg = {
    "emb_dim": 256,
    "num_experts": 4,
    "num_experts_per_tok": 2,
    "moe_intermediate_size": 768,
    "dtype": torch.float32,
}
moe = MoEFeedForward(cfg)

x = torch.randn(2, 5, cfg["emb_dim"])   # (batch, seq_len, emb_dim)

y = moe(x)                              # (2, 5, 256)

print(y.shape)

Comparing Compute Cost with Dense FFN

Benchmark sparse MoE against dense feed-forward networks:

import torch, time
from pkg.llms_from_scratch.qwen3 import FeedForward, MoEFeedForward

dense_cfg = {"emb_dim": 256, "hidden_dim": 1024, "dtype": torch.float32}
moe_cfg   = {"emb_dim": 256, "num_experts": 8,
             "num_experts_per_tok": 2,
             "moe_intermediate_size": 512,
             "dtype": torch.float32}

dense = FeedForward(dense_cfg)
moe   = MoEFeedForward(moe_cfg)

x = torch.randn(4, 128, 256)

# Warm-up

for _ in range(10):
    dense(x); moe(x)

t_dense = time.time()
for _ in range(100):
    dense(x)
t_dense = time.time() - t_dense

t_moe = time.time()
for _ in range(100):
    moe(x)
t_moe = time.time() - t_moe

print(f"Dense FFN time: {t_dense:.3f}s, MoE time: {t_moe:.3f}s")

This comparison demonstrates the speed advantage when evaluating only 2 experts per token versus a dense layer with equivalent total parameters.

Summary

• The LLMs-from-scratch repository provides both production (pkg/llms_from_scratch/qwen3.py) and educational (ch04/07_moe/gpt_with_kv_moe.py) implementations of MoE layers
• MoEFeedForward uses a gating network (self.gate) and torch.topk to route tokens to the most relevant experts
• Only num_experts_per_tok experts are activated per token, reducing active compute while maintaining high total parameter counts
• The implementation supports seamless integration via conditional instantiation in TransformerBlock, requiring no architectural changes to attention or normalization layers
• Memory estimation tools (memory_estimator_moe.py) help quantify the efficiency gains of sparse expert routing

Frequently Asked Questions

How does the gating network determine which experts process each token?

The gating network is a linear layer mapping from emb_dim to num_experts. It produces scores for each expert, and torch.topk selects the top-k highest scores per token according to the num_experts_per_tok configuration parameter. A softmax over these top-k scores yields the final routing probabilities used to weight expert outputs.

Can I convert an existing dense transformer to use MoE layers?

Yes. According to the TransformerBlock implementation in qwen3.py, you can replace FeedForward with MoEFeedForward by adding num_experts, num_experts_per_tok, and moe_intermediate_size to your configuration dictionary. The MoE layer maintains the same input and output tensor shapes, making it a drop-in replacement.

What is the difference between total parameters and active parameters in MoE models?

Total parameters include all expert networks and the gating layer, while active parameters refer only to those accessed during a specific forward pass. In the LLMs-from-scratch implementation, active parameters equal the gating layer plus num_experts_per_tok expert networks, significantly reducing memory bandwidth requirements compared to dense models with equivalent total capacity.

Where can I find unit tests for the MoE implementation?

Unit tests verifying MoE forward passes and gating logic are located in pkg/llms_from_scratch/tests/test_qwen3.py, with additional notebook-style tests for MoE with KV-cache integration in pkg/llms_from_scratch/tests/test_qwen3_nb.py. These tests validate correct routing behavior and tensor shape preservation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}