# MegaDLMs Framework Architecture: Core Components Explained

> Discover the core components of the MegaDLMs framework architecture. Learn how MegaDLMs enhances Megatron-LM for scalable diffusion language model training with specialized layers and utilities.

- Repository: [Jinjie Ni/megadlms](https://github.com/jinjieni/megadlms)
- Tags: architecture
- Published: 2026-03-04

---

**MegaDLMs extends Megatron-LM with diffusion-specific transformer layers, distributed parallelism primitives, and mixed-precision training utilities to enable scalable diffusion language model training.**

The MegaDLMs framework, available at `jinjieni/megadlms`, builds upon NVIDIA's Megatron-LM infrastructure to support diffusion-based language modeling. Its architecture comprises tightly-coupled core components ranging from low-level attention kernels to high-level generation loops, all designed to handle the unique requirements of discrete diffusion language models (DLMs).

## Foundation: MegatronModule and Configuration

Every component in the MegaDLMs framework architecture inherits from a common base class that abstracts distributed state handling and precision management.

### The MegatronModule Base Class

The `MegatronModule` class, defined in [`megatron/core/transformer/module.py`](https://github.com/jinjieni/megadlms/blob/main/megatron/core/transformer/module.py), serves as the foundation for all model components. It adds **checkpointing support**, **sharded state-dict handling**, and **FP8-aware utilities** to standard PyTorch modules. The `Float16Module` wrapper (line 37 in the same file) automatically casts tensors to FP16, BF16, or FP8 when enabled via configuration flags.

### Configuration Objects

The framework uses a two-tier configuration system:

- **`TransformerConfig`** ([`megatron/core/transformer/transformer_config.py`](https://github.com/jinjieni/megadlms/blob/main/megatron/core/transformer/transformer_config.py)) – Contains generic Megatron-LM hyperparameters including parallelism sizes, dropout rates, and precision settings.
- **`DLMConfig`** ([`tools/weights_conversion/hf_configs/gptneox_1.7b_dlm/configuration_dlm.py`](https://github.com/jinjieni/megadlms/blob/main/tools/weights_conversion/hf_configs/gptneox_1.7b_dlm/configuration_dlm.py)) – Extends the base configuration with diffusion-specific parameters such as `sliding_window` size, `mask_token_id`, and remasking strategies.

## Core Transformer Stack: Layers and Blocks

The transformer stack implements the per-layer computation flow (norm → self-attention → cross-attention → MLP) with specialized variants for diffusion modeling.

### TransformerLayer Implementations

The `TransformerLayer` class in [`megatron/core/transformer/transformer_layer.py`](https://github.com/jinjieni/megadlms/blob/main/megatron/core/transformer/transformer_layer.py) provides the standard dense transformer implementation with optional bias-dropout-add fusion. For diffusion-specific workloads, the `DiffLMTransformerLayer` subclass (starting at line 776 in the same file) adds **rotary position handling** and **sliding-window attention** support required for discrete diffusion training.

### TransformerBlock Abstractions

Layers are grouped into blocks via the `TransformerBlock` class in [`megatron/core/transformer/transformer_block.py`](https://github.com/jinjieni/megadlms/blob/main/megatron/core/transformer/transformer_block.py). This component manages **pipeline parallelism**, **CUDA-graph caching**, and **activation checkpointing**. The diffusion variant, `DiffLMTransformerBlock` (line 800), inherits this logic but instantiates `DiffLMTransformerLayer` instead of standard layers.

## Diffusion-Specific Components

These components differentiate MegaDLMs from standard transformer implementations by integrating diffusion timestep handling and token remasking logic.

### Diffusion-Aware Attention Kernels

The attention mechanism is implemented in [`megatron/core/transformer/attention.py`](https://github.com/jinjieni/megadlms/blob/main/megatron/core/transformer/attention.py). The `DiffLMAttention` class (line 1226) and its specialized variants (`DiffLMSelfAttention`) integrate **rotary embeddings**, **sliding-window masks**, and optional **Flash Attention** backends. These kernels handle the unique attention patterns required when processing partially masked token sequences during diffusion steps.

### The DiffLM Model Architecture

The top-level `DiffLM` class in [`tools/weights_conversion/hf_configs/gptneox_1.7b_dlm/modeling_dlm.py`](https://github.com/jinjieni/megadlms/blob/main/tools/weights_conversion/hf_configs/gptneox_1.7b_dlm/modeling_dlm.py) (line 442) provides a Hugging Face-compatible interface that stitches together the decoder with a language modeling head. It implements the `generate()` method (line 730) containing the **diffusion generation loop** and **token-mask-remasking logic** required for DLM inference.

The actual transformer decoder is implemented in `DLMModel` (line 504), which contains an embedding layer, a stack of `DLMDecoderLayer` instances (each wrapping a `DiffLMTransformerLayer`), RMSNorm layers, and rotary-embedding support.

## Distributed Training Infrastructure

MegaDLMs weaves parallelism throughout the architecture via the `parallel_state` module and related utilities.

### Parallelism Primitives

The framework supports **data parallelism (DP)**, **tensor parallelism (TP)**, **pipeline parallelism (PP)**, **context parallelism (CP)**, and **expert parallelism (EP)** for MoE models. These are injected into `MegatronModule` and block/layer constructors through the `parallel_state` utilities found in [`megatron/core/parallel_state.py`](https://github.com/jinjieni/megadlms/blob/main/megatron/core/parallel_state.py). Training scripts configure these via command-line arguments parsed in [`megatron/training/arguments.py`](https://github.com/jinjieni/megadlms/blob/main/megatron/training/arguments.py) and wired together in [`megatron/training/training.py`](https://github.com/jinjieni/megadlms/blob/main/megatron/training/training.py).

## Mixed-Precision and Checkpointing

The architecture includes native support for reduced-precision training and distributed checkpointing.

### FP8 and Mixed-Precision Engine

When `config.fp8` or `config.bf16` is enabled, the framework automatically injects appropriate Tensor-Engine (TE) scaling hooks through the `Float16Module` wrapper. This enables **FP8 (e4m3) training** with automatic loss scaling without manual tensor casting in model code.

### Sharded Checkpointing Utilities

The `megatron/core/dist_checkpointing/` directory contains utilities for **sharded checkpoint creation**, **CUDA-graph serialization**, and **TensorRT-LLM model export**. These integrate with the `sharded_state_dict()` method defined in `MegatronModule` to handle distributed model states across hundreds of GPUs.

## Practical Implementation Examples

### Instantiating a Diffusion Language Model

```python
from tools.weights_conversion.hf_configs.gptneox_1.7b_dlm.modeling_dlm import DiffLM, DLMConfig

config = DLMConfig(
    vocab_size=50257,
    hidden_size=2048,
    num_hidden_layers=24,
    num_attention_heads=16,
    num_key_value_heads=4,
    max_position_embeddings=4096,
    sliding_window=2048,
    mask_token_id=50256,
    rms_norm_eps=1e-6,
    attn_dropout=0.1,
    dropout=0.1,
)

model = DiffLM(config).cuda()

```

### Running Inference with the Diffusion Loop

```python
import torch
from transformers import GenerationConfig

# Standard forward pass

input_ids = torch.randint(0, config.vocab_size, (2, 16)).cuda()
outputs = model(input_ids)
logits = outputs.logits  # shape: (2, 16, vocab_size)

# Diffusion generation

gen_cfg = GenerationConfig(
    max_new_tokens=64,
    temperature=0.8,
    top_p=0.95,
    cfg=2.0,  # classifier-free guidance

    sample_steps=8,
    inference_block_size=8,
    remasking="low_confidence",
)

prompt = torch.tensor([[50256, 50257, 12345]]).cuda()
generated = model.generate(input_ids=prompt, generation_config=gen_cfg)

```

### Configuring Distributed Training

```python
from megatron.training.arguments import parse_args
from megatron.training.training import train

args = parse_args([
    "--num-layers", "24",
    "--hidden-size", "2048",
    "--tensor-model-parallel-size", "4",
    "--pipeline-model-parallel-size", "2",
    "--micro-batch-size", "2",
    "--global-batch-size", "32",
    "--fp8", "e4m3",
])

train(args)  # Automatically wires TP/PP through MegatronModule

```

## Summary

- **MegaDLMs** extends Megatron-LM with diffusion-specific transformer components hosted at `jinjieni/megadlms`.
- The **`MegatronModule`** base class in [`megatron/core/transformer/module.py`](https://github.com/jinjieni/megadlms/blob/main/megatron/core/transformer/module.py) provides checkpointing, sharding, and FP8 support to all components.
- **`DiffLMTransformerLayer`** and **`DiffLMTransformerBlock`** (in [`transformer_layer.py`](https://github.com/jinjieni/megadlms/blob/main/transformer_layer.py) and [`transformer_block.py`](https://github.com/jinjieni/megadlms/blob/main/transformer_block.py)) handle diffusion-specific attention patterns and sliding-window constraints.
- The **`DiffLM`** class in [`modeling_dlm.py`](https://github.com/jinjieni/megadlms/blob/main/modeling_dlm.py) integrates the decoder stack with the diffusion generation loop and token remasking logic.
- **Parallelism primitives** (TP, PP, CP, EP) are woven throughout the architecture via `parallel_state` utilities.
- **Mixed-precision training** (FP8/BF16) is handled automatically through configuration flags and the `Float16Module` wrapper.

## Frequently Asked Questions

### What is the difference between TransformerLayer and DiffLMTransformerLayer?

`TransformerLayer` in [`megatron/core/transformer/transformer_layer.py`](https://github.com/jinjieni/megadlms/blob/main/megatron/core/transformer/transformer_layer.py) implements the standard dense transformer computation flow, while `DiffLMTransformerLayer` (starting at line 776) extends this with diffusion-specific features including rotary position embeddings for timestep handling and sliding-window attention masks required for discrete diffusion training.

### How does MegaDLMs handle distributed training across multiple GPUs?

The framework injects parallelism primitives into the `MegatronModule` base class and block constructors via the `parallel_state` utilities. It supports tensor parallelism (TP), pipeline parallelism (PP), context parallelism (CP), and expert parallelism (EP) for MoE models, configured through command-line arguments in [`megatron/training/arguments.py`](https://github.com/jinjieni/megadlms/blob/main/megatron/training/arguments.py).

### What configuration class should I use for diffusion-specific parameters?

Use `DLMConfig` defined in [`tools/weights_conversion/hf_configs/gptneox_1.7b_dlm/configuration_dlm.py`](https://github.com/jinjieni/megadlms/blob/main/tools/weights_conversion/hf_configs/gptneox_1.7b_dlm/configuration_dlm.py) for diffusion-specific settings like `sliding_window`, `mask_token_id`, and remasking strategies. For general Megatron-LM parameters like parallelism sizes and dropout, use `TransformerConfig` from [`megatron/core/transformer/transformer_config.py`](https://github.com/jinjieni/megadlms/blob/main/megatron/core/transformer/transformer_config.py).

### How does the built-in generation loop work in DiffLM?

The `DiffLM.generate()` method in [`modeling_dlm.py`](https://github.com/jinjieni/megadlms/blob/main/modeling_dlm.py) (line 730) implements the discrete diffusion generation loop, which starts from a fully masked sequence and iteratively denoises tokens according to the configured `sample_steps`, `remasking` strategy (e.g., "low_confidence"), and classifier-free guidance scale (`cfg`).