# Understanding the Internals of PyTorch torch.nn Modules: A Deep Dive into nn.Module

> Explore PyTorch nnModule internals. Discover how it registers parameters, handles forward passes via __call__, and manages state with state_dict.

- Repository: [Andrej/nn-zero-to-hero](https://github.com/karpathy/nn-zero-to-hero)
- Tags: deep-dive
- Published: 2026-05-23

---

**PyTorch's `torch.nn` modules are built on the abstract `nn.Module` base class, which automatically registers parameters via `__setattr__`, executes forward passes through a `__call__` wrapper, and manages hierarchical state through `state_dict` and `load_state_dict`.**

The `torch.nn` API provides the high-level building blocks for deep learning in PyTorch, powering everything from simple linear layers to complex transformer architectures. Understanding the internals of PyTorch `torch.nn` modules reveals how parameters are tracked, gradients flow through nested containers, and models serialize their state. This knowledge is essential when implementing custom layers in the spirit of Andrej Karpathy's *nn-zero-to-hero* educational approach, where we replicate framework functionality from scratch to demystify deep learning internals.

## Core Architecture of torch.nn

### The nn.Module Base Class

Located in [`torch/nn/modules/module.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/module.py), the `nn.Module` class serves as the foundation for all neural network components. It maintains internal dictionaries `_parameters` and `_modules` to track learnable tensors and child modules, while providing the `__call__` mechanism that triggers `forward` execution.

### Parameter Registration with nn.Parameter

The `nn.Parameter` class (defined in [`torch/nn/parameter.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/parameter.py)) is a thin subclass of `torch.Tensor` that signals to `nn.Module` that a tensor requires gradient tracking and inclusion in `state_dict`. When you assign an attribute in a module's `__init__`, `Module.__setattr__` intercepts the assignment and registers `Parameter` objects in the `_parameters` dictionary.

### Container Modules

Container classes like `Sequential`, `ModuleList`, and `ModuleDict` (implemented in [`torch/nn/modules/container.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/container.py)) enable hierarchical model composition. These containers automatically register their child modules in the parent’s `_modules` dictionary, allowing `parameters()` to recursively traverse the entire model tree.

### Specialized Layer Implementations

- **Linear layers**: [`torch/nn/modules/linear.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/linear.py) implements fully-connected transformations `y = xAᵀ + b` using `nn.init.kaiming_uniform_` for weight initialization and `nn.init.zeros_` for bias.
- **Convolutional layers**: [`torch/nn/modules/conv.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/conv.py) delegates spatial operations to low-level ATen/CUDA kernels while handling shape bookkeeping and bias addition in Python.
- **Normalization layers**: [`torch/nn/modules/batchnorm.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/batchnorm.py) maintains running statistics via `register_buffer` and learnable affine parameters (`weight`, `bias`).
- **Activation functions**: Stateless operations in [`torch/nn/modules/activation.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/activation.py) that wrap functional ATen calls without registering parameters.

## How nn.Module Works Internally

### Construction and Attribute Assignment

When subclassing `nn.Module`, any assignment of `nn.Parameter` or another `nn.Module` to an attribute triggers `Module.__setattr__`. This interception mechanism populates the internal `_parameters` and `_modules` dictionaries, enabling automatic gradient tracking and sub-module discovery.

### The Forward Pass Mechanism

Calling a module instance (`output = module(input)`) invokes `Module.__call__` rather than `forward` directly. This wrapper executes registered pre-hooks, runs the user-defined `forward` method, applies post-hooks, and ensures `torch.autograd.Function` operations integrate with the computation graph.

### Parameter Collection

The `parameters()` method yields an iterator over all leaf `Parameter` objects by recursively traversing the module hierarchy stored in `_modules`. This iterator is consumed by optimizers to determine which tensors receive gradient updates during `optimizer.step()`.

### State Serialization

`state_dict()` compiles parameters and persistent buffers into an `OrderedDict`, while `load_state_dict()` restores them with shape validation. These methods, defined in [`module.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/module.py), handle missing keys and unexpected keys through strict mode checking.

### Device and Dtype Migration

Methods like `to()`, `cuda()`, and `float()` utilize `apply(fn)` to recursively transform every child module and parameter, ensuring the entire model tree moves to the target device or data type consistently.

## Practical Code Examples

### Implementing a Custom Linear Layer

```python
import torch
from torch import nn

class MyLinear(nn.Module):
    def __init__(self, in_features, out_features):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(out_features, in_features))
        self.bias = nn.Parameter(torch.zeros(out_features))
    
    def forward(self, x):
        return torch.nn.functional.linear(x, self.weight, self.bias)

# Usage

layer = MyLinear(10, 5)
output = layer(torch.randn(2, 10))

```

This implementation mirrors the official `Linear` layer in [`torch/nn/modules/linear.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/linear.py), demonstrating how `nn.Parameter` registration occurs during attribute assignment.

### Building Models with Containers

```python
model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.BatchNorm1d(256),
    nn.Linear(256, 10)
)

logits = model(torch.randn(32, 784))

```

As implemented in [`torch/nn/modules/container.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/container.py), `nn.Sequential` automatically registers each child module, enabling recursive parameter traversal.

### Inspecting Model Parameters

```python
for name, param in model.named_parameters():
    print(name, param.shape, param.requires_grad)

```

Behind the scenes in [`torch/nn/modules/module.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/module.py), `named_parameters` walks the `_modules` hierarchy to generate fully-qualified parameter names.

### Saving and Loading State

```python
torch.save(model.state_dict(), "model.pth")

# Later

model = MyLinear(10, 5)
model.load_state_dict(torch.load("model.pth"))

```

The `state_dict` and `load_state_dict` methods handle serialization logic defined in [`module.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/module.py), preserving both parameters and registered buffers.

## Key Source Files in PyTorch

Understanding the internals of PyTorch `torch.nn` modules requires familiarity with these specific source locations:

- **[`torch/nn/modules/module.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/module.py)**: Core `Module` class, `__setattr__` registration logic, `__call__` wrapper, and utility methods (`state_dict`, `apply`, `to`).
- **[`torch/nn/parameter.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/parameter.py)**: Definition of `Parameter` as a `Tensor` subclass.
- **[`torch/nn/modules/container.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/container.py)**: `Sequential`, `ModuleList`, and `ModuleDict` implementations.
- **[`torch/nn/modules/linear.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/linear.py)**: Fully-connected layer with Kaiming initialization.
- **[`torch/nn/modules/conv.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/conv.py)**: Convolutional layers delegating to ATen kernels.
- **[`torch/nn/modules/activation.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/activation.py)**: Stateless activation wrappers.
- **[`torch/nn/modules/batchnorm.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/batchnorm.py)**: Running statistics management via `register_buffer`.
- **[`torch/nn/modules/loss.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/loss.py)**: Loss function modules wrapping functional implementations.

## Summary

- **`nn.Module`** in [`torch/nn/modules/module.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/module.py) provides the base class for all layers, implementing `__call__` to wrap `forward` execution and `__setattr__` to auto-register parameters.
- **`nn.Parameter`** signals learnable tensors that optimizers update via the `parameters()` iterator.
- **Container modules** like `nn.Sequential` automatically track child modules in `_modules`, enabling recursive tree traversal.
- **State management** occurs through `state_dict()` and `load_state_dict()`, which serialize parameters and buffers to `OrderedDict` structures.
- **Device migration** uses `apply()` to recursively transform the entire module hierarchy via `to()`, `cuda()`, or `cpu()`.

## Frequently Asked Questions

### How does PyTorch know which tensors are trainable parameters?

PyTorch identifies trainable parameters through the `nn.Parameter` wrapper class defined in [`torch/nn/parameter.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/parameter.py). When you assign a `Parameter` to a module attribute, `Module.__setattr__` in [`torch/nn/modules/module.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/module.py) intercepts the assignment and registers it in the internal `_parameters` dictionary. Only tensors wrapped as `Parameter` objects appear in `module.parameters()` and receive gradients during backpropagation.

### What is the difference between `register_parameter` and `register_buffer`?

`register_parameter` adds tensors to the `_parameters` dictionary, marking them as trainable and including them in `state_dict` and optimizer updates. `register_buffer` (used in [`torch/nn/modules/batchnorm.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/batchnorm.py) for `running_mean` and `running_var`) adds persistent state that should be saved with the model but not updated by optimizers, such as batch normalization statistics. Buffers appear in `state_dict` but not in `module.parameters()`.

### Why does calling a module execute `forward` indirectly through `__call__`?

`Module.__call__` in [`torch/nn/modules/module.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/torch/nn/modules/module.py) provides a hook mechanism that executes registered pre-hooks and post-hooks around the `forward` method. This wrapper ensures proper integration with the autograd engine, handles forward pre-hooks for debugging or quantization, and maintains consistency across all module executions. Directly calling `forward` would bypass these essential framework features.

### How do `state_dict` and `load_state_dict` handle missing keys?

When loading a state dictionary, `load_state_dict` defined in [`module.py`](https://github.com/karpathy/nn-zero-to-hero/blob/main/module.py) compares keys against the model's current parameters and buffers. With `strict=True` (the default), it raises a `RuntimeError` for missing keys in the model or unexpected keys in the state dict. Setting `strict=False` allows partial loading, returning `missing_keys` and `unexpected_keys` lists without raising errors, useful for transfer learning or architecture modifications.