# How to Efficiently Manage Tensor Shapes in PyTorch: Patterns from nn-zero-to-hero

> Master PyTorch tensor shapes using explicit contracts and reusable classes. Prevent bugs and write cleaner code with these nn-zero-to-hero patterns.

- Repository: [Andrej/nn-zero-to-hero](https://github.com/karpathy/nn-zero-to-hero)
- Tags: tutorial
- Published: 2026-05-23

---

**Write explicit shape contracts like `B, T, C = x.shape` and encapsulate reshape logic in reusable classes to prevent silent broadcasting bugs and maintain readable PyTorch code.**

Managing tensor dimensions is the primary source of bugs in deep learning pipelines. The `karpathy/nn-zero-to-hero` repository demonstrates disciplined patterns for handling high-dimensional tensors that eliminate guesswork and make models robust to batch size changes.

## Declare Explicit Shape Contracts

Always unpack tensor dimensions immediately after creation to establish a readable contract. In `lectures/makemore/makemore_part5_cnn1.ipynb`, the code declares `# (B, T, C) = x.shape` before any reshaping operations (lines 51-53), serving as both documentation and a runtime assertion that the tensor has the expected rank.

This pattern prevents silent broadcasting errors by making dimensional intent explicit:

```python
def forward(x):
    # Establish contract: x should be (batch, time, channels)

    B, T, C = x.shape
    # Subsequent operations can safely assume these dimensions

    return x.view(B, T // 2, C * 2)

```

## Centralize Reshape Logic with Utility Layers

Avoid scattering ad-hoc `view` calls throughout your model. The repository implements a `FlattenConsecutive` class in `makemore_part5_cnn1.ipynb` (lines 45-56) that collapses consecutive dimensions while preserving the batch axis, centralizing complex reshape logic in one tested location.

```python
class FlattenConsecutive:
    """Collapse every `n` consecutive time-steps into the channel dimension."""
    def __init__(self, n):
        self.n = n

    def __call__(self, x):
        B, T, C = x.shape
        x = x.view(B, T // self.n, C * self.n)
        if x.shape[1] == 1:
            x = x.squeeze(1)
        return x

```

Using `torch.view` performs a zero-copy reshape of the underlying storage, adding negligible overhead while keeping the model definition clean.

## Validate Shapes During Development

Insert lightweight shape checks after dataset creation and during forward passes. The notebooks in `makemore_part5_cnn1.ipynb` (lines 12-14) print shapes immediately after loading data to catch mismatches between inputs and labels before training begins.

```python

# Sanity check after dataset creation

print(f"Input shape: {X.shape}, Target shape: {Y.shape}")

```

Remove these prints once the pipeline stabilizes, or replace them with `logging.debug` to maintain visibility during debugging sessions.

## Write Dimension-Agnostic Normalization

Design layers that introspect tensor rank to handle multiple input formats. The `BatchNorm1d` implementation in `makemore_part5_cnn1.ipynb` (lines 01-08) detects whether the input is 2D `[N, C]` or 3D `[N, T, C]` by checking `x.ndim`, selecting reduction dimensions automatically.

```python
class BatchNorm1d:
    def __init__(self, dim, eps=1e-5, momentum=0.1):
        self.eps = eps
        self.momentum = momentum
        self.training = True
        self.gamma = torch.ones(dim)
        self.beta = torch.zeros(dim)
        self.running_mean = torch.zeros(dim)
        self.running_var = torch.ones(dim)

    def __call__(self, x):
        if self.training:
            # Auto-detect reduction dimensions based on rank

            dim = 0 if x.ndim == 2 else (0, 1)
            mean = x.mean(dim, keepdim=True)
            var = x.var(dim, keepdim=True)
        else:
            mean, var = self.running_mean, self.running_var
        x_hat = (x - mean) / torch.sqrt(var + self.eps)
        out = self.gamma * x_hat + self.beta
        return out

```

This eliminates the need for manual reshaping when switching between fully-connected and sequence models.

## Flatten Embeddings Predictably

When preparing embedding outputs for linear layers, use dynamic batch sizing rather than hard-coding dimensions. In `makemore_part4_backprop.ipynb` (lines 212-214), the repository flattens embeddings using `emb.view(emb.shape[0], -1)`, ensuring the batch dimension remains flexible while collapsing all feature dimensions.

```python

# emb has shape (batch, seq_len, embed_dim)

embcat = emb.view(emb.shape[0], -1)  # Results in (batch, seq_len*embed_dim)

```

This pattern automatically adapts when you change batch sizes for debugging or production inference.

## Summary

- **Unpack dimensions explicitly** with `B, T, C = x.shape` to create self-documenting code and catch rank mismatches early.
- **Encapsulate reshapes** in dedicated classes like `FlattenConsecutive` to avoid copy-paste errors and centralize logic.
- **Use `tensor.view` over manual reshaping** for zero-copy dimension manipulation that adds no runtime overhead.
- **Introspect `tensor.ndim`** when writing normalization layers to handle both 2D and 3D inputs without conditional reshaping.
- **Reference `tensor.shape[0]`** instead of hard-coded batch sizes to make models robust to varying batch dimensions.

## Frequently Asked Questions

### What is the difference between `torch.view` and `torch.reshape`?

Both operations return a view of the underlying storage without copying data, but `torch.view` requires contiguous memory and is the preferred choice in the nn-zero-to-hero codebase when you have established explicit shape contracts. According to the repository's implementation in `makemore_part5_cnn1.ipynb`, `view` is used extensively inside utility layers because the preceding operations guarantee contiguous outputs, providing the most efficient zero-cost reshaping.

### When should I use the `FlattenConsecutive` pattern instead of standard reshaping?

Use `FlattenConsecutive` when you need to collapse a specific number of consecutive dimensions—such as merging time steps into channel dimensions—while preserving the batch axis. This pattern appears in the CNN implementation of the makemore series to replace ad-hoc `view` calls, making the model architecture readable and the reshape logic testable as a standalone unit.

### How does the repository handle batch normalization for different tensor ranks?

The `BatchNorm1d` class automatically adapts by checking `x.ndim` at runtime. For 2D inputs `[N, C]`, it reduces over dimension 0; for 3D inputs `[N, T, C]`, it reduces over dimensions `(0, 1)`. As implemented in `makemore_part5_cnn1.ipynb`, this allows the same normalization layer to work seamlessly with both fully-connected networks and sequence models without manual dimension juggling.

### Why is `B, T, C = x.shape` preferred over indexing like `x.shape[0]`?

Unpacking with pattern matching serves as a runtime assertion that the tensor has exactly three dimensions while assigning semantic meaning to each axis (batch, time, channels). This explicit contract, demonstrated in `makemore_part5_cnn1.ipynb` (lines 51-53), makes the code self-documenting and immediately raises an error if a layer receives an unexpected tensor rank, preventing silent broadcasting bugs that could propagate through the network.