# Hyperparameter Tuning in d2l-zh: Best Practices from Dive into Deep Learning

> Master hyperparameter tuning in d2l-zh Learn best practices like validation sets and K-fold cross-validation to optimize your deep learning models.

- Repository: [Dive into Deep Learning (D2L.ai)/d2l-zh](https://github.com/d2l-ai/d2l-zh)
- Tags: best-practices
- Published: 2026-03-01

---

**Hyperparameter tuning in the d2l-zh repository relies on validation sets, K-fold cross-validation, and systematic one-at-a-time experimentation to optimize learning rates, regularization strengths, and model architectures.**

The Chinese edition of *Dive into Deep Learning* (`d2l-ai/d2l-zh`) provides a structured methodology for hyperparameter tuning that spans its linear regression, multilayer perceptron, and optimization chapters. Rather than treating hyperparameters as arbitrary knobs, the text establishes reproducible workflows that isolate variables, measure generalization, and leverage cross-validation for robust evaluation.

## Core Principles of Hyperparameter Tuning in d2l-zh

### Establish a Validation Set

The foundation of tuning in d2l-zh begins with reserving a portion of training data for validation. This practice is introduced in [`chapter_linear-networks/linear-regression.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_linear-networks/linear-regression.md) and applied extensively in [`chapter_multilayer-perceptrons/kaggle-house-price_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_multilayer-perceptrons/kaggle-house-price_origin.md). The validation set provides an unbiased signal for comparing hyperparameter configurations without contaminating the test set, preventing overfitting to the training data during the selection process.

### Start with Sensible Defaults

Before conducting extensive searches, d2l-zh advocates establishing baselines using community-tested defaults. In [`chapter_multilayer-perceptrons/mlp-scratch_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_multilayer-perceptrons/mlp-scratch_origin.md) (lines 34-40), the text specifies default values such as `num_hiddens=256`, learning rate `0.1` for SGD, and `batch_size=256`. These defaults reduce the initial search space and provide reference points for subsequent adjustments, ensuring that experimentation begins from a functional baseline rather than random initialization.

### Tune One Hyperparameter at a Time

To isolate the impact of individual settings, the repository emphasizes varying a single hyperparameter while holding others constant. The exercises in [`chapter_multilayer-perceptrons/mlp-scratch_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_multilayer-perceptrons/mlp-scratch_origin.md) explicitly prompt readers to experiment with `num_hiddens`, learning rate, and epoch counts independently. This controlled approach prevents confounding variables and reveals clear performance trends, avoiding the combinatorial explosion that occurs when multiple parameters change simultaneously.

## Advanced Techniques: Cross-Validation and Regularization

### K-Fold Cross-Validation for Robust Evaluation

For datasets where validation splits might be noisy or sample sizes limited, d2l-zh implements K-fold cross-validation. The [`chapter_multilayer-perceptrons/kaggle-house-price_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_multilayer-perceptrons/kaggle-house-price_origin.md) file contains the canonical implementation with `get_k_fold_data` and `k_fold` functions. This technique trains K models on different data partitions and averages validation scores, yielding more stable hyperparameter rankings than single-split validation, particularly crucial when selecting regularization strengths or learning rates on smaller datasets.

### Regularization Strategies

To prevent overfitting during hyperparameter exploration, the text integrates **weight decay** (`wd`) and **dropout** as tunable regularization parameters. The [`chapter_multilayer-perceptrons/weight-decay.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_multilayer-perceptrons/weight-decay.md) chapter demonstrates how to treat `wd` as a hyperparameter alongside learning rate, while [`chapter_multilayer-perceptrons/dropout.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_multilayer-perceptrons/dropout.md) explores dropout probability settings. These regularization hyperparameters interact with model capacity and require joint consideration during tuning, typically evaluated through the same validation framework used for other parameters.

## Optimizer Selection and Monitoring

### Choosing the Right Optimizer

The choice of optimizer itself constitutes a critical hyperparameter decision. According to [`chapter_optimization/adam.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_optimization/adam.md), **Adam** is less sensitive to initial learning rate selection and often works well with default parameters, reducing the tuning burden. Conversely, [`chapter_optimization/momentum.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_optimization/momentum.md) discusses how momentum coefficients can accelerate SGD but require careful tuning. Selecting an appropriate optimizer early narrows the effective search space for secondary hyperparameters, with Adam generally recommended for rapid prototyping and SGD with momentum for fine-tuned final models.

### Monitoring Training Dynamics

Systematic tuning requires observing both training and validation curves to detect divergence. Throughout the book, particularly in [`chapter_optimization/optimization-intro.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_optimization/optimization-intro.md), plotting utilities visualize loss and accuracy per epoch. Practitioners watch for validation loss increasing while training loss decreases—a signal to adjust learning rate or regularization. This monitoring discipline prevents wasted computation on poorly configured runs and guides intelligent iteration on hyperparameter ranges.

## Practical Implementation: Code Examples from d2l-zh

The following snippets illustrate the core tuning workflow used throughout the repository.

### Grid Search Over Learning Rate and Hidden Units

This example from [`chapter_multilayer-perceptrons/mlp-scratch_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_multilayer-perceptrons/mlp-scratch_origin.md) demonstrates a systematic sweep over MLP configurations:

```python
from d2l import mxnet as d2l
from mxnet import np, npx
npx.set_np()

def train_mlp(num_hiddens, lr, num_epochs=10):
    net = d2l.nn.Sequential()
    net.add(d2l.nn.Dense(num_hiddens, activation='relu'),
            d2l.nn.Dense(10))
    net.initialize()
    loss = d2l.nn.SoftmaxCrossEntropyLoss()
    trainer = d2l.optim.SGD(lr=lr)
    train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=256)
    d2l.train_ch3(net, train_iter, test_iter, loss,
                   num_epochs, lambda bs: trainer.step(bs))
    return d2l.evaluate_accuracy(net, test_iter)

# Grid‑search over two hyperparameters

candidates = [(256, 0.1), (512, 0.05), (128, 0.2)]
best_acc = 0
for h, lr in candidates:
    acc = train_mlp(h, lr)
    print(f'hiddens={h}, lr={lr} → test accuracy: {acc:.4f}')
    if acc > best_acc:
        best_acc, best_cfg = acc, (h, lr)
print(f'Best config: hidden={best_cfg[0]}, lr={best_cfg[1]}, acc={best_acc:.4f}')

```

### K-Fold Cross-Validation Pipeline

This implementation from [`chapter_multilayer-perceptrons/kaggle-house-price_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_multilayer-perceptrons/kaggle-house-price_origin.md) provides robust evaluation across data splits:

```python
def k_fold_cv(k, X, y, lr, weight_decay, num_epochs=100, batch_size=64):
    # Returns mean validation RMSE across k folds

    from d2l import mxnet as d2l
    train_l, valid_l = d2l.k_fold(k, X, y,
                                 num_epochs, lr, weight_decay, batch_size)
    return valid_l

# Example search over learning‑rate and weight‑decay

candidate_lrs = [0.1, 1, 5]
candidate_wd = [0, 1e-4, 5e-4]
best_rmse = float('inf')
for lr in candidate_lrs:
    for wd in candidate_wd:
        rmse = k_fold_cv(k=5, X=train_features, y=train_labels,
                         lr=lr, weight_decay=wd)
        print(f'lr={lr}, wd={wd} → validation RMSE={rmse:.4f}')
        if rmse < best_rmse:
            best_rmse, best_cfg = rmse, (lr, wd)
print(f'Best hyper‑params: lr={best_cfg[0]}, weight_decay={best_cfg[1]}, RMSE={best_rmse:.4f}')

```

### Early Stopping with Validation Monitoring

This pattern from [`chapter_linear-networks/linear-regression.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_linear-networks/linear-regression.md) prevents overfitting during the tuning process:

```python
def train_with_early_stop(net, train_iter, valid_iter,
                          loss, num_epochs, lr):
    trainer = d2l.optim.SGD(lr=lr)
    best_val_loss = float('inf')
    for epoch in range(num_epochs):
        d2l.train_ch3(net, train_iter, None, loss,
                      1, lambda _: trainer.step(1))
        val_loss = d2l.evaluate_loss(net, valid_iter, loss)
        if val_loss < best_val_loss:
            best_val_loss = val_loss
            net.save_parameters('best.params')
        else:
            print(f'Stopping early at epoch {epoch+1}')
            break
    net.load_parameters('best.params')
    return net

```

## Summary

- **Establish a validation set** early to measure generalization without contaminating the test set, as demonstrated in [`chapter_linear-networks/linear-regression.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_linear-networks/linear-regression.md).
- **Begin with sensible defaults** (learning rate 0.1, batch size 256, 256 hidden units) before conducting extensive searches.
- **Tune one hyperparameter at a time** to isolate effects and avoid combinatorial explosion.
- **Implement K-fold cross-validation** using `get_k_fold_data` and `k_fold` from [`chapter_multilayer-perceptrons/kaggle-house-price_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_multilayer-perceptrons/kaggle-house-price_origin.md) for robust evaluation on smaller datasets.
- **Integrate regularization** (weight decay and dropout) as tunable hyperparameters to control model capacity.
- **Select optimizers strategically**—Adam reduces learning rate sensitivity while SGD with momentum offers fine-grained control.
- **Monitor training and validation curves** to detect overfitting early and guide iterative refinement.

## Frequently Asked Questions

### What is the recommended starting learning rate for SGD in d2l-zh?

The repository recommends starting with a learning rate of **0.1** for standard SGD, as implemented in [`chapter_multilayer-perceptrons/mlp-scratch_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_multilayer-perceptrons/mlp-scratch_origin.md). However, when using adaptive optimizers like Adam described in [`chapter_optimization/adam.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_optimization/adam.md), the default learning rate of 0.001 typically works without modification, significantly reducing the tuning burden.

### How does K-fold cross-validation work in the d2l-zh implementation?

The implementation in [`chapter_multilayer-perceptrons/kaggle-house-price_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_multilayer-perceptrons/kaggle-house-price_origin.md) provides the `get_k_fold_data` and `k_fold` functions to partition data into K folds. The procedure trains K separate models, each using K-1 folds for training and the remaining fold for validation, then averages the validation scores. This approach provides more stable hyperparameter rankings than single-split validation, particularly when working with limited data samples.

### Should weight decay and learning rate be tuned simultaneously?

While these hyperparameters interact, d2l-zh recommends establishing a baseline learning rate first, then introducing **weight decay** (`wd`) as a secondary tuning parameter. The [`chapter_multilayer-perceptrons/weight-decay.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_multilayer-perceptrons/weight-decay.md) chapter demonstrates treating `wd` as a configurable hyperparameter that should be evaluated through the same validation framework used for other parameters, typically using the K-fold cross-validation pipeline from the Kaggle house price example.

### What is the best optimizer for reducing hyperparameter tuning effort?

According to [`chapter_optimization/adam.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_optimization/adam.md), **Adam** is generally the best choice for reducing tuning burden because it adapts learning rates per-parameter and is less sensitive to the initial global learning rate compared to vanilla SGD. While SGD with momentum (covered in [`chapter_optimization/momentum.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_optimization/momentum.md)) can achieve superior final performance, it requires more careful tuning of both the learning rate and momentum coefficients.