Hyperparameter Tuning in d2l-zh: Best Practices from Dive into Deep Learning

Hyperparameter tuning in the d2l-zh repository relies on validation sets, K-fold cross-validation, and systematic one-at-a-time experimentation to optimize learning rates, regularization strengths, and model architectures.

The Chinese edition of Dive into Deep Learning (d2l-ai/d2l-zh) provides a structured methodology for hyperparameter tuning that spans its linear regression, multilayer perceptron, and optimization chapters. Rather than treating hyperparameters as arbitrary knobs, the text establishes reproducible workflows that isolate variables, measure generalization, and leverage cross-validation for robust evaluation.

Core Principles of Hyperparameter Tuning in d2l-zh

Establish a Validation Set

The foundation of tuning in d2l-zh begins with reserving a portion of training data for validation. This practice is introduced in chapter_linear-networks/linear-regression.md and applied extensively in chapter_multilayer-perceptrons/kaggle-house-price_origin.md. The validation set provides an unbiased signal for comparing hyperparameter configurations without contaminating the test set, preventing overfitting to the training data during the selection process.

Start with Sensible Defaults

Before conducting extensive searches, d2l-zh advocates establishing baselines using community-tested defaults. In chapter_multilayer-perceptrons/mlp-scratch_origin.md (lines 34-40), the text specifies default values such as num_hiddens=256, learning rate 0.1 for SGD, and batch_size=256. These defaults reduce the initial search space and provide reference points for subsequent adjustments, ensuring that experimentation begins from a functional baseline rather than random initialization.

Tune One Hyperparameter at a Time

To isolate the impact of individual settings, the repository emphasizes varying a single hyperparameter while holding others constant. The exercises in chapter_multilayer-perceptrons/mlp-scratch_origin.md explicitly prompt readers to experiment with num_hiddens, learning rate, and epoch counts independently. This controlled approach prevents confounding variables and reveals clear performance trends, avoiding the combinatorial explosion that occurs when multiple parameters change simultaneously.

Advanced Techniques: Cross-Validation and Regularization

K-Fold Cross-Validation for Robust Evaluation

For datasets where validation splits might be noisy or sample sizes limited, d2l-zh implements K-fold cross-validation. The chapter_multilayer-perceptrons/kaggle-house-price_origin.md file contains the canonical implementation with get_k_fold_data and k_fold functions. This technique trains K models on different data partitions and averages validation scores, yielding more stable hyperparameter rankings than single-split validation, particularly crucial when selecting regularization strengths or learning rates on smaller datasets.

Regularization Strategies

To prevent overfitting during hyperparameter exploration, the text integrates weight decay (wd) and dropout as tunable regularization parameters. The chapter_multilayer-perceptrons/weight-decay.md chapter demonstrates how to treat wd as a hyperparameter alongside learning rate, while chapter_multilayer-perceptrons/dropout.md explores dropout probability settings. These regularization hyperparameters interact with model capacity and require joint consideration during tuning, typically evaluated through the same validation framework used for other parameters.

Optimizer Selection and Monitoring

Choosing the Right Optimizer

The choice of optimizer itself constitutes a critical hyperparameter decision. According to chapter_optimization/adam.md, Adam is less sensitive to initial learning rate selection and often works well with default parameters, reducing the tuning burden. Conversely, chapter_optimization/momentum.md discusses how momentum coefficients can accelerate SGD but require careful tuning. Selecting an appropriate optimizer early narrows the effective search space for secondary hyperparameters, with Adam generally recommended for rapid prototyping and SGD with momentum for fine-tuned final models.

Monitoring Training Dynamics

Systematic tuning requires observing both training and validation curves to detect divergence. Throughout the book, particularly in chapter_optimization/optimization-intro.md, plotting utilities visualize loss and accuracy per epoch. Practitioners watch for validation loss increasing while training loss decreases—a signal to adjust learning rate or regularization. This monitoring discipline prevents wasted computation on poorly configured runs and guides intelligent iteration on hyperparameter ranges.

Practical Implementation: Code Examples from d2l-zh

The following snippets illustrate the core tuning workflow used throughout the repository.

Grid Search Over Learning Rate and Hidden Units

This example from chapter_multilayer-perceptrons/mlp-scratch_origin.md demonstrates a systematic sweep over MLP configurations:

from d2l import mxnet as d2l
from mxnet import np, npx
npx.set_np()

def train_mlp(num_hiddens, lr, num_epochs=10):
    net = d2l.nn.Sequential()
    net.add(d2l.nn.Dense(num_hiddens, activation='relu'),
            d2l.nn.Dense(10))
    net.initialize()
    loss = d2l.nn.SoftmaxCrossEntropyLoss()
    trainer = d2l.optim.SGD(lr=lr)
    train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=256)
    d2l.train_ch3(net, train_iter, test_iter, loss,
                   num_epochs, lambda bs: trainer.step(bs))
    return d2l.evaluate_accuracy(net, test_iter)

# Grid‑search over two hyperparameters

candidates = [(256, 0.1), (512, 0.05), (128, 0.2)]
best_acc = 0
for h, lr in candidates:
    acc = train_mlp(h, lr)
    print(f'hiddens={h}, lr={lr} → test accuracy: {acc:.4f}')
    if acc > best_acc:
        best_acc, best_cfg = acc, (h, lr)
print(f'Best config: hidden={best_cfg[0]}, lr={best_cfg[1]}, acc={best_acc:.4f}')

K-Fold Cross-Validation Pipeline

This implementation from chapter_multilayer-perceptrons/kaggle-house-price_origin.md provides robust evaluation across data splits:

def k_fold_cv(k, X, y, lr, weight_decay, num_epochs=100, batch_size=64):
    # Returns mean validation RMSE across k folds

    from d2l import mxnet as d2l
    train_l, valid_l = d2l.k_fold(k, X, y,
                                 num_epochs, lr, weight_decay, batch_size)
    return valid_l

# Example search over learning‑rate and weight‑decay

candidate_lrs = [0.1, 1, 5]
candidate_wd = [0, 1e-4, 5e-4]
best_rmse = float('inf')
for lr in candidate_lrs:
    for wd in candidate_wd:
        rmse = k_fold_cv(k=5, X=train_features, y=train_labels,
                         lr=lr, weight_decay=wd)
        print(f'lr={lr}, wd={wd} → validation RMSE={rmse:.4f}')
        if rmse < best_rmse:
            best_rmse, best_cfg = rmse, (lr, wd)
print(f'Best hyper‑params: lr={best_cfg[0]}, weight_decay={best_cfg[1]}, RMSE={best_rmse:.4f}')

Early Stopping with Validation Monitoring

This pattern from chapter_linear-networks/linear-regression.md prevents overfitting during the tuning process:

def train_with_early_stop(net, train_iter, valid_iter,
                          loss, num_epochs, lr):
    trainer = d2l.optim.SGD(lr=lr)
    best_val_loss = float('inf')
    for epoch in range(num_epochs):
        d2l.train_ch3(net, train_iter, None, loss,
                      1, lambda _: trainer.step(1))
        val_loss = d2l.evaluate_loss(net, valid_iter, loss)
        if val_loss < best_val_loss:
            best_val_loss = val_loss
            net.save_parameters('best.params')
        else:
            print(f'Stopping early at epoch {epoch+1}')
            break
    net.load_parameters('best.params')
    return net

Summary

• Establish a validation set early to measure generalization without contaminating the test set, as demonstrated in chapter_linear-networks/linear-regression.md.
• Begin with sensible defaults (learning rate 0.1, batch size 256, 256 hidden units) before conducting extensive searches.
• Tune one hyperparameter at a time to isolate effects and avoid combinatorial explosion.
• Implement K-fold cross-validation using get_k_fold_data and k_fold from chapter_multilayer-perceptrons/kaggle-house-price_origin.md for robust evaluation on smaller datasets.
• Integrate regularization (weight decay and dropout) as tunable hyperparameters to control model capacity.
• Select optimizers strategically—Adam reduces learning rate sensitivity while SGD with momentum offers fine-grained control.
• Monitor training and validation curves to detect overfitting early and guide iterative refinement.

Frequently Asked Questions

What is the recommended starting learning rate for SGD in d2l-zh?

The repository recommends starting with a learning rate of 0.1 for standard SGD, as implemented in chapter_multilayer-perceptrons/mlp-scratch_origin.md. However, when using adaptive optimizers like Adam described in chapter_optimization/adam.md, the default learning rate of 0.001 typically works without modification, significantly reducing the tuning burden.

How does K-fold cross-validation work in the d2l-zh implementation?

The implementation in chapter_multilayer-perceptrons/kaggle-house-price_origin.md provides the get_k_fold_data and k_fold functions to partition data into K folds. The procedure trains K separate models, each using K-1 folds for training and the remaining fold for validation, then averages the validation scores. This approach provides more stable hyperparameter rankings than single-split validation, particularly when working with limited data samples.

Should weight decay and learning rate be tuned simultaneously?

While these hyperparameters interact, d2l-zh recommends establishing a baseline learning rate first, then introducing weight decay (wd) as a secondary tuning parameter. The chapter_multilayer-perceptrons/weight-decay.md chapter demonstrates treating wd as a configurable hyperparameter that should be evaluated through the same validation framework used for other parameters, typically using the K-fold cross-validation pipeline from the Kaggle house price example.

What is the best optimizer for reducing hyperparameter tuning effort?

According to chapter_optimization/adam.md, Adam is generally the best choice for reducing tuning burden because it adapts learning rates per-parameter and is less sensitive to the initial global learning rate compared to vanilla SGD. While SGD with momentum (covered in chapter_optimization/momentum.md) can achieve superior final performance, it requires more careful tuning of both the learning rate and momentum coefficients.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}