How d2l-zh Explains Gradient Descent and Its Variants: From First Principles to Adaptive Optimization

The d2l-zh repository teaches gradient descent and its variants by starting with the scalar Taylor expansion, deriving the negative gradient update rule, and progressively layering in stochastic sampling, momentum, and per-coordinate adaptive learning rates.

The Chinese edition of Dive into Deep Learning (d2l-zh) provides a mathematically rigorous yet accessible pedagogical pipeline for understanding optimization. Rather than treating algorithms as black-box hyperparameters, the text anchors every variant—from vanilla gradient descent to Adam—to the core intuition of adjusting step direction and magnitude to minimize an objective function.

Core Foundations of Gradient Descent

Mathematical Derivation from First Principles

In chapter_optimization/gd_origin.md, the authors introduce gradient descent by considering a scalar objective $f(x)$. Using a first-order Taylor expansion, they demonstrate that moving a small step $-\eta f'(x)$ in the direction of the negative gradient guarantees a reduction in function value. This derivation appears in lines 10–28, where the text explicitly connects the gradient to the direction of steepest descent.

The repository provides a concrete implementation in Python:

def f(x): 
    return x ** 2

def f_grad(x): 
    return 2 * x

def gd(eta, f_grad):
    x = 10.0
    results = [x]
    for _ in range(10):
        x -= eta * f_grad(x)
        results.append(float(x))
    return results

results = gd(0.2, f_grad)  # Converges to 0 with eta=0.2

The Critical Role of Learning Rate

The text emphasizes that the learning rate $\eta$ is not merely a hyperparameter but a stability constant. In gd_origin.md (lines 96–107), the authors illustrate divergence using $\eta = 1.1$ on the quadratic objective $f(x) = x^2$. When the step size exceeds the threshold allowed by the local curvature, the iterates oscillate with increasing magnitude, demonstrating the necessity of tuning $\eta$ to the problem's Lipschitz constant.

Extension to Multivariate Optimization

The generalization to vector-valued parameters $\mathbf{x} \in \mathbb{R}^d$ appears in the same chapter. The authors define the gradient vector $\nabla f(\mathbf{x})$ and present the update rule:

$$ \mathbf{x} \leftarrow \mathbf{x} - \eta \nabla f(\mathbf{x}) $$

A concrete 2-D example $f(\mathbf{x}) = x_1^2 + 2x_2^2$ visualizes the trajectory, showing how gradient descent follows a curved path toward the origin due to differing curvatures along each axis. The corresponding code in gd_origin.md uses show_trace_2d to render the optimization path.

Stochastic and Mini-Batch Variants

Stochastic Gradient Descent Fundamentals

In chapter_optimization/sgd_origin.md, the text introduces Stochastic Gradient Descent (SGD) as a practical approximation to full-batch gradient descent. Rather than computing the gradient over the entire dataset (cost $O(n)$), SGD samples a single example uniformly at random, reducing the per-step cost to $O(1)$. Lines 50–58 establish that this stochastic gradient is an unbiased estimator of the true gradient, preserving the expected direction of descent while introducing variance that can aid escape from shallow local minima.

Mini-Batch Trade-offs

The repository extends SGD to mini-batch processing, where the gradient is computed over a small random subset of size batch_size. In sgd_origin.md (lines 270–282), the authors analyze the statistical efficiency versus computational throughput trade-off: larger batches provide more accurate gradient estimates but reduce the number of parameter updates per epoch, while smaller batches offer faster convergence in wall-clock time on modern hardware due to vectorized operations.

The implementation wraps the optimization loop:

def train_sgd(net, train_iter, loss, num_epochs, batch_size, eta):
    trainer = d2l.SGD(net.parameters(), learning_rate=eta, batch_size=batch_size)
    for epoch in range(num_epochs):
        for X, y in train_iter:
            with d2l.autograd.record():
                l = loss(net(X), y)
            l.backward()
            trainer.step(batch_size)

Adaptive Optimization Methods

Momentum and Velocity Accumulation

The text addresses SGD's limitation in navigating ravines—areas where the curvature varies significantly across dimensions. In chapter_optimization/momentum_origin.md, Momentum is introduced as a method that accumulates a velocity vector $\mathbf{v}$ using an exponential moving average of past gradients:

$$ \mathbf{v} \leftarrow \beta \mathbf{v} + (1 - \beta) \nabla f(\mathbf{x}) $$

$$ \mathbf{x} \leftarrow \mathbf{x} - \eta \mathbf{v} $$

Lines 38–46 derive this update, explaining how the momentum coefficient $\beta \in [0,1)$ effectively smooths the optimization trajectory, dampening oscillations in high-curvature directions and accelerating progress along consistent directions.

Per-Coordinate Adaptation with AdaGrad

In chapter_optimization/adagrad_origin.md, the authors present AdaGrad as a solution to sparse gradient problems. The algorithm maintains a running sum of squared gradients $\mathbf{s}$:

$$ \mathbf{s} \leftarrow \mathbf{s} + \nabla f(\mathbf{x}) \odot \nabla f(\mathbf{x}) $$

The update then scales each coordinate inversely proportional to the square root of its accumulated history:

$$ \mathbf{x} \leftarrow \mathbf{x} - \frac{\eta}{\sqrt{\mathbf{s}} + \epsilon} \odot \nabla f(\mathbf{x}) $$

Lines 15–23 emphasize that this provides per-coordinate learning rates, automatically reducing the step size for frequently updated parameters while keeping large steps for rare features.

RMSProp and Exponential Moving Averages

The repository identifies AdaGrad's critical flaw: the learning rate monotonically decreases to zero, eventually halting training. In chapter_optimization/rmsprop_origin.md, RMSProp fixes this by replacing the cumulative sum with an exponential moving average:

$$ \mathbf{s} \leftarrow \rho \mathbf{s} + (1 - \rho) \nabla f(\mathbf{x}) \odot \nabla f(\mathbf{x}) $$

Lines 5–9 explain that the decay rate $\rho$ (typically 0.9) allows the algorithm to "forget" old gradients, preventing the aggressive decay that plagues AdaGrad while retaining the per-coordinate scaling benefits.

Adam and Beyond

While not detailed in the provided excerpts, chapter_optimization/adam_origin.md completes the pedagogical arc by combining momentum (first moment) with RMSProp-style adaptation (second moment), resulting in the Adam optimizer that dominates modern deep learning practice.

Second-Order and Advanced Techniques

Newton's Method

Beyond first-order methods, gd_origin.md (lines 98–108) introduces Newton's method, which utilizes second-order curvature via the Hessian matrix $\mathbf{H}$:

$$ \mathbf{x} \leftarrow \mathbf{x} - \mathbf{H}^{-1} \nabla f(\mathbf{x}) $$

This approach achieves quadratic convergence on convex problems but at the cost of $O(d^2)$ memory and computation, making it impractical for high-dimensional deep learning without approximations.

Line Search and Preconditioning

The text also covers practical enhancements to basic gradient descent:

• Line-search GD: Augments the algorithm with a binary search to select an optimal $\eta$ each iteration, improving robustness at the cost of extra function evaluations (lines 19–23 in gd_origin.md).
• Preconditioning: Scales each coordinate by the diagonal of the Hessian (or an approximation), effectively giving each variable its own learning rate and mitigating scale mismatch between parameters (lines 7–15 in gd_origin.md).

Summary

• d2l-zh teaches gradient descent and its variants by deriving the core update rule from Taylor expansions, establishing that the negative gradient direction minimizes local function values.
• The repository emphasizes the learning rate as a stability constant, demonstrating divergence when $\eta$ exceeds curvature limits and convergence when properly tuned.
• Stochastic Gradient Descent (SGD) and mini-batch processing reduce per-step complexity from $O(n)$ to $O(1)$, trading gradient variance for computational efficiency.
• Adaptive methods (Momentum, AdaGrad, RMSProp, Adam) address specific pathologies: momentum dampens oscillations, AdaGrad provides per-coordinate scaling, and RMSProp fixes AdaGrad's aggressive learning rate decay via exponential moving averages.
• Second-order methods like Newton's method offer quadratic convergence but remain impractical for high-dimensional deep learning due to Hessian computation costs.

Frequently Asked Questions

What is the fundamental intuition behind gradient descent according to d2l-zh?

According to the source code in chapter_optimization/gd_origin.md, the fundamental intuition derives from a first-order Taylor expansion of the objective function. The text demonstrates that moving a small step $-\eta f'(x)$ in the direction of the negative gradient guarantees a reduction in the function value for sufficiently small $\eta$. This establishes gradient descent as a method that iteratively adjusts parameters by following the locally steepest downward slope.

How does d2l-zh explain the difference between Stochastic Gradient Descent and full-batch gradient descent?

In chapter_optimization/sgd_origin.md, the authors explain that full-batch gradient descent computes the gradient over the entire dataset, incurring a per-step cost of $O(n)$ where $n$ is the number of examples. In contrast, Stochastic Gradient Descent samples a single example uniformly at random, reducing the per-step cost to $O(1)$. The text emphasizes that while this introduces variance into the gradient estimate, the stochastic gradient remains an unbiased estimator of the true gradient, preserving the expected direction of descent while significantly accelerating computation.

Why does d2l-zh introduce RMSProp as an improvement over AdaGrad?

The repository identifies a critical limitation of AdaGrad in chapter_optimization/rmsprop_origin.md: because AdaGrad accumulates squared gradients indefinitely, the effective learning rate decays monotonically toward zero, eventually halting training prematurely. RMSProp resolves this by replacing the cumulative sum with an exponential moving average of squared gradients, controlled by a decay rate $\rho$ (typically 0.9). This allows the algorithm to "forget" stale gradient information, preventing aggressive decay while maintaining the per-coordinate scaling benefits that make AdaGrad effective for sparse gradients.

What second-order optimization methods does d2l-zh discuss, and why are they less common in deep learning?

According to chapter_optimization/gd_origin.md, the text introduces Newton's method, which utilizes the Hessian matrix $\mathbf{H}$ to compute the update step $-\mathbf{H}^{-1}\nabla f(\mathbf{x})$. This method achieves quadratic convergence on convex problems, significantly faster than the linear convergence of first-order methods. However, the repository notes that Newton's method requires $O(d^2)$ memory to store the Hessian and $O(d^3)$ computation to invert it, making it impractical for high-dimensional deep learning models where $d$ can be in the millions. Consequently, first-order methods with adaptive learning rates remain the practical standard.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}