# How d2l-zh Explains Gradient Descent and Its Variants: From First Principles to Adaptive Optimization

> Explore how d2l-zh explains gradient descent and its variants using Taylor expansion, stochastic sampling, momentum, and adaptive learning rates for efficient optimization.

- Repository: [Dive into Deep Learning (D2L.ai)/d2l-zh](https://github.com/d2l-ai/d2l-zh)
- Tags: deep-dive
- Published: 2026-03-01

---

**The d2l-zh repository teaches gradient descent and its variants by starting with the scalar Taylor expansion, deriving the negative gradient update rule, and progressively layering in stochastic sampling, momentum, and per-coordinate adaptive learning rates.**

The Chinese edition of *Dive into Deep Learning* (d2l-zh) provides a mathematically rigorous yet accessible pedagogical pipeline for understanding optimization. Rather than treating algorithms as black-box hyperparameters, the text anchors every variant—from vanilla gradient descent to Adam—to the core intuition of adjusting step direction and magnitude to minimize an objective function.

## Core Foundations of Gradient Descent

### Mathematical Derivation from First Principles

In [`chapter_optimization/gd_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_optimization/gd_origin.md), the authors introduce gradient descent by considering a scalar objective $f(x)$. Using a first-order Taylor expansion, they demonstrate that moving a small step $-\eta f'(x)$ in the direction of the **negative gradient** guarantees a reduction in function value. This derivation appears in lines 10–28, where the text explicitly connects the gradient to the direction of steepest descent.

The repository provides a concrete implementation in Python:

```python
def f(x): 
    return x ** 2

def f_grad(x): 
    return 2 * x

def gd(eta, f_grad):
    x = 10.0
    results = [x]
    for _ in range(10):
        x -= eta * f_grad(x)
        results.append(float(x))
    return results

results = gd(0.2, f_grad)  # Converges to 0 with eta=0.2

```

### The Critical Role of Learning Rate

The text emphasizes that the **learning rate** $\eta$ is not merely a hyperparameter but a stability constant. In [`gd_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/gd_origin.md) (lines 96–107), the authors illustrate divergence using $\eta = 1.1$ on the quadratic objective $f(x) = x^2$. When the step size exceeds the threshold allowed by the local curvature, the iterates oscillate with increasing magnitude, demonstrating the necessity of tuning $\eta$ to the problem's Lipschitz constant.

### Extension to Multivariate Optimization

The generalization to vector-valued parameters $\mathbf{x} \in \mathbb{R}^d$ appears in the same chapter. The authors define the **gradient vector** $\nabla f(\mathbf{x})$ and present the update rule:

$$
\mathbf{x} \leftarrow \mathbf{x} - \eta \nabla f(\mathbf{x})
$$

A concrete 2-D example $f(\mathbf{x}) = x_1^2 + 2x_2^2$ visualizes the trajectory, showing how gradient descent follows a curved path toward the origin due to differing curvatures along each axis. The corresponding code in [`gd_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/gd_origin.md) uses `show_trace_2d` to render the optimization path.

## Stochastic and Mini-Batch Variants

### Stochastic Gradient Descent Fundamentals

In [`chapter_optimization/sgd_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_optimization/sgd_origin.md), the text introduces **Stochastic Gradient Descent (SGD)** as a practical approximation to full-batch gradient descent. Rather than computing the gradient over the entire dataset (cost $O(n)$), SGD samples a single example uniformly at random, reducing the per-step cost to $O(1)$. Lines 50–58 establish that this stochastic gradient is an **unbiased estimator** of the true gradient, preserving the expected direction of descent while introducing variance that can aid escape from shallow local minima.

### Mini-Batch Trade-offs

The repository extends SGD to **mini-batch** processing, where the gradient is computed over a small random subset of size `batch_size`. In [`sgd_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/sgd_origin.md) (lines 270–282), the authors analyze the statistical efficiency versus computational throughput trade-off: larger batches provide more accurate gradient estimates but reduce the number of parameter updates per epoch, while smaller batches offer faster convergence in wall-clock time on modern hardware due to vectorized operations.

The implementation wraps the optimization loop:

```python
def train_sgd(net, train_iter, loss, num_epochs, batch_size, eta):
    trainer = d2l.SGD(net.parameters(), learning_rate=eta, batch_size=batch_size)
    for epoch in range(num_epochs):
        for X, y in train_iter:
            with d2l.autograd.record():
                l = loss(net(X), y)
            l.backward()
            trainer.step(batch_size)

```

## Adaptive Optimization Methods

### Momentum and Velocity Accumulation

The text addresses SGD's limitation in navigating ravines—areas where the curvature varies significantly across dimensions. In [`chapter_optimization/momentum_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_optimization/momentum_origin.md), **Momentum** is introduced as a method that accumulates a velocity vector $\mathbf{v}$ using an exponential moving average of past gradients:

$$
\mathbf{v} \leftarrow \beta \mathbf{v} + (1 - \beta) \nabla f(\mathbf{x})
$$

$$
\mathbf{x} \leftarrow \mathbf{x} - \eta \mathbf{v}
$$

Lines 38–46 derive this update, explaining how the momentum coefficient $\beta \in [0,1)$ effectively smooths the optimization trajectory, dampening oscillations in high-curvature directions and accelerating progress along consistent directions.

### Per-Coordinate Adaptation with AdaGrad

In [`chapter_optimization/adagrad_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_optimization/adagrad_origin.md), the authors present **AdaGrad** as a solution to sparse gradient problems. The algorithm maintains a running sum of squared gradients $\mathbf{s}$:

$$
\mathbf{s} \leftarrow \mathbf{s} + \nabla f(\mathbf{x}) \odot \nabla f(\mathbf{x})
$$

The update then scales each coordinate inversely proportional to the square root of its accumulated history:

$$
\mathbf{x} \leftarrow \mathbf{x} - \frac{\eta}{\sqrt{\mathbf{s}} + \epsilon} \odot \nabla f(\mathbf{x})
$$

Lines 15–23 emphasize that this provides **per-coordinate learning rates**, automatically reducing the step size for frequently updated parameters while keeping large steps for rare features.

### RMSProp and Exponential Moving Averages

The repository identifies AdaGrad's critical flaw: the learning rate monotonically decreases to zero, eventually halting training. In [`chapter_optimization/rmsprop_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_optimization/rmsprop_origin.md), **RMSProp** fixes this by replacing the cumulative sum with an exponential moving average:

$$
\mathbf{s} \leftarrow \rho \mathbf{s} + (1 - \rho) \nabla f(\mathbf{x}) \odot \nabla f(\mathbf{x})
$$

Lines 5–9 explain that the decay rate $\rho$ (typically 0.9) allows the algorithm to "forget" old gradients, preventing the aggressive decay that plagues AdaGrad while retaining the per-coordinate scaling benefits.

### Adam and Beyond

While not detailed in the provided excerpts, [`chapter_optimization/adam_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_optimization/adam_origin.md) completes the pedagogical arc by combining **momentum** (first moment) with **RMSProp-style adaptation** (second moment), resulting in the Adam optimizer that dominates modern deep learning practice.

## Second-Order and Advanced Techniques

### Newton's Method

Beyond first-order methods, [`gd_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/gd_origin.md) (lines 98–108) introduces **Newton's method**, which utilizes second-order curvature via the Hessian matrix $\mathbf{H}$:

$$
\mathbf{x} \leftarrow \mathbf{x} - \mathbf{H}^{-1} \nabla f(\mathbf{x})
$$

This approach achieves **quadratic convergence** on convex problems but at the cost of $O(d^2)$ memory and computation, making it impractical for high-dimensional deep learning without approximations.

### Line Search and Preconditioning

The text also covers practical enhancements to basic gradient descent:

- **Line-search GD**: Augments the algorithm with a binary search to select an optimal $\eta$ each iteration, improving robustness at the cost of extra function evaluations (lines 19–23 in [`gd_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/gd_origin.md)).
- **Preconditioning**: Scales each coordinate by the diagonal of the Hessian (or an approximation), effectively giving each variable its own learning rate and mitigating scale mismatch between parameters (lines 7–15 in [`gd_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/gd_origin.md)).

## Summary

- d2l-zh teaches **gradient descent and its variants** by deriving the core update rule from Taylor expansions, establishing that the negative gradient direction minimizes local function values.
- The repository emphasizes the **learning rate** as a stability constant, demonstrating divergence when $\eta$ exceeds curvature limits and convergence when properly tuned.
- **Stochastic Gradient Descent (SGD)** and **mini-batch** processing reduce per-step complexity from $O(n)$ to $O(1)$, trading gradient variance for computational efficiency.
- **Adaptive methods** (Momentum, AdaGrad, RMSProp, Adam) address specific pathologies: momentum dampens oscillations, AdaGrad provides per-coordinate scaling, and RMSProp fixes AdaGrad's aggressive learning rate decay via exponential moving averages.
- **Second-order methods** like Newton's method offer quadratic convergence but remain impractical for high-dimensional deep learning due to Hessian computation costs.

## Frequently Asked Questions

### What is the fundamental intuition behind gradient descent according to d2l-zh?

According to the source code in [`chapter_optimization/gd_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_optimization/gd_origin.md), the fundamental intuition derives from a first-order Taylor expansion of the objective function. The text demonstrates that moving a small step $-\eta f'(x)$ in the direction of the **negative gradient** guarantees a reduction in the function value for sufficiently small $\eta$. This establishes gradient descent as a method that iteratively adjusts parameters by following the locally steepest downward slope.

### How does d2l-zh explain the difference between Stochastic Gradient Descent and full-batch gradient descent?

In [`chapter_optimization/sgd_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_optimization/sgd_origin.md), the authors explain that full-batch gradient descent computes the gradient over the entire dataset, incurring a per-step cost of $O(n)$ where $n$ is the number of examples. In contrast, **Stochastic Gradient Descent** samples a single example uniformly at random, reducing the per-step cost to $O(1)$. The text emphasizes that while this introduces variance into the gradient estimate, the stochastic gradient remains an **unbiased estimator** of the true gradient, preserving the expected direction of descent while significantly accelerating computation.

### Why does d2l-zh introduce RMSProp as an improvement over AdaGrad?

The repository identifies a critical limitation of AdaGrad in [`chapter_optimization/rmsprop_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_optimization/rmsprop_origin.md): because AdaGrad accumulates squared gradients indefinitely, the effective learning rate decays monotonically toward zero, eventually halting training prematurely. **RMSProp** resolves this by replacing the cumulative sum with an **exponential moving average** of squared gradients, controlled by a decay rate $\rho$ (typically 0.9). This allows the algorithm to "forget" stale gradient information, preventing aggressive decay while maintaining the per-coordinate scaling benefits that make AdaGrad effective for sparse gradients.

### What second-order optimization methods does d2l-zh discuss, and why are they less common in deep learning?

According to [`chapter_optimization/gd_origin.md`](https://github.com/d2l-ai/d2l-zh/blob/main/chapter_optimization/gd_origin.md), the text introduces **Newton's method**, which utilizes the Hessian matrix $\mathbf{H}$ to compute the update step $-\mathbf{H}^{-1}\nabla f(\mathbf{x})$. This method achieves **quadratic convergence** on convex problems, significantly faster than the linear convergence of first-order methods. However, the repository notes that Newton's method requires $O(d^2)$ memory to store the Hessian and $O(d^3)$ computation to invert it, making it impractical for high-dimensional deep learning models where $d$ can be in the millions. Consequently, first-order methods with adaptive learning rates remain the practical standard.