# How to Implement Gradient Descent from Scratch for Linear Regression

> Implement gradient descent from scratch for linear regression. Learn to optimize parameters iteratively with a pure Python guide from the ai-engineering-from-scratch repository.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-07-31

---

**Gradient descent optimizes linear regression by iteratively adjusting the weight and bias parameters in the opposite direction of the Mean Squared Error gradient, and this guide walks through the exact pure-Python implementation found in the `ai-engineering-from-scratch` repository.**

Learning to implement gradient descent from scratch for linear regression builds the mathematical intuition required for advanced machine learning engineering. The open-source curriculum `rohitg00/ai-engineering-from-scratch` provides a complete, dependency-free implementation that demonstrates how this optimization algorithm minimizes prediction error through analytical derivatives. This article breaks down the source code structure, mathematical foundations, and practical usage of the gradient descent trainer contained in [`phases/02-ml-fundamentals/02-linear-regression/code/linear_regression.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/02-ml-fundamentals/02-linear-regression/code/linear_regression.py).

## The Mathematical Foundation

Linear regression models the relationship **y ≈ w·x + b**, where **w** represents the weight (slope) and **b** represents the bias (intercept). The **Mean Squared Error (MSE)** loss function quantifies prediction error across **n** samples:

\[
J(w,b)=\frac{1}{n}\sum_{i=1}^{n}(w\,x_i+b-y_i)^2
\]

Because this loss is differentiable, we compute gradients analytically using the chain rule. The **gradient with respect to w** is:

\[
\frac{\partial J}{\partial w}= \frac{2}{n}\sum_{i=1}^{n}(w\,x_i+b-y_i)\,x_i
\]

The **gradient with respect to b** is:

\[
\frac{\partial J}{\partial b}= \frac{2}{n}\sum_{i=1}^{n}(w\,x_i+b-y_i)
\]

Gradient descent updates parameters by subtracting the gradient scaled by a **learning rate η**:

```

w ← w – η · ∂J/∂w
b ← b – η · ∂J/∂b

```

## Core Implementation in [`linear_regression.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/linear_regression.py)

The repository implements this algorithm in **[`phases/02-ml-fundamentals/02-linear-regression/code/linear_regression.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/02-ml-fundamentals/02-linear-regression/code/linear_regression.py)**. The `LinearRegression` class encapsulates the full training pipeline without external dependencies.

### Initialization and Prediction

The constructor at **lines 18‑22** initializes parameters to zero and stores the learning rate:

```python
def __init__(self, learning_rate=0.001):
    self.w = 0.0
    self.b = 0.0
    self.lr = learning_rate

```

The `predict` method at **lines 25‑27** computes the linear hypothesis:

```python
def predict(self, X):
    return [self.w * x + self.b for x in X]

```

### Computing the Cost Function

The `compute_cost` method at **lines 28‑33** calculates the MSE by averaging squared residuals across the dataset:

```python
def compute_cost(self, X, y):
    n = len(X)
    predictions = self.predict(X)
    cost = sum((pred - actual) ** 2 for pred, actual in zip(predictions, y)) / n
    return cost

```

### Calculating Gradients

The `compute_gradients` method at **lines 34‑39** implements the analytical derivatives derived above:

```python
def compute_gradients(self, X, y):
    n = len(X)
    predictions = self.predict(X)
    dw = (2/n) * sum((pred - actual) * x for pred, actual, x in zip(predictions, y, X))
    db = (2/n) * sum(pred - actual for pred, actual in zip(predictions, y))
    return dw, db

```

### The Training Loop

The `fit` method at **lines 41‑49** orchestrates the iterative optimization. **Lines 41‑46** perform the parameter updates, while **lines 47‑49** track cost history for convergence monitoring:

```python
def fit(self, X, y, epochs=1000, print_every=100):
    for epoch in range(epochs):
        dw, db = self.compute_gradients(X, y)
        self.w -= self.lr * dw
        self.b -= self.lr * db
        
        cost = self.compute_cost(X, y)
        if epoch % print_every == 0:
            print(f"Epoch {epoch}: Cost = {cost:.4f}")

```

The class also provides an `r_squared` method at **lines 52‑57** for model evaluation.

## Practical Code Examples

### Simple 1D Linear Regression

This example trains a model on synthetic data using the `LinearRegression` class:

```python
from phases.02_ml_fundamentals.02_linear_regression.code.linear_regression import LinearRegression
import random

random.seed(42)
X = [random.uniform(0, 10) for _ in range(100)]
y = [3.0 * x + 7.0 + random.gauss(0, 2.0) for x in X]

model = LinearRegression(learning_rate=0.005)
model.fit(X, y, epochs=1000, print_every=200)

print(f"Learned line: y = {model.w:.4f}·x + {model.b:.4f}")
print(f"R² = {model.r_squared(X, y):.4f}")

```

### Multiple Linear Regression with Standardization

For multi-feature problems, the repository provides `MultipleLinearRegression` and a `standardize` utility. Feature scaling is essential for stable gradient descent convergence:

```python
from phases.02_ml_fundamentals.02_linear_regression.code.linear_regression import MultipleLinearRegression, standardize
import random

random.seed(42)
X_multi = [[random.uniform(500, 3000), random.randint(1, 5), random.uniform(0, 50)] for _ in range(100)]
y_multi = [50*x[0] + 10000*x[1] - 1000*x[2] + 50000 + random.gauss(0, 20000) for x in X_multi]

X_scaled, _, _ = standardize(X_multi)
multi = MultipleLinearRegression(n_features=3, learning_rate=0.01)
multi.fit(X_scaled, y_multi, epochs=1000, print_every=200)

print("Weights:", [round(w, 4) for w in multi.weights])
print("R² =", multi.r_squared(X_scaled, y_multi))

```

### Advanced Extensions

The same [`linear_regression.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/linear_regression.py) file contains additional implementations built on the gradient descent core:
- **`LinearRegressionNormal`**: Closed-form solution via the normal equation for comparison.
- **Polynomial regression**: Extends the feature space for non-linear relationships.
- **Ridge (L2) regularization**: Adds penalty terms to the cost function to prevent overfitting.

## Summary

- **Analytical gradients** enable efficient parameter updates by computing exact partial derivatives of the MSE loss with respect to **w** and **b**.
- The **`LinearRegression`** class in [`linear_regression.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/linear_regression.py) implements a five-stage pipeline: initialization, prediction, cost computation, gradient calculation, and iterative optimization.
- **Parameter updates** occur via `self.w -= self.lr * dw` and `self.b -= self.lr * db` inside the `fit` method loop.
- **Feature standardization** is required when using `MultipleLinearRegression` to ensure all features contribute equally to the gradient magnitude.
- The repository includes evaluation metrics (`r_squared`) and alternative solvers (normal equation) for validation and comparison.

## Frequently Asked Questions

### What is gradient descent in the context of linear regression?

Gradient descent is an iterative optimization algorithm that minimizes the Mean Squared Error loss by computing partial derivatives of the cost function with respect to the model parameters. In the `ai-engineering-from-scratch` implementation, these derivatives are calculated analytically in the `compute_gradients` method, then used to update the weight and bias via the learning rate scaling factor.

### Why must we compute gradients analytically rather than numerically?

Analytical gradients provide exact values and computational efficiency, requiring only a single pass through the dataset per epoch. Numerical approximation would require evaluating the cost function multiple times per parameter (finite differences), introducing rounding errors and increasing computational complexity from O(n) to O(n·p) where p is the number of parameters.

### How does the learning rate affect convergence in this implementation?

The learning rate (`learning_rate` parameter in `__init__`) controls the step size during parameter updates in the `fit` method. A rate that is too large causes divergence or oscillation around the minimum, while a rate that is too small results in prohibitively slow convergence requiring excessive epochs. The repository uses 0.001 as a default baseline for 1D problems and 0.01 for standardized multi-feature data.

### Can this implementation handle datasets with multiple input features?

Yes, the repository provides the `MultipleLinearRegression` class which extends the gradient descent logic to vectorized weights for multi-dimensional inputs. However, because gradient descent is sensitive to feature scales, the implementation includes a `standardize` function that performs z-score normalization, ensuring that features with larger magnitudes (like square footage) do not dominate the gradient updates compared to smaller-scale features (like bedroom count).