How to Implement Gradient Descent from Scratch for Linear Regression

Gradient descent optimizes linear regression by iteratively adjusting the weight and bias parameters in the opposite direction of the Mean Squared Error gradient, and this guide walks through the exact pure-Python implementation found in the ai-engineering-from-scratch repository.

Learning to implement gradient descent from scratch for linear regression builds the mathematical intuition required for advanced machine learning engineering. The open-source curriculum rohitg00/ai-engineering-from-scratch provides a complete, dependency-free implementation that demonstrates how this optimization algorithm minimizes prediction error through analytical derivatives. This article breaks down the source code structure, mathematical foundations, and practical usage of the gradient descent trainer contained in phases/02-ml-fundamentals/02-linear-regression/code/linear_regression.py.

The Mathematical Foundation

Linear regression models the relationship y ≈ w·x + b, where w represents the weight (slope) and b represents the bias (intercept). The Mean Squared Error (MSE) loss function quantifies prediction error across n samples:

[ J(w,b)=\frac{1}{n}\sum_{i=1}^{n}(w,x_i+b-y_i)^2 ]

Because this loss is differentiable, we compute gradients analytically using the chain rule. The gradient with respect to w is:

[ \frac{\partial J}{\partial w}= \frac{2}{n}\sum_{i=1}^{n}(w,x_i+b-y_i),x_i ]

The gradient with respect to b is:

[ \frac{\partial J}{\partial b}= \frac{2}{n}\sum_{i=1}^{n}(w,x_i+b-y_i) ]

Gradient descent updates parameters by subtracting the gradient scaled by a learning rate η:

w ← w – η · ∂J/∂w
b ← b – η · ∂J/∂b

Core Implementation in linear_regression.py

The repository implements this algorithm in phases/02-ml-fundamentals/02-linear-regression/code/linear_regression.py. The LinearRegression class encapsulates the full training pipeline without external dependencies.

Initialization and Prediction

The constructor at lines 18‑22 initializes parameters to zero and stores the learning rate:

def __init__(self, learning_rate=0.001):
    self.w = 0.0
    self.b = 0.0
    self.lr = learning_rate

The predict method at lines 25‑27 computes the linear hypothesis:

def predict(self, X):
    return [self.w * x + self.b for x in X]

Computing the Cost Function

The compute_cost method at lines 28‑33 calculates the MSE by averaging squared residuals across the dataset:

def compute_cost(self, X, y):
    n = len(X)
    predictions = self.predict(X)
    cost = sum((pred - actual) ** 2 for pred, actual in zip(predictions, y)) / n
    return cost

Calculating Gradients

The compute_gradients method at lines 34‑39 implements the analytical derivatives derived above:

def compute_gradients(self, X, y):
    n = len(X)
    predictions = self.predict(X)
    dw = (2/n) * sum((pred - actual) * x for pred, actual, x in zip(predictions, y, X))
    db = (2/n) * sum(pred - actual for pred, actual in zip(predictions, y))
    return dw, db

The Training Loop

The fit method at lines 41‑49 orchestrates the iterative optimization. Lines 41‑46 perform the parameter updates, while lines 47‑49 track cost history for convergence monitoring:

def fit(self, X, y, epochs=1000, print_every=100):
    for epoch in range(epochs):
        dw, db = self.compute_gradients(X, y)
        self.w -= self.lr * dw
        self.b -= self.lr * db
        
        cost = self.compute_cost(X, y)
        if epoch % print_every == 0:
            print(f"Epoch {epoch}: Cost = {cost:.4f}")

The class also provides an r_squared method at lines 52‑57 for model evaluation.

Practical Code Examples

Simple 1D Linear Regression

This example trains a model on synthetic data using the LinearRegression class:

from phases.02_ml_fundamentals.02_linear_regression.code.linear_regression import LinearRegression
import random

random.seed(42)
X = [random.uniform(0, 10) for _ in range(100)]
y = [3.0 * x + 7.0 + random.gauss(0, 2.0) for x in X]

model = LinearRegression(learning_rate=0.005)
model.fit(X, y, epochs=1000, print_every=200)

print(f"Learned line: y = {model.w:.4f}·x + {model.b:.4f}")
print(f"R² = {model.r_squared(X, y):.4f}")

Multiple Linear Regression with Standardization

For multi-feature problems, the repository provides MultipleLinearRegression and a standardize utility. Feature scaling is essential for stable gradient descent convergence:

from phases.02_ml_fundamentals.02_linear_regression.code.linear_regression import MultipleLinearRegression, standardize
import random

random.seed(42)
X_multi = [[random.uniform(500, 3000), random.randint(1, 5), random.uniform(0, 50)] for _ in range(100)]
y_multi = [50*x[0] + 10000*x[1] - 1000*x[2] + 50000 + random.gauss(0, 20000) for x in X_multi]

X_scaled, _, _ = standardize(X_multi)
multi = MultipleLinearRegression(n_features=3, learning_rate=0.01)
multi.fit(X_scaled, y_multi, epochs=1000, print_every=200)

print("Weights:", [round(w, 4) for w in multi.weights])
print("R² =", multi.r_squared(X_scaled, y_multi))

Advanced Extensions

The same linear_regression.py file contains additional implementations built on the gradient descent core:

• LinearRegressionNormal: Closed-form solution via the normal equation for comparison.
• Polynomial regression: Extends the feature space for non-linear relationships.
• Ridge (L2) regularization: Adds penalty terms to the cost function to prevent overfitting.

Summary

• Analytical gradients enable efficient parameter updates by computing exact partial derivatives of the MSE loss with respect to w and b.
• The LinearRegression class in linear_regression.py implements a five-stage pipeline: initialization, prediction, cost computation, gradient calculation, and iterative optimization.
• Parameter updates occur via self.w -= self.lr * dw and self.b -= self.lr * db inside the fit method loop.
• Feature standardization is required when using MultipleLinearRegression to ensure all features contribute equally to the gradient magnitude.
• The repository includes evaluation metrics (r_squared) and alternative solvers (normal equation) for validation and comparison.

Frequently Asked Questions

What is gradient descent in the context of linear regression?

Gradient descent is an iterative optimization algorithm that minimizes the Mean Squared Error loss by computing partial derivatives of the cost function with respect to the model parameters. In the ai-engineering-from-scratch implementation, these derivatives are calculated analytically in the compute_gradients method, then used to update the weight and bias via the learning rate scaling factor.

Why must we compute gradients analytically rather than numerically?

Analytical gradients provide exact values and computational efficiency, requiring only a single pass through the dataset per epoch. Numerical approximation would require evaluating the cost function multiple times per parameter (finite differences), introducing rounding errors and increasing computational complexity from O(n) to O(n·p) where p is the number of parameters.

How does the learning rate affect convergence in this implementation?

The learning rate (learning_rate parameter in __init__) controls the step size during parameter updates in the fit method. A rate that is too large causes divergence or oscillation around the minimum, while a rate that is too small results in prohibitively slow convergence requiring excessive epochs. The repository uses 0.001 as a default baseline for 1D problems and 0.01 for standardized multi-feature data.

Can this implementation handle datasets with multiple input features?

Yes, the repository provides the MultipleLinearRegression class which extends the gradient descent logic to vectorized weights for multi-dimensional inputs. However, because gradient descent is sensitive to feature scales, the implementation includes a standardize function that performs z-score normalization, ensuring that features with larger magnitudes (like square footage) do not dominate the gradient updates compared to smaller-scale features (like bedroom count).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}