From OLS to neural networks

Start with OLS regression: ŷ = wx + b. The model is linear; it has two parameters; we find them by minimising the sum of squared residuals. Replace x with a vector and (w, b) with a matrix and a bias, and we have multivariate regression: ŷ = Wx + b.

Stack a few regressions and you have a neural network

Take that linear layer's output and run it through a non-linear function — say ReLU, which is f(x) = max(0, x). Then feed the result into another linear layer. Then another. That stack is a neural network. The non-linear function between layers is what makes the whole thing capable of representing complex relationships.

import torch.nn as nn

net = nn.Sequential(
    nn.Linear(in_features=10, out_features=64),
    nn.ReLU(),
    nn.Linear(64, 32),
    nn.ReLU(),
    nn.Linear(32, 1)
)

The loss function and gradient descent

Same as OLS: define a loss (mean squared error for regression, cross-entropy for classification), then find the weights that minimise it. The difference: with millions or billions of weights, you can't solve for the minimum analytically. You move iteratively.

Gradient descent: compute the gradient of the loss with respect to each weight, then nudge each weight in the direction that reduces loss. Repeat. Stochastic gradient descent does this on small random batches of data rather than the full dataset, which is faster and helps the model generalise.

Backpropagation

Computing the gradient through a deep network looks intractable but isn't. Backpropagation is just the chain rule applied systematically: the gradient at any layer is the gradient at the next layer times the local derivative. Modern frameworks (PyTorch, TensorFlow, JAX) compute this automatically.

Why this scales

Stack more layers, widen them, train on more data — that's the story of deep learning from 2012 to 2024. The transformer architecture is one specific stack of layers (attention + feedforward + normalisation) that turned out to scale particularly well.