Deriving the Least Squares Solution: From Simple Regression to the Normal Equations
Retracing the derivation from first principles — from a bowl-shaped loss surface to the compact matrix equation β = (XᵀX)⁻¹Xᵀy.
Linear regression is arguably the most widely used tool in statistics and machine learning, and at its heart lies a beautifully compact formula: β = (XᵀX)⁻¹Xᵀy. This equation, involving nothing more than matrix multiplication, transposition, and inversion, delivers the optimal parameters for a regression model in a single step. But where does it come from? In this article, we retrace the derivation from first principles, beginning with simple linear regression and ending with a general result that applies to any number of features.
The Setup: Simple Linear Regression
In simple linear regression, we assume that each prediction is generated by a straight-line relationship:
ŷᵢ = β₀ + β₁xᵢ
Here, β₀ is the intercept and β₁ is the slope. To make this concrete, consider a housing example: let xᵢ be the square footage of house i and yᵢ its price. The intercept β₀ represents the “base price” of a hypothetical house with zero square footage — not physically meaningful, but mathematically necessary — while β₁ tells us how much the price increases for each additional unit of square footage.
Given a scatter of data points, our goal is to find the specific values of β₀ and β₁ that produce the line that best matches the data. But “best” needs a precise definition. We need a metric that quantifies how badly a candidate line is doing, and then we need to minimize it.
The Loss Function: Sum of Squared Errors
The metric of choice is the sum of squared errors (SSE):
L(β₀, β₁) = Σᵢ₌₁ⁿ (yᵢ − β₀ − β₁xᵢ)²
For each of our n data points, we take the actual value yᵢ (the true price of the house), subtract our prediction β₀ + β₁xᵢ, and square the result. The discrepancy between the actual and predicted value is the error, often denoted εᵢ. In the first figure above, these errors appear as the vertical gray segments connecting each data point to the fitted line.
Why square the errors? Two reasons stand out:
The sign of the error stops mattering. Whether we overprice or underprice a house, squaring makes the contribution to the loss positive. An over-prediction and an under-prediction of the same magnitude are penalized equally.
Large errors are penalized heavily. Because the penalty grows quadratically, points far from the line contribute disproportionately to the loss. The fit is pushed strongly toward avoiding big mistakes.
Squaring also gives us mathematical niceties: the loss becomes a smooth, differentiable function of the parameters, which is exactly what we need for calculus.
A crucial observation is that L depends only on β₀ and β₁. The data (the xᵢ and yᵢ values) are fixed; the only things we can change are the intercept — which moves the line up or down — and the slope, which rotates it. The loss is therefore a function over parameter space, and as the 3D figure shows, it forms a bowl-shaped surface with a single minimum.
Minimizing with Calculus
From calculus, we know that a smooth function attains its minimum where its slope is zero. Since our loss depends on two variables, we must look at the gradient: we take the partial derivative of L with respect to each parameter and set both to zero:
∂L/∂β₀ = 0 and ∂L/∂β₁ = 0
Geometrically, we are searching for the bottom of the bowl — the point (β₀, β₁) where the surface is flat in every direction, marked by the red star in the contour plot.
Derivative with respect to β₀. Applying the chain rule to L = Σ(yᵢ − β₀ − β₁xᵢ)², we bring the exponent down and multiply by the derivative of the inner function. Inside the parentheses, yᵢ is a constant (derivative 0), β₀ differentiates to 1, and β₁xᵢ is constant with respect to β₀ (derivative 0). The inner derivative is therefore −1, and:
∂L/∂β₀ = −2 Σᵢ (yᵢ − β₀ − β₁xᵢ)
Derivative with respect to β₁. The same logic applies, except now the inner derivative comes from the term β₁xᵢ: since xᵢ is a constant multiplier, the derivative is −xᵢ. Hence:
∂L/∂β₁ = −2 Σᵢ (yᵢ − β₀ − β₁xᵢ) xᵢ
Setting both expressions to zero, the factors of −2 drop out (a zero on the right absorbs any nonzero constant), leaving two equations in two unknowns.
The Normal Equations
Expanding the summations and rearranging so that all β terms sit on the left and the data sit on the right gives:
Equation 1: nβ₀ + β₁ Σxᵢ = Σyᵢ
Equation 2: β₀ Σxᵢ + β₁ Σxᵢ² = Σxᵢyᵢ
(Note the simplification in Equation 1: summing the constant β₀ over n data points just yields nβ₀; similarly, constants like β₀ and β₁ factor out of the sums.)
These are the famous normal equations. The name reflects their normalized structure: the left-hand side is a linear combination of the unknown parameters with quantities built from the inputs, while the right-hand side contains only the observed data — the labels y we are trying to predict and the inputs x we are using to predict them.
The Matrix Formulation
The structure of the normal equations looks suspiciously like the result of a matrix multiplication — and that is exactly what it is. Define the following objects:
X, the design matrix: an n × 2 matrix whose first column is all ones (for the intercept) and whose second column holds the inputs x₁, …, xₙ (the square footages).
y, an n × 1 vector of the targets (the house prices).
β, the 2 × 1 parameter vector (β₀, β₁)ᵀ.
Now compute XᵀX. Since Xᵀ is 2 × n and X is n × 2, the product is a 2 × 2 matrix, and working through the dot products of rows and columns gives:
Top-left: the ones vector dotted with itself = n
Top-right and bottom-left: the ones vector dotted with the x column = Σxᵢ
Bottom-right: the x column dotted with itself = Σxᵢ²
These are precisely the coefficients appearing on the left-hand side of the normal equations. Multiplying XᵀX by the vector β reproduces the left-hand sides exactly: the first row gives nβ₀ + β₁Σxᵢ and the second gives β₀Σxᵢ + β₁Σxᵢ².
What about the right-hand side? Compute Xᵀy: the ones row dotted with y yields Σyᵢ, and the x row dotted with y yields Σxᵢyᵢ — again a perfect match. The entire system of normal equations collapses into one compact matrix equation:
XᵀX β = Xᵀy
Solving for β
The final step is simple matrix manipulation. Left-multiply both sides by the inverse of XᵀX:
(XᵀX)⁻¹ XᵀX β = (XᵀX)⁻¹ Xᵀy
By the defining property of the inverse, (XᵀX)⁻¹XᵀX is the identity matrix I, and the identity times any matrix is the matrix itself. Therefore:
β = (XᵀX)⁻¹ Xᵀy
This is the ordinary least squares (OLS) solution. Remember what it represents: it is the point where the gradient of the loss function is zero — the exact bottom of the bowl-shaped SSE surface. To find the optimal intercept and slope, we build the X matrix (a column of ones plus the feature columns), transpose it, multiply it by itself, invert the result, multiply again by the transpose, and finally multiply by y.
Beyond Simple Regression: The General Case
Here is what makes this derivation so powerful: the final equation holds no matter how many features we have. Nothing in the matrix algebra assumed X had only two columns. For the housing example, we could add columns for the number of bedrooms, the number of bathrooms, the lot size, and so on. X becomes an n × (p+1) matrix, β becomes a (p+1)-dimensional vector, and the very same formula
β = (XᵀX)⁻¹ Xᵀy
delivers the optimal coefficients for multiple linear regression in one shot. What began as a two-variable calculus exercise — set two partial derivatives to zero and rearrange — generalizes seamlessly into a universal, closed-form solution for fitting linear models of any dimension.
Summary
The derivation follows a clean logical arc: define the prediction model (a line), choose a loss (sum of squared errors, which ignores error signs and heavily penalizes large misses), minimize with calculus (set the partial derivatives to zero), rearrange into the normal equations, recognize their structure as matrix products, and solve with a single inversion. The result is the ordinary least squares solution — one of the most elegant and practically important formulas in all of data science.