Linear Regression Bias

May 24, 2024

This post aims to show the impact and importance of the bias term in linear regression models. Linear regression models take the form

for inputs $X_{1}, \ldots, X_{p}$ (i.e. there are $p$ features).

I will be using the Kaggle housing prices dataset as a simple example, despite the collinearity of the data. We’ll start by getting the data

import numpy as np
import pandas as pd

data = pd.read_csv(DATA_PATH)

Some basic preprocessing:

# Transform categorical data
data = pd.get_dummies(data, drop_first=True, dtype=int)
Y = data['price']
X = data.drop('price', axis=1)

# Standardize
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X = pd.DataFrame(scaler.fit_transform(X), columns=X.columns)

And then we add the bias term to $X$

X['bias'] = 1
X, Y = X.to_numpy(), Y.to_numpy()

Now we will compute the coefficients $\beta$ analytically using the Normal Equation

and then use the coefficient vector to calculate $\hat{y}$ and the mean squared error

beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Y)
Yhat = X.dot(beta)
mse = np.mean((Yhat - Y) ** 2)

Now, we will see what happens when we follow the same procedure but without adding the bias term. All other steps remain the same.

And we can see the significant improvement in the mean squared error:

A 21x improvement! The plot comparing the two is even more enlightening.

You can literally see the intercept term at work “lifting” the predictions and why it makes such a simple, yet impressive difference in the performance of the model.