Home / MLP / Week 4

Week 4 · Polynomial regression and regularisation

The one-minute story

By the end of this week you can:

• create polynomial features with PolynomialFeatures and count them;
• diagnose underfitting and overfitting from train and test error;
• fit Ridge, Lasso, ElasticNet and say how coefficients behave as alpha grows;
• pick the regularisation strength with RidgeCV or GridSearchCV;
• say, for each model, which direction of its knob means more regularisation.

Words you need

Idea 1: PolynomialFeatures

from sklearn.preprocessing import PolynomialFeatures
PolynomialFeatures(degree=2).fit_transform([[2, 3]])

For the single row $(x_1,x_2)=(2,3)$ the output is

degree 2 : [1, 2, 3, 4, 6, 9]          -> 1, x1, x2, x1^2, x1*x2, x2^2   (6 columns)
degree 3 : [1, 2, 3, 4, 6, 9, 8, 12, 18, 27]   (10 columns)
degree 2, include_bias=False   : [2, 3, 4, 6, 9]     (drops the column of 1s)
degree 2, interaction_only=True, include_bias=False : [2, 3, 6]   (no pure powers)

How many output columns for $n$ input features and degree $d$ (with the bias column)? $\binom{n+d}{d}$. For $n=2,d=2$: $\binom42=6$ ✓. For $n=3,d=2$: $\binom52=\mathbf{10}$ ✓. The count explodes quickly as $n$ and $d$ grow, which is why high degrees overfit and cost a lot.

Rules of thumb:

• The default is include_bias=True: it adds a column of 1s. LinearRegression already has its own intercept, so use include_bias=False there to avoid redundancy.
• Scale after expanding: $x^{10}$ can be astronomically bigger than $x$.
• Use it inside a pipeline: make_pipeline(PolynomialFeatures(2, include_bias=False), StandardScaler(), LinearRegression()).

Idea 2: Underfitting and overfitting, with numbers

We made 30 noisy points around the curve $y=0.5x^2+x+2$ (the true shape is a parabola), split 70/30, and fitted polynomials of degree 1, 2 and 12:

rng = np.random.RandomState(1)
xs = np.sort(6 * rng.rand(30, 1) - 3, axis=0)
ys = 0.5 * xs[:, 0] ** 2 + xs[:, 0] + 2 + rng.randn(30)
x_tr, x_te, y_tr, y_te = train_test_split(xs, ys, test_size=0.3, random_state=0)

for d in (1, 2, 12):
    m = make_pipeline(PolynomialFeatures(d, include_bias=False),
                      StandardScaler(), LinearRegression()).fit(x_tr, y_tr)
    print(d, mean_squared_error(y_tr, m.predict(x_tr)),
             mean_squared_error(y_te, m.predict(x_te)))

Reading learning curves. Plot train and validation error against the number of training examples:

• High bias (underfit): both curves are high and close together; more data will not help, you need a richer model.
• High variance (overfit): a large gap: train low, validation high; more data, or a simpler/regularised model, helps.

from sklearn.model_selection import learning_curve
sizes, train_scores, val_scores = learning_curve(model, X, y, cv=5,
        scoring="neg_mean_squared_error", train_sizes=np.linspace(0.1, 1, 5))

Idea 3: Ridge, Lasso and Elastic Net

from sklearn.linear_model import Ridge, Lasso, ElasticNet
Ridge(alpha=1.0)                       # default alpha = 1.0
Lasso(alpha=1.0)
ElasticNet(alpha=1.0, l1_ratio=0.5)    # l1_ratio=1 is pure Lasso, 0 is pure Ridge

• Ridge ($\ell_2$): minimises $\|\mathbf y-X\mathbf w\|^2+\alpha\|\mathbf w\|_2^2$. It shrinks all weights smoothly and keeps them non-zero. It has a closed-form solution $(X^TX+\alpha I)^{-1}X^T\mathbf y$ (we checked: Ridge(alpha=1, fit_intercept=False) matches the formula).
• Lasso ($\ell_1$): minimises $\frac1{2n}\|\mathbf y-X\mathbf w\|^2+\alpha\|\mathbf w\|_1$ (note scikit-learn's $\frac1{2n}$ scaling). It can set weights exactly to zero: automatic feature selection.
• Elastic Net: a mix controlled by l1_ratio.
• Scale your features first: the penalty treats all weights alike.
• The intercept is not penalised.

A real run on a synthetic data set (60 samples, 8 features, only 3 informative, standardised):

LinearRegression  [ 65.6  -1.2  92.1  -1.4   1.0  91.6   0.6   2.6]
Ridge alpha=1     [ 64.8  -1.0  90.5  -1.8   0.9  89.9   0.8   2.9]
Ridge alpha=100   [ 27.1   0.8  34.3  -5.9   0.5  33.8   2.3   4.7]
Lasso alpha=1     [ 64.6  -0.6  91.4  -0.4   0.2  90.8   0.0   1.6]    (1 weight is exactly 0)
Lasso alpha=10    [ 56.3  -0.0  82.7  -0.0   0.0  82.3   0.0   0.0]    (5 weights are exactly 0)

The three big weights are the informative features. As alpha rises, ridge shrinks every weight, including the big ones (27 vs 65 at alpha 100), but none hits zero; Lasso keeps the big ones and zeroes the junk (five zeros at alpha 10).

Which knob tightens the leash? (the exam asks: "which model has the lowest overfitting, i.e. the largest regularisation?")

Idea 4: Choosing alpha

from sklearn.linear_model import RidgeCV
RidgeCV(alphas=[0.01, 0.1, 1, 10, 100], cv=5).fit(X, y)       # picks the best alpha by CV
from sklearn.model_selection import GridSearchCV
GridSearchCV(Ridge(), {"alpha": [0.01, 0.1, 1, 10]}, cv=5).fit(X, y)
GridSearchCV(pipe, {"model__alpha": [0.01, 0.1, 1, 10]}, cv=5)       # inside a pipeline

All three are correct methods to search for the best regularisation parameter; the grid key inside a pipeline needs the step name with a double underscore. There are also LassoCV and ElasticNetCV. Regularisation can also come from early stopping: SGDRegressor(early_stopping=True) holds out validation_fraction=0.1 of the data and stops when the validation score stops improving.

Solved examples

Your turn

Exam radar

Quick quiz

Recap

Practise with real exam questions

These come straight from past papers (with the real figures and the official answer key). Open 60 real questions on this week   All Quiz 1 papers