Home / MLP / Week 3

Week 3 · Linear regression and gradient descent in sklearn

The one-minute story

By the end of this week you can:

• fit a line with LinearRegression and read coef_ and intercept_;
• write batch gradient descent from scratch and see it match LinearRegression;
• explain batch, stochastic and mini-batch gradient descent;
• configure SGDRegressor: loss, penalty, alpha, max_iter, tol, learning_rate, eta0, power_t, early_stopping;
• use partial_fit for incremental learning and know which models have it.

Words you need

Idea 1: LinearRegression, the algebra route

from sklearn.linear_model import LinearRegression
import numpy as np

X = np.array([[0], [1], [2.]]); y = np.array([1, 2, 2.])
lr = LinearRegression().fit(X, y)
print(lr.coef_, lr.intercept_)       # [0.5] 1.1666...
print(lr.predict([[3]]))              # [2.6667]

It reproduces the line $\hat y=\frac76+\frac12x$ you derived by hand in MLF week 3. Under the hood it solves $\min\|X\mathbf w-\mathbf y\|^2$ using a singular value decomposition (more stable than literally inverting $X^TX$).

• No hyperparameters to tune (just fit_intercept=True by default).
• Cost grows very fast with the number of features (roughly cubic) but only linearly with the number of examples. With hundreds of thousands of features it becomes slow.
• No feature scaling needed for the answer (scaling just changes the units of the coefficients).
• Predictions are fast.

On a noisy line made from $y=4+3x+\text{noise}$ (100 points) we got intercept_ = 4.222 and coef_ = [2.968]: close to the truth $(4,3)$ but not exactly, because of the noise.

Idea 2: Gradient descent, written out

The cost is the mean squared error. For parameters $\mathbf w$ (intercept included via a column of ones in $X$):

and the update is $\mathbf w\leftarrow\mathbf w-\eta\nabla J$ (MLF week 7). In NumPy:

Xb = np.c_[np.ones(len(X)), X]          # add the column of ones
w = np.zeros(2); eta = 0.1
for _ in range(1000):
    w -= eta * (2 / len(Xb)) * Xb.T @ (Xb @ w - y)

On the noisy-line data, after 1000 steps this gives [4.222, 2.968]: exactly what LinearRegression found. Same destination, different route.

Feature scaling matters for GD. If one feature is 1000 times bigger than another, the cost surface is a long narrow valley and gradient descent zig-zags and crawls. With StandardScaler the valley is round and it goes straight down. (Also: $\eta$ too big diverges, too small crawls.)

Idea 3: SGDRegressor

from sklearn.linear_model import SGDRegressor
sgd = SGDRegressor(max_iter=1000, tol=1e-5, penalty=None, eta0=0.1, random_state=42)
sgd.fit(X, y.ravel())
print(sgd.intercept_, sgd.coef_, sgd.n_iter_)

On our noisy line this gave intercept_ ≈ 4.31, coef_ ≈ 3.05 after n_iter_ = 17 epochs: near the exact answer, found by 17 passes of one-example-at-a-time updates.

The dials and their defaults in recent scikit-learn (your course version may differ slightly, so check the docs if a question gives a specific one):

The four learning-rate schedules (learning_rate=):

How to write them in code (a favourite question):

SGDRegressor(learning_rate="invscaling", eta0=0.01, power_t=0.25)
SGDRegressor(learning_rate="adaptive",   eta0=0.01)
SGDRegressor(learning_rate="constant",   eta0=0.05)
SGDClassifier(learning_rate="optimal")      # default for the classifier

(The argument is learning_rate, a string, plus eta0 for the number.)

When does it stop? SGD stops when the loss has failed to improve by more than tol for n_iter_no_change epochs in a row (this is the meaning of the tolerance), or when max_iter epochs have run, whichever is first. n_iter_ records how many epochs it actually used. A bigger tol stops earlier (less accurate); a smaller tol runs longer.

Scale first. SGD is sensitive to feature scales. Put a StandardScaler in front, in a pipeline.

Idea 4: Learning in instalments: partial_fit

If the data does not fit in memory (or arrives as a stream), feed it in chunks. partial_fit performs one pass over the batch you give it and keeps the learned weights, so you can call it again and again:

sg = SGDRegressor(random_state=0)
for X_chunk, y_chunk in data_in_batches():     # your own generator
    sg.partial_fit(X_chunk, y_chunk)

(Do not use fit in a loop: each fit forgets what came before.)

Estimators that offer partial_fit ("incremental learners"):

• Yes: SGDRegressor, SGDClassifier, Perceptron, PassiveAggressive*, MultinomialNB, BernoulliNB, GaussianNB, MiniBatchKMeans, MLPClassifier, MLPRegressor, IncrementalPCA.
• No: LinearRegression, LogisticRegression, Ridge, Lasso, SVC, decision trees, random forests, KNeighbors*.

(For SGDClassifier.partial_fit you must pass classes= on the first call.) Using the LinearRegression class is not possible for out-of-core learning; SGDRegressor is the standard answer.

Solved examples

Your turn

Exam radar

Quick quiz

Recap

Practise with real exam questions

These come straight from past papers (with the real figures and the official answer key). Open 60 real questions on this week   All Quiz 1 papers