Home / MLP / Week 11

Week 11 · Neural networks in sklearn

The one-minute story

By the end of this week you can:

• train MLPClassifier / MLPRegressor and read coefs_, intercepts_, n_layers_, n_iter_, loss_curve_;
• set hidden_layer_sizes, activation, solver, alpha, max_iter, early_stopping;
• count parameters and know the output activation for each task;
• explain what happens with no non-linearity and with too few hidden units.

Words you need

Idea 1: The API

from sklearn.neural_network import MLPClassifier, MLPRegressor
mlp = MLPClassifier(hidden_layer_sizes=(100,), activation="relu", solver="adam",
                    alpha=0.0001, batch_size="auto", learning_rate_init=0.001,
                    max_iter=200, early_stopping=False, random_state=42)

Those are (almost) the defaults. The ones to remember:

You do not specify the input or output layer sizes: they are read from the data (number of features, number of classes). The output activation is chosen for you and stored in out_activation_:

Idea 2: A real run on digits

The 1797 digit images (64 pixels each), standardised, 75/25 split, with the default network (one hidden layer of 100 ReLU neurons):

mlp = make_pipeline(StandardScaler(), MLPClassifier(random_state=42, max_iter=300))
mlp.fit(X_train, y_train)

test accuracy 0.980     train accuracy 1.000
n_iter_ = 147    n_layers_ = 3    n_outputs_ = 10    out_activation_ = 'softmax'

(n_layers_ counts input + hidden + output layers: $1+1+1=3$.) Inspect the learned numbers:

[c.shape for c in mlp[-1].coefs_]        # [(64, 100), (100, 10)]   one weight matrix per layer
[b.shape for b in mlp[-1].intercepts_]   # [(100,), (10,)]          one bias vector per layer

Counting parameters (it is the sum over layers of $m\times n+n$):

For hidden_layer_sizes=(50, 50) the shapes become [(64,50), (50,50), (50,10)] and the count is $(3200+50)+(2500+50)+(500+10)=\mathbf{6310}$, with n_layers_=4.

loss_curve_ has one entry per epoch (147 here) and falls from about 2.52 to 0.0055: plot it to watch training. predict_proba(X) returns an $(m,10)$ array whose rows sum to 1 (softmax).

Idea 3: The dials, with evidence

Lessons:

• Capacity: too few hidden units underfit; many units can overfit. Tune the architecture with cross-validation.
• alpha is the regulariser: larger = simpler weights = less overfitting. (And early_stopping=True is another regulariser.)
• With activation="identity" the whole network is a single linear map, no matter how many layers: this is what "introducing non-linearity" fixes. To make a network non-linear, use a non-linear activation ('relu', 'tanh', 'logistic').
• Scale your inputs. Networks train much better when inputs are standardised or in $[0,1]$. (The digit pixels are all between 0 and 16 so the unscaled run still reached 0.978; with features on very different scales the gap is large.)
• Results change between runs because the start is random (random_state). For reproducible results set it.

Idea 4: Other points

• Binary classification gives n_outputs_=1, out_activation_='logistic', and the last weight matrix has shape (n_hidden, 1).
• MLPRegressor has an identity output; scale the target too when it is large.
• partial_fit is available (on adam and sgd): for classification pass classes= the first time. Training in chunks like this is not the same as a full-epoch fit; our 3-pass demo only reached 0.53.
• Hyperparameters to tune: hidden_layer_sizes, alpha, learning_rate_init, activation. Use GridSearchCV with the pipeline name: {"mlpclassifier__hidden_layer_sizes": [(50,), (100,), (50, 50)], "mlpclassifier__alpha": [1e-4, 1e-2]}.
• scikit-learn's MLP is fine for small and medium data. For big images or text use a deep-learning library (PyTorch, TensorFlow/Keras) with GPUs.
• Neural nets are non-convex: different starts can land in different minima.

Solved examples

Your turn

Exam radar

Quick quiz

Recap

Practise with real exam questions

These come straight from past papers (with the real figures and the official answer key). Open 14 real questions on this week   All End Term papers