Home / MLP / Week 10

Week 10 · Random forests, boosting and tuning

The one-minute story

By the end of this week you can:

• train AdaBoostClassifier with decision stumps and read its weights;
• explain gradient boosting, learning_rate and n_estimators, and stop early;
• build a StackingClassifier;
• set up and count a grid search over an ensemble using estimator__... names;
• pick between bagging, forests and boosting.

Words you need

Idea 1: AdaBoost in scikit-learn

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier

ada = AdaBoostClassifier(estimator=DecisionTreeClassifier(max_depth=1),
                         n_estimators=50, learning_rate=0.5, random_state=42)
ada.fit(X_train, y_train)

(In older scikit-learn the keyword is base_estimator=; from version 1.2 it is estimator=. The default base learner is a depth-1 tree, a stump.) Defaults: n_estimators=50, learning_rate=1.0.

On the noisy "two moons" data (500 points, 70/30 split):

One stump is a straight cut; fifty stumps combine into a curved boundary. Notice the last row: past a point, more rounds stop helping and can hurt: boosting can overfit noisy data (the hardest, noisiest points keep getting heavier).

How the vote is weighted (MLT week 11). Each learner gets a weight based on its weighted error $\varepsilon_t$; sklearn stores them in ada.estimator_weights_ and the errors in ada.estimator_errors_. For our first learner the error was $0.18$ and the stored weight $1.516=\ln\frac{0.82}{0.18}$.

Requirements for the base learner: it must support sample weights (trees do). The key hyperparameters: n_estimators, learning_rate (a smaller rate shrinks each learner's contribution; trade it against more estimators), and the base estimator's own settings (max_depth...), which you reach as estimator__max_depth.

Idea 2: Gradient boosting

For regression with squared error the idea is beautifully simple:

1. Fit a small tree to the targets.
2. Compute the residuals (target minus prediction).
3. Fit the next small tree to those residuals.
4. Add it to the ensemble, scaled by learning_rate. Repeat.

The final prediction is the sum of all the trees' (shrunken) outputs.

from sklearn.ensemble import GradientBoostingClassifier   # or ...Regressor
gb = GradientBoostingClassifier(n_estimators=300, learning_rate=0.1, max_depth=2,
                                random_state=42)

Defaults: n_estimators=100, learning_rate=0.1, max_depth=3, subsample=1.0 (set below 1 for stochastic gradient boosting: each tree trains on a random fraction of the data, adding regularisation and speed).

• Smaller learning_rate $\Rightarrow$ need more n_estimators (a regularising trade-off; a small rate with many trees usually generalises best).
• Too many trees overfit. Choose the number by early stopping:

gb = GradientBoostingClassifier(n_estimators=300, learning_rate=0.1, max_depth=2,
                                n_iter_no_change=10, validation_fraction=0.2)
gb.fit(X_train, y_train)
gb.n_estimators_            # how many trees it actually kept (we got 104)

or by looking at the test error per number of trees with staged_predict:

errors = [1 - (p == y_test).mean() for p in gb.staged_predict(X_test)]
best_n = int(np.argmin(errors)) + 1

On our moons run the test error bottomed out at 27 trees (8.7%) and had crept up to 9.3% by 300 trees: a small but real overfitting.

Also available: HistGradientBoostingClassifier/Regressor (fast, handles missing values, built for big data), and the external libraries XGBoost, LightGBM and CatBoost.

Idea 3: Stacking

Instead of voting by a fixed rule, learn how to combine. The base models make predictions; a final estimator (the "blender", often a logistic regression) is trained on those predictions:

from sklearn.ensemble import StackingClassifier
stack = StackingClassifier(
    estimators=[("rf", RandomForestClassifier(random_state=42)),
                ("svc", SVC(probability=True, random_state=42))],
    final_estimator=LogisticRegression(), cv=5)

To avoid the blender learning from predictions on data the base models had already memorised, scikit-learn trains it on out-of-fold predictions (that is what cv=5 does). On moons the stack scored 0.913. Stacking is often the most accurate (and slowest and hardest to explain).

Idea 4: Tuning an ensemble and counting the work

All the nested pieces use the double-underscore rule from week 2. For AdaBoost with a tree base learner:

from sklearn.model_selection import GridSearchCV
params = {"estimator__max_depth": [1, 2],
          "n_estimators": [30, 35, 40],
          "learning_rate": [0.1, 0.5, 1.0]}
GridSearchCV(AdaBoostClassifier(estimator=DecisionTreeClassifier()), params, cv=5)

Number of combinations: $2\times3\times3=18$. With cv=5: $18\times5=\mathbf{90}$ fits (+1 for the final refit $=91$).

A real exam question describes the specification in words ("base estimator decision tree classifier with max_depth=1; number of estimators from 30 to 40 in steps of 5; learning rate in ...") and asks how many models are trained or which code does the right search. Convert each phrase to a list, multiply the list lengths, then multiply by the number of CV folds.

Idea 5: Which ensemble when?

Advantages of ensembles (a standard question): better accuracy than single models, less overfitting/variance, robustness. Drawbacks: slower to train and predict, less interpretable, more memory.

Solved examples

Your turn

Exam radar

Quick quiz

Recap

Practise with real exam questions

These come straight from past papers (with the real figures and the official answer key). Open 6 real questions on this week   All End Term papers