Home / MLP / Week 2

Week 2 · The project continued: pipelines and tuning

The one-minute story

By the end of this week you can:

• build Pipelines and ColumnTransformers and set their parameters with double underscores;
• explain why a decision tree overfits and show it with train vs cross-validated error;
• use cross-validation (cross_val_score, KFold, LeaveOneOut) and handle negative scoring;
• count the models a GridSearchCV or RandomizedSearchCV will train;
• evaluate the final model on the test set once.

Words you need

Idea 1: Pipelines

from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression

pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale",  StandardScaler()),
    ("model",  LinearRegression()),
])
pipe.fit(X_train, y_train)          # fit_transform through the first two, fit on the last
pipe.predict(X_test)                 # transform through the first two, predict with the last
pipe.score(X_test, y_test)

Rules the exam likes:

• All steps but the last must be transformers (they need fit and transform). The last step can be anything: another transformer or a predictor. (So "the last step must be a transformer" is false.)
• A Pipeline is itself an estimator: you can fit, predict, put it in a grid search, or nest it in another pipeline (composition).
• make_pipeline(StandardScaler(), LinearRegression()) is the shortcut: it names the steps for you in lowercase class names: ['standardscaler', 'linearregression'].
• Access a step: pipe.named_steps["model"].
• Set or search a nested hyperparameter with double underscores: "step_name__param_name", e.g. model__fit_intercept, impute__strategy.

pipe.set_params(model__fit_intercept=False)

Why pipelines? (1) No data leakage: inside cross-validation the imputer and scaler are re-fitted on each training fold only. (2) One object to fit, predict, save and tune. (3) Less code, fewer mistakes.

Idea 2: ColumnTransformer

Numbers and categories need different treatment:

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder

num_pipe = Pipeline([("imp", SimpleImputer(strategy="median")),
                     ("sc",  StandardScaler())])

prep = ColumnTransformer([
    ("num", num_pipe, ["area", "rooms"]),                     # name, transformer, columns
    ("cat", OneHotEncoder(handle_unknown="ignore"), ["city"]),
])
full = Pipeline([("prep", prep), ("model", LinearRegression())])

On our 8-house table (6 training rows) the transformed matrix is (6, 5): two scaled numeric columns plus three one-hot city columns, named by prep.get_feature_names_out():

['num__area', 'num__rooms', 'cat__city_A', 'cat__city_B', 'cat__city_C']

• Columns you do not list are dropped by default (remainder="drop"). Use remainder="passthrough" to keep them untouched. (With only area listed and the default, the output has 1 column; with passthrough it keeps the other two as well: 3 columns.)
• make_column_transformer(...) is the name-free shortcut; make_column_selector(dtype_include=np.number) picks columns by type.
• If any piece returns a sparse matrix (one-hot), ColumnTransformer decides whether the result is sparse by density.

Idea 3: Train a model, and watch it overfit

On the diabetes data set that ships with scikit-learn (442 patients, 10 features, target = disease progression), with an 80/20 split and random_state=42:

from sklearn.metrics import root_mean_squared_error
from sklearn.model_selection import cross_val_score

for model in [LinearRegression(), DecisionTreeRegressor(random_state=0)]:
    model.fit(X_train, y_train)
    rmse_train = root_mean_squared_error(y_train, model.predict(X_train))
    cv_rmse = np.sqrt(-cross_val_score(model, X_train, y_train, cv=5,
                                       scoring="neg_mean_squared_error"))

The numbers we got (your last digits could differ with other library versions):

Lesson: training error alone is a lie. Always judge a model by cross-validation (or a held-out set).

Note on API versions: older tutorials write mean_squared_error(y, pred, squared=False) for RMSE. That argument was removed in newer scikit-learn; use root_mean_squared_error (or take the np.sqrt yourself). If a question shows the old form, treat it as the RMSE idea.

Idea 4: Cross-validation

scores = cross_val_score(model, X_train, y_train, cv=5)     # array of 5 numbers

• The default score is the estimator's own score: $R^2$ for regressors, accuracy for classifiers. For linear regression on diabetes, cv=5 gave $R^2$ values [0.43 0.523 0.483 0.426 0.55], mean $0.482$.
• scikit-learn maximises scores, so error metrics are returned negated: scoring="neg_mean_squared_error" gives negative numbers. To get RMSE: np.sqrt(-scores).
• cv=5 with a classifier automatically uses stratified folds (class ratios preserved).
• KFold(n_splits=4) on 10 points gives validation sizes 3, 3, 2, 2 (the first folds take the extra points).
• LeaveOneOut(): $n$ folds, each holding out one example: almost unbiased but costs $n$ fits. Use it as cross_val_score(model, X, y, cv=LeaveOneOut()).
• cross_val_predict returns, for each example, the prediction made when it was in the validation fold (useful for confusion matrices in week 6). ShuffleSplit takes random subsamples that may overlap.

Idea 5: Fine-tuning hyperparameters

A hyperparameter is chosen by you; a parameter is learned. Instead of guessing, search with cross-validation.

from sklearn.model_selection import GridSearchCV
param_grid = {"max_depth": [2, 3, 4, 5], "min_samples_leaf": [1, 5, 10]}
grid = GridSearchCV(DecisionTreeRegressor(random_state=0), param_grid,
                    cv=5, scoring="neg_mean_squared_error")
grid.fit(X_train, y_train)
grid.best_params_, grid.best_score_, grid.best_estimator_

Counting the work (a favourite question):

Here $4\times3=12$ combinations, $\times\,5$ folds $=60$ fits, plus 1 refit of the best model on all training data (refit=True is the default). For the grid {"n_estimators": [100,200,300], "max_depth": [None,5,10,20], "max_features": ["sqrt","log2"]} there are $3\times4\times2=24$ combinations, so with cv=5 that is 120 fits (121 with the refit). If the grid is a list of dictionaries, the combination counts add: [{"a":[1,2,3],"b":[4,5]}, {"c":[6,7]}] gives $6+2=8$.

RandomizedSearchCV samples n_iter random combinations (from lists or from probability distributions) instead of trying all of them:

from sklearn.model_selection import RandomizedSearchCV
rs = RandomizedSearchCV(est, {"max_depth": range(1, 11), "min_samples_leaf": range(1, 21)},
                        n_iter=7, cv=3, random_state=0)

That tests only $7$ of the $10\times20=200$ possible combinations: $7\times3=21$ fits. Its main advantage: the cost is fixed by you (n_iter) however big the search space, it handles continuous ranges, and in practice it finds near-best settings much faster than a full grid.

Useful attributes: best_params_, best_score_ (a mean CV score, negated if you used a neg_ metric), best_estimator_, cv_results_.

Idea 6: The final evaluation, once

After tuning, evaluate the chosen model on the test set once:

final_model = grid.best_estimator_
rmse_test = root_mean_squared_error(y_test, final_model.predict(X_test))

Expect it to land close to the CV estimate (often slightly worse, because you picked the winner using the training data). Do not go back and tweak after seeing the test score, or the test set stops being honest. A 95% confidence interval for the generalisation error can be built from the per-example squared errors with a $t$ distribution (scipy.stats.t.interval). Save the model with joblib.dump(final_model, "model.pkl") and reload with joblib.load.

Custom transformers. To make your own, write a class with fit (returns self) and transform, and inherit BaseEstimator and TransformerMixin (you then get get_params/set_params and fit_transform for free). FunctionTransformer(np.log1p) wraps a plain function.

Solved examples

Your turn

Exam radar

Quick quiz

Recap

Practise with real exam questions

These come straight from past papers (with the real figures and the official answer key). Open 40 real questions on this week   All Quiz 1 papers