Home / MLP / Week 6

Week 6 · Binary classifiers and how to judge them

Official syllabus line: Classification on scikit-learn - Binary classifier

About 90 minQuiz 2 material (see exam radar)BSCS2008 · Ashish Tendulkar

Before you start: MLP week 5 (logistic regression, predict, predict_proba, decision_function) and week 2 (cross-validation).

Watch first: the animated story of this weekAbout 1-2 minutes of real 3D animation with a friendly voice. Pause any time.

Read the words of the video

Imagine a smoke alarm. It can fail by staying silent during a real fire, which is terrible. Or, it can scream every time you make toast, which is very annoying! A good alarm finds the right balance. Binary classifiers, like those in machine learning, face similar trade-offs in their predictions.
Just like smoke alarms, machine learning models can make two kinds of mistakes. Accuracy alone can be misleading, especially for rare events like diseases. We'll learn to count these mistakes in a confusion matrix, then use precision, recall, and F1 score to truly understand how well a model performs.
Let's work through an example. Out of 500 patients, 110 are truly sick. Our model predicted 120 as sick, and 90 of those were correct. From this, we can calculate our true positives, false positives, false negatives, and true negatives, then precision, recall, and F1 score.
For your exams, always sketch the confusion matrix first! Remember scikit-learn's specific layout. Be careful with precision's (predicted positives) and recall's (actual positives) denominators. Accuracy often misleads on imbalanced data. Raising a threshold usually increases precision but lowers recall. You've learned crucial evaluation metrics this week!

The one-minute story

Think of it like this

A smoke alarm can fail in two opposite ways. It can stay silent in a real fire (terrible), or it can scream every time you make toast (annoying, and soon people ignore it). A good alarm balances both.

A binary classifier has the same two kinds of mistakes, and the number "% correct" hides the difference. Picture a disease that only 2 people in 100 have. A "classifier" that always says "healthy" is 98% correct and catches zero sick people: useless!

This week you learn to count the two kinds of mistakes in a small table (the confusion matrix), turn them into precision, recall and F1, slide the decision threshold to trade one for the other, and summarise the whole trade-off with the ROC curve and its area (AUC).

By the end of this week you can:

build and read a confusion matrix (TP, FP, FN, TN);
compute accuracy, precision, recall, specificity, F1 by hand and with sklearn;
explain why accuracy fails on imbalanced data and use a DummyClassifier baseline;
move the threshold using decision_function, and read the precision-recall and ROC curves;
get honest predictions with cross_val_predict.

Words you need

Positive class - the class we care about finding (disease, fraud, spam). Not "good".
TP / TN - true positive / true negative: correct predictions.
FP (false alarm, "Type I") - predicted positive, actually negative.
FN (miss, "Type II") - predicted negative, actually positive.
Precision - of everything we called positive, how much was truly positive?
Recall (sensitivity, TPR) - of all the truly positive, how many did we catch?
Threshold - the score cut-off for calling something positive.

Idea 1: The confusion matrix

Predictions versus truth, counted in a $2\times2$ table. In scikit-learn the layout is

\begin{pmatrix}TN&FP\\FN&TP\end{pmatrix}\qquad\text{rows = actual (0, 1), columns = predicted (0, 1)}

from sklearn.metrics import confusion_matrix
confusion_matrix(y_true, y_pred)
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()

The idea

\text{accuracy}=\frac{TP+TN}{TP+TN+FP+FN}\qquad\text{precision}=\frac{TP}{TP+FP}\qquad\text{recall}=\frac{TP}{TP+FN}

\text{specificity}=\frac{TN}{TN+FP}\qquad F_1=\frac{2\,P\,R}{P+R}\ \ (\text{harmonic mean of precision and recall})

Think of it like this

Precision asks "when you shout fire, how often is there a fire?" (penalises false alarms). Recall asks "of all the fires, how many did you shout about?" (penalises misses). $F_1$ is high only when both are high.

Solved example: 1. The metrics from a count

Question. A binary classifier is evaluated on 500 test samples. It labels 120 samples as positive; of those, 90 are truly positive. The test set contains 110 actual positives. Find the confusion matrix and all the metrics.

Solution.

$TP=90$ . Predicted positive $=120$ , so $FP=120-90=30$ .
Actual positives $=110$ , so $FN=110-90=20$ .
Actual negatives $=500-110=390$ , so $TN=390-30=360$ .

\begin{pmatrix}TN&FP\\FN&TP\end{pmatrix}=\begin{pmatrix}360&30\\20&90\end{pmatrix}

accuracy $=\frac{90+360}{500}=\mathbf{0.90}$
precision $=\frac{90}{120}=\mathbf{0.75}$
recall $=\frac{90}{110}=\mathbf{0.818}$
$F_1=\frac{2(0.75)(0.818)}{0.75+0.818}=\mathbf{0.783}$
specificity $=\frac{360}{390}=0.923$

(We checked these with confusion_matrix, precision_score, recall_score and f1_score.)

Idea 2: Accuracy can lie

If 98% of the examples are negative, the "classifier" that always predicts negative has 98% accuracy and 0% recall. That is exactly what a DummyClassifier(strategy="most_frequent") does, and it is the baseline any real model must beat:

from sklearn.dummy import DummyClassifier
DummyClassifier(strategy="most_frequent").fit(X_train, y_train).score(X_test, y_test)

On the breast-cancer data (212 malignant out of 569; we take malignant as the positive class), the dummy scores 0.629 accuracy and 0.0 recall: a decent-looking number with no cancer ever found.

So for imbalanced data, do not report accuracy alone. Use precision, recall, $F_1$ , the PR curve, or AUC. Which to favour depends on the cost of each mistake:

Missing a cancer (FN) is far worse than a false alarm: favour high recall.
Marking a good email as spam (FP) is worse than seeing one spam: favour high precision.

Remember the stratify= option of train_test_split to keep the rare class represented in both parts.

Idea 3: A real classifier, step by step

from sklearn.linear_model import SGDClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

clf = make_pipeline(StandardScaler(), SGDClassifier(loss="log_loss", random_state=42))
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
confusion_matrix(y_test, y_pred)

We got (143 test patients, 53 malignant):

[[85  5]       TN=85  FP=5
 [ 7 46]]      FN=7   TP=46
accuracy 0.916   precision 0.902   recall 0.868   F1 0.885   specificity 0.944

Of the 53 truly malignant cases, the model found 46 (recall 86.8%) and missed 7; of the 51 it flagged, 46 were right (precision 90.2%).

Honest predictions on training data: cross_val_predict. It returns, for each training example, the prediction made when that example was in the validation fold (so no peeking), and is the standard way to build a confusion matrix without touching the test set:

from sklearn.model_selection import cross_val_predict
y_cv = cross_val_predict(clf, X_train, y_train, cv=3)
confusion_matrix(y_train, y_cv)         # [[261   6] [ 10 149]]

(cross_val_score returns one score per fold; cross_val_predict returns one prediction per example. Do not confuse them.)

Idea 4: The threshold, and the precision-recall trade-off

predict internally compares the score (decision_function) to a threshold (0 for this model, which is probability 0.5). Move the threshold yourself:

scores = clf.decision_function(X_test)
y_pred_new = (scores > threshold)

Threshold	Precision	Recall
$-2$ (lenient: flag more)	0.891	0.925
$0$ (default)	0.902	0.868
$+2$ (strict: flag fewer)	0.957	0.849

Raising the threshold raises precision and lowers recall; lowering it does the opposite. You can have a lot of one only by sacrificing some of the other.

precision_recall_curve(y, scores) returns precisions, recalls, thresholds. precisions and recalls have one more entry than thresholds. Plot precision against recall to see the whole trade-off.
ROC curve: roc_curve(y, scores) returns false-positive rate $\frac{FP}{FP+TN}$ and true-positive rate (= recall) for every threshold. A useless model hugs the diagonal; a great one hugs the top-left corner. AUC (roc_auc_score) is the area under it: 0.5 = random guessing, 1.0 = perfect. Ours: 0.973.
Use the PR curve rather than ROC when the positive class is rare (ROC can look flattering because the many negatives make the FP rate small).

Idea 5: Other classifiers, same metrics

Everything above works for any binary classifier (SGDClassifier, LogisticRegression, SVC, trees...). classification_report(y, pred) prints precision, recall, $F_1$ and support per class plus averages:

macro average: the plain mean over classes (all classes equal).
weighted average: weighted by class size.

Solved examples

Solved example: 1. Which metric?

Question. For each, which metric matters most? (a) a screening test for a serious but treatable disease; (b) a spam filter where users hate losing real mail; (c) a data set with 99% negatives.

Solution. (a) Recall (do not miss sick people). (b) Precision (do not call real mail spam). (c) Not accuracy: use precision/recall/ $F_1$ or PR-AUC.

Solved example: 2. The 98 / 2 trap

Question. 1000 patients, 20 have the disease. A model predicts everyone as healthy. Give its accuracy, precision and recall.

Solution. Correct on all 980 healthy: accuracy $=\frac{980}{1000}=\mathbf{98\%}$ . It never predicts positive, so $TP=0$ : recall $=\frac0{20}=\mathbf0$ , and precision is undefined ( $\frac00$ ; scikit-learn returns 0 with a warning). A 98% accurate model that finds no one.

Solved example: 3. Effect of the threshold

Question. You lower the decision threshold. What happens to precision, recall, FP and FN?

Solution. More examples get called positive: recall rises (fewer misses, FN falls), FP rises, and precision usually falls.

Your turn

Your turn: 1. From a matrix

confusion_matrix gives [[50, 10], [5, 35]]. Compute precision, recall, accuracy (4 decimal places is fine).

Need a hint?

$TN=50,\,FP=10,\,FN=5,\,TP=35$ .

Show the full solution

Precision $=\frac{35}{45}=0.7778$ . Recall $=\frac{35}{40}=0.875$ . Accuracy $=\frac{85}{100}=0.85$ .

Your turn: 2. Build the matrix

200 samples: 60 positives. The model finds 45 of them and also wrongly flags 20 negatives. Give TP, FP, FN, TN, precision and recall.

Need a hint?

$TP=45$ , so $FN=60-45$ .

Show the full solution

$TP=45$ , $FN=15$ , $FP=20$ , $TN=140-20=120$ . Precision $=\frac{45}{65}=0.692$ . Recall $=\frac{45}{60}=0.75$ .

Your turn: 3. F1

Precision $=0.6$ and recall $=0.9$ . Find $F_1$ . Why is it closer to the smaller one?

Need a hint?

Harmonic mean.

Show the full solution

$F_1=\frac{2(0.6)(0.9)}{1.5}=\mathbf{0.72}$ . The harmonic mean punishes imbalance: it sits nearer the lower of the two, unlike the arithmetic mean $0.75$ .

Your turn: 4. Which AUC?

A model has AUC 0.5. What does that mean? What about 1.0?

Need a hint?

Think of random guessing.

Show the full solution

$0.5$ : no better than random (the ROC curve is the diagonal). $1.0$ : a perfect ranking (all positives scored above all negatives).

Your turn: 5. Pick the tool

You want out-of-fold predictions for every training row to draw a confusion matrix. Which: cross_val_score or cross_val_predict?

Need a hint?

One per fold or one per example?

Show the full solution

cross_val_predict (one prediction per example). cross_val_score gives one score per fold.

Exam radar

MLP Quiz 2 includes: "calculate the recall (up to 3 decimals) from the confusion matrix", "calculate the precision", "a data set with 98% negatives and 2% positives: which evaluation metrics are suitable?", "which of the following are correct for the precision-recall curve?". The End Term has the same: a 500-sample test set with TP and TN given, asking for precision or recall or $F_1$ , plus questions on cross_val_predict, threshold effects and ROC/AUC.

Always sketch the $2\times2$ table first, note scikit-learn's row = actual, column = predicted layout, and compute from it.

Watch out - real exam trap

scikit-learn's layout is [[TN, FP], [FN, TP]]. A hand-drawn table in the question may have a different layout: read the labels.
Precision's denominator is predicted positives ( $TP+FP$ ); recall's is actual positives ( $TP+FN$ ).
Accuracy is misleading on imbalanced data.
Raising the threshold: precision $\uparrow$ (usually), recall $\downarrow$ .
precision_recall_curve thresholds are one shorter than precisions/recalls.
cross_val_predict $\neq$ cross_val_score.

Quick quiz

A model has TP=40, FP=10, FN=20, TN=130. Its recall is...

0.80
0.667
0.50
0.929

Recall = TP / (TP + FN) = 40 / 60 = 0.667. (Precision would be 40/50 = 0.8.)

On a data set with 98 percent negatives, a model that always predicts negative has recall...

98 percent
50 percent
0
undefined

It never finds a positive, so TP = 0 and recall = 0, even though accuracy is 98 percent.

Raising the decision threshold of a classifier usually...

increases precision and decreases recall
increases both
decreases both
has no effect

Fewer examples are called positive: the ones left are more likely to be truly positive, but more positives are missed.

Select all TRUE statements. (select all that apply)

F1 is the harmonic mean of precision and recall.
AUC of 0.5 corresponds to random guessing.
cross_val_predict returns one score per fold.
Accuracy is the best metric for highly imbalanced data.

cross_val_predict returns a prediction per example, and accuracy hides failures on the rare class.

Recap

Remember this

Confusion matrix (sklearn): $\begin{pmatrix}TN&FP\\FN&TP\end{pmatrix}$ . Precision $=\frac{TP}{TP+FP}$ , recall $=\frac{TP}{TP+FN}$ , $F_1=\frac{2PR}{P+R}$ .
Accuracy misleads on imbalanced data; compare with a DummyClassifier.
Threshold up $\Rightarrow$ precision up, recall down. PR curve for rare positives; ROC/AUC (0.5 random, 1 perfect).
cross_val_predict $\to$ out-of-fold predictions; cross_val_score $\to$ per-fold scores.

Formulas to keep

accuracy $=\frac{TP+TN}{n}$ ; precision $=\frac{TP}{TP+FP}$ ; recall (TPR) $=\frac{TP}{TP+FN}$ ; specificity $=\frac{TN}{TN+FP}$ ; FPR $=\frac{FP}{FP+TN}$
$F_1=\frac{2PR}{P+R}$
confusion_matrix(y_true, y_pred) $=[[TN,FP],[FN,TP]]$
precision_recall_curve, roc_curve, roc_auc_score, classification_report

Connected weeks (learn them together)

MLT week 9 - Perceptron and logistic regression: Binary classifiers and their metrics.

Practise with real exam questions

These come straight from past papers (with the real figures and the official answer key). Open 60 real questions on this week All Quiz 2 papers