Home / MLP / Week 5

Week 5 · Logistic regression in sklearn

Official syllabus line: Logistic regression

About 80 minQuiz 2 material (see exam radar)BSCS2008 · Ashish Tendulkar

Before you start: MLP weeks 3-4 (training and regularising a model in sklearn). MLT week 9 and MLF week 9 for the maths of logistic regression.

Watch first: the animated story of this weekAbout 1-2 minutes of real 3D animation with a friendly voice. Pause any time.

Read the words of the video

Imagine trying to guess if a flower is red or blue. Logistic regression helps by giving you a number between 0 and 1. A number like 0.9 means "almost certainly red", while 0.1 means "almost certainly blue". It tells you how likely something is to belong to a specific group.
First, it calculates a simple "score" for the flower, like summing up points for its petal length and width. Then, it uses a special S-shaped curve, called a sigmoid. This curve transforms the score into a probability, so big scores become high chances, and small scores become low chances.
Let's try an example. If a flower has petal length 4 and width 2, we first calculate its score. Using our model's recipe, this score turns out to be 0.6. Then, using our S-shaped curve, this 0.6 score becomes a probability of about 0.65. So, it's about 65% likely to be class 1.
Remember to know your methods: predict_proba gives you percentages, and decision_function gives the raw score. Also, lowering 'C' makes your model simpler and less likely to overfit. Keep practicing how scores become probabilities. You've got this, great job learning logistic regression!

The one-minute story

Think of it like this

Despite the name, logistic regression is a classifier. It answers "how likely is it that this flower is class 1?" with a number between 0 and 1.

It works in two small steps. First it computes a score for the example, exactly as a linear model would: a weighted sum of the features plus a bias. Then it squeezes that score through an S-shaped curve (the sigmoid) so that big positive scores turn into "almost certainly yes", big negative scores into "almost certainly no", and a score of exactly 0 into "50/50".

Everything else in this week (probabilities, thresholds, C, three classes) is just how scikit-learn lets you read and tune those two steps.

By the end of this week you can:

train LogisticRegression and read coef_, intercept_, classes_;
tell predict, predict_proba and decision_function apart and convert between them;
explain the role of C and the default settings;
handle more than two classes with softmax regression;
use SGDClassifier(loss="log_loss") for the incremental version.

Words you need

Class / label - the category being predicted.
Score / logit / decision function - the raw number $z=\mathbf w^T\mathbf x+b$ .
Probability - $\sigma(z)$ , between 0 and 1.
Decision threshold - the probability above which we say "class 1" (default 0.5).
C - inverse regularisation strength (small C = strong regularisation).
Softmax regression - the multi-class version of logistic regression.
Log loss / cross-entropy - the training loss.

Idea 1: From score to probability

For one example $\mathbf x$ :

z=\mathbf w^T\mathbf x+b,\qquad p=P(y=1\mid\mathbf x)=\sigma(z)=\frac1{1+e^{-z}}

$z=0\iff p=0.5$ . The decision boundary is the straight line (plane) $z=0$ .
$z>0\iff p>0.5\iff$ predict class 1.
Training minimises the log loss $-\frac1n\sum[y\ln p+(1-y)\ln(1-p)]$ (plus a penalty), which is convex; gradient descent (the lbfgs solver by default) finds the best weights.

Idea 2: Training and reading the model

We take the iris flower data, keep only two species (setosa $=0$ and versicolor $=1$ ) and two features (petal length and width):

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression

X, y = load_iris(return_X_y=True)
mask = y < 2
X2, y2 = X[mask][:, 2:], y[mask]            # petal length, petal width
clf = LogisticRegression().fit(X2, y2)

clf.coef_, clf.intercept_, clf.classes_

coef_       [[2.583 1.069]]      # one row of weights: shape (1, 2) for a binary problem
intercept_  [-7.808]
classes_    [0 1]

So $z=2.583\,x_{\text{len}}+1.069\,x_{\text{wid}}-7.808$ . The boundary is $z=0$ : with petal width 1, that is petal length $\approx2.61$ .

Three methods give three views of the same thing, for the example $\mathbf x=(3,1)$ :

x = [[3.0, 1.0]]
clf.decision_function(x)   # [1.009]            the score z
clf.predict_proba(x)       # [[0.2672 0.7328]]  P(class 0), P(class 1)
clf.predict(x)             # [1]                the class (threshold 0.5)

Check by hand: $z=2.583\cdot3+1.069\cdot1-7.808=1.009$ ✓, and $\sigma(1.009)=\frac1{1+e^{-1.009}}=0.7328$ ✓ (the second column of predict_proba).

Method	Returns	Shape for $m$ examples, 2 classes
`decision_function(X)`	the score $z$ (a real number)	$(m,)$
`predict_proba(X)`	one probability per class; each row sums to 1	$(m,2)$
`predict(X)`	the predicted class label	$(m,)$
`score(X, y)`	accuracy	a single number

On this easy data the accuracy is 1.0 and the average log loss is $0.026$ .

A different threshold? predict always uses 0.5. To use another cut-off, take the probabilities yourself: (clf.predict_proba(X)[:, 1] > 0.3).astype(int). This matters for precision/recall in week 6.

Idea 3: The knobs: `C`, `penalty`, `solver`

LogisticRegression(penalty="l2", C=1.0, solver="lbfgs", max_iter=100)   # classic defaults

C (default 1.0) is the inverse of the regularisation strength. Small C = strong regularisation (weights shrink, simpler boundary); large C = weak regularisation. On the iris data, the total size of the weights $\sum|w_j|$ is 0.62 for C=0.01 and 8.85 for C=100.
penalty: 'l2' by default (shrinks all weights), 'l1' (sparse weights; needs a solver such as liblinear or saga). In the newest scikit-learn versions this argument is being replaced by l1_ratio; the idea is unchanged.
solver: lbfgs (default), liblinear, newton-cg, sag, saga. Different solvers support different penalties and sizes of data.
max_iter (default 100): raise it if you see a ConvergenceWarning (and scale your features, which helps convergence a lot).
If the classes are perfectly separable, without regularisation the weights would grow without limit; the default C=1 prevents that.

Idea 4: More than two classes

For $K$ classes the model has one weight vector per class and uses softmax (MLT week 12):

P(y=k\mid\mathbf x)=\frac{e^{z_k}}{\sum_je^{z_j}},\qquad z_k=\mathbf w_k^T\mathbf x+b_k

On all three iris species with the two petal features:

softmax_reg = LogisticRegression(C=30, max_iter=1000).fit(X[:, 2:], y)
softmax_reg.coef_.shape                         # (3, 2): one row per class
softmax_reg.predict_proba([[5.0, 2.0]])         # [[0.    0.029 0.971]]
softmax_reg.predict([[5.0, 2.0]])               # [2]    (virginica)

The three probabilities add to 1, and the prediction is the class with the largest one. (Older scikit-learn versions used multi_class="multinomial" for this; recent ones do it automatically with lbfgs.) coef_ has shape (n_classes, n_features) here but (1, n_features) in the binary case.

Idea 5: The incremental version

from sklearn.linear_model import SGDClassifier
SGDClassifier(loss="log_loss")          # logistic regression trained by SGD; has partial_fit

The default loss of SGDClassifier is 'hinge', which gives a linear SVM, not logistic regression. To get logistic regression you must set loss="log_loss". With the SGD version you also get partial_fit for data that arrives in chunks (week 3).

Solved examples

Solved example: 1. Turning a score into a probability

Question. A trained model has coef_ = [[1.5, -2.0]] and intercept_ = [0.5]. For $\mathbf x=(2,1)$ find decision_function, predict_proba and predict.

Solution.

$z=1.5\cdot2-2.0\cdot1+0.5=\mathbf{1.5}$ .
$p_1=\sigma(1.5)=\frac1{1+e^{-1.5}}=\frac1{1+0.223}=\mathbf{0.818}$ , so $p_0=0.182$ . predict_proba $\approx[[0.182,\,0.818]]$ (columns in the order of classes_).
$p_1>0.5$ , so predict $=1$ .

Solved example: 2. Shapes

Question. A 4-class problem has 6 features. What is the shape of coef_, intercept_, and of predict_proba for 10 test examples?

Solution. coef_: $(4,6)$ . intercept_: $(4,)$ . predict_proba(X_test): $(10,4)$ , and each row sums to 1.

Solved example: 3. Effect of C

Question. You raise C from 0.01 to 100. What happens to the training accuracy, the size of the weights and the risk of overfitting?

Solution. Weaker regularisation: the weights grow, the training accuracy usually rises (the boundary fits the training data more tightly), and the overfitting risk goes up.

Your turn

Your turn: 1. Probability by hand

coef_=[[0.8, 0.2]], intercept_=[-3]. Give the class-1 probability for $\mathbf x=(4,2)$ .

Need a hint?

Compute $z$ first.

Show the full solution

$z=0.8\cdot4+0.2\cdot2-3=3.2+0.4-3=0.6$ . $p=\sigma(0.6)=\frac1{1+e^{-0.6}}=\frac1{1+0.549}=\mathbf{0.646}$ .

Your turn: 2. Which method?

Which call returns an array of probabilities that sums to 1 in each row: predict, predict_proba, decision_function, score?

Need a hint?

Read the table.

Show the full solution

predict_proba.

Your turn: 3. Threshold

predict_proba(X)[:, 1] returns [0.1, 0.45, 0.55, 0.9]. What does predict return, and what class labels do you get if you use a threshold of 0.4 instead?

Need a hint?

Default threshold 0.5.

Show the full solution

predict: [0, 0, 1, 1]. With threshold 0.4: [0, 1, 1, 1] (0.45 now counts as class 1).

Your turn: 4. Make it logistic

A colleague writes SGDClassifier().fit(X, y) and calls it logistic regression. Is it? Fix it.

Need a hint?

Check the default loss.

Show the full solution

No: the default loss is 'hinge', a linear SVM. Use SGDClassifier(loss="log_loss").

Your turn: 5. Regularisation

For LogisticRegression, C=0.1 versus C=10: which is more regularised, and which is more likely to overfit?

Need a hint?

C is the inverse of the strength.

Show the full solution

C=0.1 is more regularised; C=10 is more likely to overfit.

Exam radar

MLP: logistic regression is the topic of week 5 and appears in classification questions throughout; predict_proba/decision_function and the effect of C are standard. Look for: "What is the shape of coef_ for a 3-class problem?", "Which method returns class probabilities?", "What does decreasing C do?", "What is the default loss of SGDClassifier and which model is that?", and probability-from-weights calculations.

From MLT, logistic regression shows up in 23 End Term papers, and the sigmoid in 10.

Watch out - real exam trap

predict_proba returns one column per class, in the order of classes_.
C is the inverse of regularisation strength.
SGDClassifier's default is hinge (SVM), not logistic.
score is accuracy, not likelihood.
Binary coef_ has shape (1, n_features).
Scale the features and raise max_iter if there is a convergence warning.

Quick quiz

With the default settings, LogisticRegression predicts class 1 when...

the score is negative
the probability of class 1 exceeds 0.5
the probability exceeds 0.9
the weights are positive

The decision threshold is 0.5, equivalent to the score z being positive.

A 3-class LogisticRegression is fitted on 5 features. The shape of coef_ is...

(5,)
(5, 3)
(3, 5)
(1, 5)

One weight vector per class: (n_classes, n_features).

Decreasing C in LogisticRegression...

weakens the regularisation
strengthens the regularisation
changes the number of classes
turns off the bias

C is the inverse of the regularisation strength.

Select all TRUE statements. (select all that apply)

predict_proba rows sum to 1.
SGDClassifier needs loss="log_loss" to behave like logistic regression.
The default loss of SGDClassifier is log_loss.
decision_function returns probabilities.

The default is hinge, and decision_function returns the raw score.

Recap

Remember this

$z=\mathbf w^T\mathbf x+b$ , $p=\sigma(z)$ ; boundary $z=0$ ; decision_function $=z$ , predict_proba $=\sigma(z)$ , predict = threshold at 0.5.
LogisticRegression defaults: C=1.0, $\ell_2$ , lbfgs, max_iter=100. Small C = strong regularisation.
Multi-class: one weight row per class, softmax, coef_.shape=(K, d).
SGDClassifier(loss="log_loss") = incremental logistic regression (default loss is hinge).

Formulas to keep

$\sigma(z)=\frac1{1+e^{-z}}$ ; softmax $p_k=\frac{e^{z_k}}{\sum e^{z_j}}$
Log loss $-\frac1n\sum[y\ln p+(1-y)\ln(1-p)]$
predict_proba $\to(m,K)$ ; decision_function $\to(m,)$ binary, $(m,K)$ multi-class; coef_ $\to(1,d)$ or $(K,d)$

Connected weeks (learn them together)

MLF week 9 - Optimising with rules: Lagrange and KKT: Logistic regression in sklearn.
MLT week 9 - Perceptron and logistic regression: Logistic regression: the theory, then the code.

Practise with real exam questions

These come straight from past papers (with the real figures and the official answer key). Open 28 real questions on this week All Quiz 2 papers

Week 5 · Logistic regression in sklearn

The one-minute story

Words you need

Idea 1: From score to probability

Idea 2: Training and reading the model

Idea 3: The knobs: C, penalty, solver

Idea 4: More than two classes

Idea 5: The incremental version

Solved examples

Your turn

Exam radar

Quick quiz

Recap

Practise with real exam questions

Idea 3: The knobs: `C`, `penalty`, `solver`