Home / MLP / Week 8

Week 8 · SVMs in sklearn

The one-minute story

By the end of this week you can:

• choose between LinearSVC, SVC and SVR;
• set C, kernel, degree, gamma, coef0, and predict what each does;
• read support_vectors_, n_support_, coef_ and compute the margin;
• explain why SVMs need scaled features and why l1 LinearSVC gives sparse weights.

Words you need

Idea 1: The three classes

• Training time. LinearSVC grows roughly linearly with the number of examples; kernel SVC grows between quadratically and cubically, so it is impractical beyond tens of thousands of rows.
• SVC(kernel="linear") is slower than LinearSVC but equivalent in spirit. (SGDClassifier(loss="hinge") is a third way to train a linear SVM, and the only one with partial_fit.)
• Multiclass: SVC uses one-vs-one internally; LinearSVC uses one-vs-rest (week 7).
• No probabilities by default. SVMs output scores (decision_function). SVC(probability=True) can calibrate probabilities by running an extra internal cross-validation: slow.

Idea 2: Always scale

Distances and dot products drive an SVM, so a feature with big numbers dominates the margin. Put a StandardScaler first, in a pipeline:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
model = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1, gamma="scale"))

Idea 3: Curvy boundaries two ways

(a) Add features yourself, then use a linear SVM:

make_pipeline(PolynomialFeatures(3), StandardScaler(), LinearSVC(C=10, max_iter=20000))

(b) Use the kernel trick (no explicit new features):

SVC(kernel="poly", degree=3, coef0=1, C=5)      # polynomial kernel
SVC(kernel="rbf",  gamma=0.5, C=1)              # Gaussian / RBF kernel

Option (b) avoids the explosion in the number of features (high degrees are cheap), while (a) can be slower or impossible as the degree and number of features grow. We tried both on the "two moons" data set (300 points with noise, 70/30 split):

The straight line underfits the curved moons; both polynomial routes fix it.

coef0 controls how much the polynomial kernel is influenced by high-degree vs low-degree terms (a constant added inside). gamma="scale" (the default) means $\gamma=1/(n_{\text{features}}\cdot\operatorname{Var}(X))$; on standardised data with 2 features this equals $0.5$. The RBF kernel is $e^{-\gamma\|\mathbf x-\mathbf x'\|^2}$.

Idea 4: C and gamma change everything

Same moons data, RBF kernel:

Notice the support-vector counts: the under-fit and very-local models use lots of support vectors (about 200 of 210 training points), while a neat model uses far fewer.

Idea 5: Looking inside

svc = SVC(kernel="linear", C=1).fit(X_train_scaled, y_train)
svc.support_vectors_.shape      # (74, 2): the 74 points that define the road
svc.n_support_                  # [38, 36]: how many per class
svc.dual_coef_.shape            # (1, 74): alpha_i * y_i for each support vector
svc.coef_, svc.intercept_       # the weights: ONLY for kernel="linear"

On our scaled moons data $\mathbf w=(0.743,\,-1.299)$ so the margin width $=\frac2{\|\mathbf w\|}=\frac{2}{1.497}=1.337$. For non-linear kernels coef_ does not exist; predictions use dual_coef_ and support_vectors_. Only the support vectors matter: you could delete every other point and the fitted model would not change.

Idea 6: LinearSVC details and sparsity

• The loss default is 'squared_hinge'; loss='hinge' is the plain SVM loss.
• penalty='l2' (default) shrinks weights. penalty='l1' gives a sparse coef_ (many exact zeros) and needs dual=False. On a 20-feature problem with only 3 informative features, we got 0 zero weights with l2 and 8 with l1.
• LinearSVC also penalises the bias and benefits from scaling more than most.

Idea 7: SVR (regression)

SVR fits a function surrounded by a tube of half-width $\varepsilon$: errors inside the tube are free; only points on or outside the tube become support vectors. A wider tube (larger epsilon) means fewer support vectors and a smoother fit: with $\varepsilon=0.05$ we had 23 support vectors, with $\varepsilon=0.3$ only 4. C again controls how hard points outside the tube are punished.

Solved examples

Your turn

Exam radar

Quick quiz

Recap

Practise with real exam questions

These come straight from past papers (with the real figures and the official answer key). Open 22 real questions on this week   All Quiz 2 papers