Lecture 12: Support Vector Machines

Babafemi Sorinolu

Learning Goals

By the end of this lecture, you should be able to:

  1. Explain what a margin and a support vector are.
  2. Explain why a soft margin handles outliers better than a hard margin.
  3. Describe what the hyperparameter C controls.
  4. Explain the kernel idea: move data to a higher dimension where a straight line works.
  5. Describe the polynomial and RBF kernels and what gamma controls.
  6. Train and tune an SVM in scikit-learn (with feature scaling!).

The Big Picture

Step Idea Name
1 Separate two classes with the widest possible gap Maximal margin classifier
2 Allow a few mistakes so outliers don’t ruin it Support Vector Classifier (soft margin)
3 If no straight line works, add dimensions until one does Support Vector Machine (kernels)

Running example

A doctor gives patients different doses of a drug (mg). We want to predict whether a new patient will be cured (blue) or not cured (orange).

Part 1: Thresholds and Margins

Classifying with One Feature

Three number lines showing the data, a bad threshold and a maximal-margin threshold

Margin and Support Vectors

  • Threshold (decision boundary): the point where we switch from “not cured” to “cured”.
  • Margin: the distance between the threshold and the closest points.
  • Maximal margin classifier: put the threshold halfway between the closest points, which makes the margin as big as possible.
  • Support vectors: the closest points (circled), which define the threshold.

Key insight

Only the support vectors matter. Move any other point (without crossing the margin) and the threshold does not change.

Problem: Outliers

Hard margin threshold dragged by an outlier versus soft margin threshold

Soft Margin = Allow Some Mistakes

  • A hard margin must classify every training point correctly, so it is very sensitive to outliers.
  • A soft margin allows some training points to be on the wrong side (or inside the margin).
  • This is the bias–variance tradeoff again:
    • a little more bias (we accept mistakes on the training data) …
    • … for less variance (better predictions on new data).

Important

A classifier that uses a soft margin is called a Support Vector Classifier (SVC). How many mistakes to allow is a hyperparameter, so we pick it with cross-validation.

More Features = More Dimensions

Number of features Data lives in … The Support Vector Classifier is a …
1 (dose) a number line point
2 (dose, age) a plane line
3 (dose, age, weight) 3-D space plane
4 or more can’t draw it! hyperplane

The idea is always the same: find the boundary with the biggest (soft) margin.

A Support Vector Classifier in 2-D

Two clusters separated by a line with dashed margin lines and three circled support vectors

The Hyperparameter C

In scikit-learn, C controls how much mistakes are punished.

Linear SVC with C equal to 0.01, 1 and 100
  • Small C: mistakes are cheap, giving a wide margin, more support vectors, and simpler models (risk of underfitting).
  • Large C: mistakes are expensive, giving a narrow margin and models that chase every point (risk of overfitting).

Check Your Understanding

  1. If we delete a point that is not a support vector and retrain, does the boundary change?
  2. You increase C from 1 to 1000. Does the margin get wider or narrower?
  3. Why might a soft margin give lower training accuracy but higher test accuracy?
  1. No. Only support vectors define the boundary.
  2. Narrower. Mistakes become very expensive, so the model squeezes the margin to avoid them.
  3. It ignores outliers / noise instead of memorizing them, so it generalizes better (lower variance).

Part 2: Support Vector Machines

When a Straight Line Is Not Enough

Cured patients in the middle surrounded by not-cured patients; every threshold makes mistakes

The Trick: Add a Dimension

One-dimensional data lifted to x and x squared, where a straight line separates the classes

The 3 Steps of a Support Vector Machine

  1. Start with data in a low dimension (here: 1 feature, the dose \(x\)).
  2. Create new dimensions from the existing data (here: add \(x^2\)).
  3. Find a Support Vector Classifier that separates the data in the higher dimension.

Back in the original space …

A straight line in \((x, x^2)\) space becomes two thresholds on the original number line: “cured” if the dose is in the middle. That’s a non-linear boundary.

But how do we choose the new dimensions? \(x^2\)? \(x^3\)? \(\sqrt{x}\)? This is where kernels come in.

Kernel Functions

A kernel is a function \(K(a, b)\) that takes two data points and returns a number that measures how they relate in a higher dimension.

Two popular kernels:

Kernel Formula Hyperparameters
Polynomial \(K(a, b) = (a \times b + r)^d\) \(r\) (coef0), \(d\) (degree)
Radial Basis Function (RBF) \(K(a, b) = e^{-\gamma (a - b)^2}\) \(\gamma\) (gamma)

Both are chosen with cross-validation, along with C.

The Polynomial Kernel Is a Dot Product

Take \(r = \tfrac{1}{2}\) and \(d = 2\):

\[ \begin{aligned} \left(ab + \tfrac{1}{2}\right)^2 &= a^2b^2 + ab + \tfrac{1}{4} \\[4pt] &= \underbrace{a \cdot b}_{\text{1st}} + \underbrace{a^2 \cdot b^2}_{\text{2nd}} + \underbrace{\tfrac{1}{2} \cdot \tfrac{1}{2}}_{\text{3rd}} \\[4pt] &= \left(a,\ a^2,\ \tfrac{1}{2}\right) \cdot \left(b,\ b^2,\ \tfrac{1}{2}\right) \end{aligned} \]

  • A dot product multiplies matching entries and adds them up.
  • So this kernel acts as if each point had coordinates \((x,\ x^2,\ \tfrac{1}{2})\), which is exactly the \(x^2\) trick from before!
  • The last coordinate is \(\tfrac{1}{2}\) for everyone, so we can ignore it.

Try It: Kernel = Dot Product

Two patients received doses \(a = 2\) and \(b = 3\). Use \(r = \tfrac{1}{2}\), \(d = 2\).

  1. Compute the kernel directly: \((a \times b + \tfrac{1}{2})^2\)
  2. Compute the dot product: \((a,\ a^2,\ \tfrac{1}{2}) \cdot (b,\ b^2,\ \tfrac{1}{2})\)
  1. \((2 \times 3 + 0.5)^2 = 6.5^2 = 42.25\)
  2. \((2)(3) + (4)(9) + (0.5)(0.5) = 6 + 36 + 0.25 = 42.25\)

Same answer, and we never had to build the new coordinates! 🎉

The Kernel Trick

Without kernels

  1. Transform every point into high dimensions
  2. Store the new (possibly huge) feature vectors
  3. Fit a classifier there

With kernels

  1. Compute \(K(a, b)\) for pairs of points
  2. That’s all the SVM needs to find the best classifier
  3. No transformation is ever stored

The kernel trick

SVMs only need the relationships between pairs of points (dot products in the high-dimensional space), and a kernel computes those directly from the original data. This makes high (even infinite!) dimensions affordable.

The actual optimization (finding which points are support vectors) uses a method called Lagrange multipliers; we let scikit-learn handle it.

The RBF (Radial) Kernel

\[ K(a, b) = e^{-\gamma\,(a - b)^2} \]

  • It measures similarity: close points → value near 1, far points → value near 0.
  • So it behaves like a weighted nearest neighbors: a new point is classified mostly by the training points closest to it.
  • It works in infinitely many dimensions, but the formula stays this simple.
  • It is the default kernel in scikit-learn.

RBF kernel value versus distance for gamma 0.1, 1 and 10

The Hyperparameter gamma

gamma controls how far the influence of one training point reaches.

RBF SVM boundaries for gamma 0.05, 1 and 100 on moon-shaped data
  • Small gamma: each point influences a wide area, so the boundary is smooth (risk: underfit).
  • Large gamma: each point influences only its neighborhood, so the boundary is wiggly (risk: overfit).

Comparing Kernels

Linear, polynomial and RBF kernels on moon-shaped data
  • Linear: a straight boundary. Fast, a good baseline, and great when you have many features (e.g., text).
  • Polynomial / RBF: curved boundaries for non-linear patterns.

Part 3: SVMs in scikit-learn

Training an SVM

from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

svm = make_pipeline(
    StandardScaler(),                  # SVMs are distance-based → ALWAYS scale!
    SVC(kernel="rbf", C=1.0, gamma="scale")
)
svm.fit(X_train, y_train)
print("Test accuracy:", svm.score(X_test, y_test))
Argument Meaning Typical values to try
kernel "linear", "poly", "rbf" start with "rbf"
C penalty for mistakes 0.01, 0.1, 1, 10, 100
gamma reach of each point (RBF / poly) 0.001 … 10, or "scale"
degree, coef0 \(d\) and \(r\) of the polynomial kernel 2, 3 / 0, 1

Why Scaling Matters (Remember Lecture 6?)

  • Kernels use distances and dot products between points.
  • If one feature is in the thousands (e.g., income) and another is between 0 and 1, the big feature dominates every distance.
  • Put a StandardScaler inside a pipeline so that cross-validation scales correctly (no data leakage).

Warning

An unscaled SVM can perform much worse than a scaled one on the same data. You’ll see this in the practice notebook.

Tuning C and gamma with Cross-Validation

from sklearn.model_selection import GridSearchCV

param_grid = {
    "svc__C":     [0.01, 0.1, 1, 10, 100],
    "svc__gamma": [0.001, 0.01, 0.1, 1, 10],
}
search = GridSearchCV(svm, param_grid, cv=5)
search.fit(X_train, y_train)

print(search.best_params_)
print("Best CV accuracy:", search.best_score_)
print("Test accuracy:   ", search.score(X_test, y_test))
  • svc__C means “the C of the step named svc in the pipeline”.
  • Search on a log scale (0.01, 0.1, 1, …) because these hyperparameters work by orders of magnitude.

Looking Inside a Trained SVM

model = search.best_estimator_.named_steps["svc"]

model.support_vectors_     # the support vectors themselves
model.n_support_           # how many per class
model.decision_function(X) # signed distance from the boundary
  • decision_function > 0 → one class, < 0 → the other; the size tells you how confident the model is.
  • SVC does not give probabilities by default. Use SVC(probability=True) if you need predict_proba (slower).

Strengths and Weaknesses

👍 Strengths 👎 Weaknesses
Works well with many features, even more features than samples Slow on large datasets (training grows quickly with the number of rows)
Kernels capture non-linear patterns Must scale features and tune C / gamma
Only the support vectors matter, so the model is memory efficient Hard to interpret (no feature importances for RBF)
Soft margin is robust to outliers No probabilities by default

Wrap-up

Quick Check

Match each change to its effect:

Change Effect?
A. Increase C 1. Boundary becomes smoother
B. Decrease gamma 2. Margin becomes narrower
C. Switch from linear to RBF kernel 3. Model can learn curved boundaries
D. Forget to scale features 4. Large-valued features dominate the distances

A → 2, B → 1, C → 3, D → 4

Key Takeaways

  • An SVC finds the boundary with the largest margin; support vectors are the points that define it.
  • A soft margin (controlled by C) trades a few training mistakes for better generalization.
  • An SVM uses a kernel to work in a higher dimension where a straight boundary exists, without ever computing the new coordinates.
  • Polynomial kernel \((ab + r)^d\); RBF kernel \(e^{-\gamma (a-b)^2}\), where gamma controls each point’s reach.
  • In practice: scale → SVC → grid search C and gamma with cross-validation.

Class Practice 💻

Open the practice notebook and work through it with a partner:

📓 Download svm_practice.ipynb

You will:

  1. Find a maximal-margin threshold by hand, then check it with SVC.
  2. Explore how C changes the margin and the number of support vectors.
  3. Verify that a polynomial kernel equals a dot product.
  4. See an RBF SVM curve around non-linear data, and tune gamma.
  5. Classify breast cancer tumours: scaling, pipelines, and GridSearchCV.