Support Vector Classifiers

Theory, Mathematics, Kernels, and Python

1 Learning Objectives

By the end of this lesson, students should be able to:

1. Explain Support Vector Classifiers (SVC).
2. Define hyperplanes, margins, and support vectors.
3. Understand maximum-margin classification mathematically.
4. Distinguish hard-margin and soft-margin classification.
5. Explain the regularization parameter C.
6. Understand kernels and nonlinear classification.
7. Explain the importance of feature scaling.
8. Build an SVC model using Python.
9. Evaluate and tune an SVC model.
10. Identify the advantages and limitations of SVC.

2 1. Introduction to Classification

Classification is a supervised machine learning task in which a model learns to assign observations to predefined categories.

Examples include:

• Email spam detection.
• Medical diagnosis.
• Fraud detection.
• Image recognition.
• Customer churn prediction.

A dataset contains input features (X) and target labels (y).

For binary classification, the labels can be represented as

[ y_i {-1,+1}. ]

The objective is to learn a function that predicts the correct class for new observations.

2.1 What Is a Support Vector Classifier?

A Support Vector Classifier is a supervised learning algorithm that finds a decision boundary separating classes while maximizing the margin between them, subject to constraints that control classification errors.

SVC belongs to the Support Vector Machine (SVM) family.

The central idea is:

Find a decision boundary that separates classes with the widest possible margin while controlling classification errors.

3 2. Geometric Intuition

3.1 2.1 The Decision Boundary

For a linear classifier, the decision boundary is a hyperplane:

[ w^x+b=0 ]

where:

• (x) is the feature vector.
• (w) is the weight vector.
• (b) is the intercept.

In two dimensions, the boundary is a line. In three dimensions, it is a plane.

The decision function is

[ f(x)=w^x+b. ]

For binary classification, the predicted class is

[ y = \[\begin{cases} +1 & \text{if }f(x)\geq 0,\\ -1 & \text{if }f(x)<0. \end{cases}\]

]

3.2 2.2 The Margin

The margin is the distance between the decision boundary and the nearest training observations.

For a linear SVM, the two canonical margin boundaries are

[ w^x+b=+1 ]

and

[ w^x+b=-1. ]

The margin width is

[ . ]

Maximizing the margin is equivalent to minimizing the norm of the weight vector.

3.3 2.3 Support Vectors

Support vectors are training observations that constrain the position of the decision boundary.

They are important because the fitted SVM decision function depends directly on them.

Observations far from the boundary generally have less influence on the fitted solution.

Discussion question: Why might a wider margin help a classifier generalize to unseen data?

4 3. Mathematical Foundation

4.1 3.1 Hard-Margin SVM

Suppose the training data is perfectly linearly separable.

The optimization problem is

[ \[\begin{aligned} \min_{w,b}\quad & \frac12\|w\|^2\\ \text{subject to}\quad & y_i(w^\top x_i+b)\geq 1. \end{aligned}\]

]

The constraints ensure that each training observation is correctly classified and lies outside or on its corresponding margin boundary.

Hard-margin SVM does not permit margin violations.

Its main limitation is sensitivity to noise and outliers.

4.2 3.2 Soft-Margin SVM

Real-world datasets often contain overlapping classes and noisy observations.

Soft-margin SVM introduces slack variables (_i) to allow margin violations.

The optimization problem becomes

[ \[\begin{aligned} \min_{w,b,\xi}\quad& \frac12\|w\|^2+C\sum_{i=1}^{n}\xi_i\\ \text{subject to}\quad& y_i(w^\top x_i+b)\geq1-\xi_i,\\ &\xi_i\geq0. \end{aligned}\]

]

The parameter (C) controls the trade-off between maximizing the margin and penalizing violations.

4.3 3.3 Hinge Loss

Hinge loss is defined as

[ L(y,f(x))=(0,1-yf(x)). ]

The regularized objective can be written as

[ {w,b} |w|^2+ C{i=1}^{n}(0,1-y_if(x_i)). ]

Hinge loss is zero when an observation is correctly classified with a sufficient margin.

It is positive when an observation violates the margin constraint.

5 4. Understanding the Parameter C

The parameter C controls how strongly the model penalizes margin violations.

• Small C: More tolerance for violations and stronger regularization.
• Large C: Stronger penalties for violations and a tighter fit to the training data.

A large value of C can increase overfitting risk, particularly when the dataset is noisy.

5.1 Demonstration: Comparing C Values

import numpy as np import matplotlib.pyplot as plt

from sklearn.datasets import make_classification from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler from sklearn.svm import SVC

X, y = make_classification( n_samples=180, n_features=2, n_redundant=0, n_informative=2, n_clusters_per_class=1, flip_y=0.08, random_state=42 )

C_values = [0.01, 1, 100]

fig, axes = plt.subplots(1, 3, figsize=(15, 4))

for ax, C in zip(axes, C_values): model = make_pipeline( StandardScaler(), SVC(kernel=“linear”, C=C) )

model.fit(X, y)

xx, yy = np.meshgrid(
    np.linspace(X[:, 0].min() - 1, X[:, 0].max() + 1, 250),
    np.linspace(X[:, 1].min() - 1, X[:, 1].max() + 1, 250)
)

grid = np.c_[xx.ravel(), yy.ravel()]
Z = model.predict(grid).reshape(xx.shape)

ax.contourf(xx, yy, Z, alpha=0.2, cmap="coolwarm")
ax.scatter(
    X[:, 0], X[:, 1],
    c=y, cmap="coolwarm",
    edgecolors="k", s=25
)

ax.set_title(f"C = {C}")
ax.set_xlabel("Feature 1")
ax.set_ylabel("Feature 2")

plt.tight_layout() plt.show()