Machine LearningIntermediate1:30 video6 chapters

Support Vector Machines and the Kernel Trick — lecture notes

Of all the lines that separate two classes, pick the widest street. Then lift the data into a new dimension to separate what a line cannot.

▶ Watch the animated lecture

0:001. Introduction

Introduction — Support Vector Machines and the Kernel Trick

Support vector machines were among the most successful classifiers before deep learning. Their idea is elegant: find the widest possible street between two classes.

0:112. Maximum margin

Maximum margin — Support Vector Machines and the Kernel Trick

Many different lines separate the pink and blue points. Which is best? An SVM chooses the line with the widest margin, the empty street between the classes. The points touching the edges of the street are the support vectors. Only they decide where the boundary goes.

0:303. Why margins matter

Why margins matter — Support Vector Machines and the Kernel Trick

A wide margin leaves more room for new points that are a little different. Only the support vectors matter. When classes overlap, a soft margin allows a few mistakes, and the parameter C balances width against errors.

0:464. The kernel trick

The kernel trick — Support Vector Machines and the Kernel Trick

What if no straight line works? Here the blue points sit on both sides of the pink ones. No single threshold separates them. But add a new feature, x squared, and the blue points lift up. Now a straight line separates them easily. That is the idea behind the kernel trick.

1:085. Kernels

Kernels — Support Vector Machines and the Kernel Trick

Common kernels include linear, polynomial and the radial basis function. The clever part is that kernels compute similarities in the higher-dimensional space without ever building those features explicitly.

1:206. Recap

Recap — Support Vector Machines and the Kernel Trick

To recap. SVMs pick the widest street. Support vectors define it. Soft margins tolerate overlap. And kernels unlock curved boundaries.

Key takeaways

An SVM chooses the separating boundary with the largest margin.
Support vectors are the training points that lie on the margin.
Soft margins (parameter C) allow some mistakes when classes overlap.
Kernels map data to higher dimensions so a linear boundary can separate it.

Check yourself

Which points determine the SVM boundary?
Show answer
The support vectors on the margin — Only support vectors affect the maximum-margin solution.
In the kernel example, which new feature made the data separable?
Show answer
x² — Lifting points by x² separated the outer blue points from the inner pink points.
What does a soft margin allow?
Show answer
A few misclassified points when classes overlap — Soft margins trade a little error for a wider, more robust margin.

Go deeper

Support Vector Machines: Maximum-Margin Classification · The AI Lecture Hall
Kernel Methods and the Kernel Trick · The AI Lecture Hall