6. Kernels

When pre-training on broad data and fine tuning later: Fix $\Phi (x)$, find linear function on few fine tuning data.

$(1+x_1^T{x}_1^{’}{)}^m=1\dots$

Kernel functions must be symmetric and the kernel matrix PSD

Problem: Some problems are only linearly separable in very high dimensional spaces (simple ex.: circle of one color in other). Let $\varphi (x):{\mathbb{R}}^n\to {\mathbb{R}}^p$ be the func that maps to that higher dimension space.

Minimizer:

$$\arg \min L(y,\underset{\left[\begin{array}{c}—\varphi (x_1{)}^{\top }—\\ ⋮\\ —\varphi (x_n{)}^{\top }—\end{array}\right]}{\underbrace{\Phi }}w)$$

Predictor:

$$\begin{array}{rl}f(x)& =w^T\varphi (x)\\ & =({\Phi }^T\alpha {)}^T\varphi (x)\\ & =\sum _{i=1}^n{\alpha }_i\varphi (x_i{)}^T\varphi (x)\end{array}$$

We define $k(x,y)=\varphi (x{)}^T\varphi (y)$.

$$f(x)=\sum _{i=1}^n{\alpha }_{i}k(x_i,x)$$ $$\underset{\alpha \in {\mathbb{R}}^n}{\min }\frac{1}{n}\sum _{i=1}^n\ell \left(y_i,\sum _{j=1}^n{\alpha }_{j}k(x_i,x_j)\right)=\underset{\alpha \in {\mathbb{R}}^n}{\min }\frac{1}{n}\sum _{i=1}^n\ell (y_i,(K\alpha {)}_i).$$

Gaussian & Laplacian Kernels

How prevent overfitting?