Regression

Loss Functions

Def: Characterize how “bad” a prediction is. Of format $L(r)$, where $r=y-\hat{y}$ is the residual.

• Square/L2 loss: $\ell (r)=\frac{1}{2}{r}^2$
  • $\frac{1}{2}$ is a constant factor and therefore scales all losses equally. It is typically included b/c $L_{\text{square}}^{\prime }=r$
  • Very sensitive to outliers since squared
• Absolute/L1 loss: ${\ell }_{abs}(r)=|r|$
  • Not differentiable at zero.
• Huber loss: ${\ell }_{\text{huber}}(r)=\{\begin{array}{ll}\frac{1}{2}{r}^2& |r|\le \delta ,\\ \delta |r|-\frac{1}{2}{\delta }^2& |r|>\delta .\end{array}$
  • Uses the square loss for $[-\delta ,\delta ]$ and a linear loss for the rest
  • For the graph to be smooth (continuously differentiable):
    • The first derivative of linear parts must match at the transition $⟹\delta$ as slope of the abs part since $L_{\text{square}}^{\prime }=r$
    • The y-coordinate must match. $\frac{1}{2}(\pm \delta {)}^2=\frac{1}{2}{\delta }^2$ for the square part, but $\delta |\pm \delta |={\delta }^2$ $⟹$ we have to subtract $\frac{1}{2}{\delta }^2$
  • Advantage: Less sensitive to outliers while still differentiable
• Asymmetric loss (diff than graphic): ${\ell }_{\tau }(r)=\tau \max \{r,0\}+(1-\tau )\max \{-r,0\}$
  • Like absolute loss, but with two different slopes on both sides of the y-axis.
  • Higher $\tau$ correspond to a steeper graph when over-shooting, lower $\tau$ to a steeper graph when under-shooting.

For multiple datapoints: Calculate the average loss. (e.g. mean squared error or mean huber loss).

Gradient

• Loss gradient: n-dimensional, where each axis is a parameter of the model that we can tweak.
• We try to find the global minimum, i.e. where the mean loss is the least.

Closed Form for MSE Regression

$y_i$ is the $i$-th output and

$$\sum _{i=1}^n{\left(y_i-{\mathbf{w}}^{\top }{\mathbf{x}}_i\right)}^2=\Vert \mathbf{y}-\mathbf{X}\mathbf{w}{\Vert }_2^2$$

The optimal weights $\hat{w}$ are defined by:

$$\hat{\mathbf{w}}=\arg \underset{\mathbf{w}\in {\mathbb{R}}^d}{\min }\Vert \mathbf{y}-\mathbf{X}\mathbf{w}{\Vert }_2^2$$

Since minima always have a slope of 0, we are looking for $\hat{\mathbf{w}}$ s.t.

$$\begin{array}{rl}{\nabla }_{\mathbf{w}}\Vert \mathbf{y}-\mathbf{X}\hat{\mathbf{w}}{\Vert }_2^2=0& ⟺2{\mathbf{X}}^{\top }\left(\mathbf{X}\hat{\mathbf{w}}-\mathbf{y}\right)=0\\ & ⟺{\mathbf{X}}^{\top }\left(\mathbf{X}\hat{\mathbf{w}}-\mathbf{y}\right)=0\\ & ⟺{\mathbf{X}}^{\top }\mathbf{X}\hat{\mathbf{w}}={\mathbf{X}}^{\top }\mathbf{y}\end{array}$$

which is the normal equation we already know 🥳.