1. Loss Functions

Def: Characterize how “bad” a prediction is.

Regression

Of format $\ell (r)$, where $r=y-\hat{y}$ is the residual.

• Square/L2 loss: $\ell (r)=\frac{1}{2}{r}^2$
  • $\frac{1}{2}$ is a constant factor and therefore scales all losses equally. It is typically included b/c $L_{\text{square}}^{\prime }=r$
  • Very sensitive to outliers since squared
• Absolute/L1 loss: ${\ell }_{abs}(r)=|r|$
  • Not differentiable at zero.
• Huber loss: ${\ell }_{\text{huber}}(r)=\{\begin{array}{ll}\frac{1}{2}{r}^2& |r|\le \delta ,\\ \delta |r|-\frac{1}{2}{\delta }^2& |r|>\delta .\end{array}$
  • Uses the square loss for $[-\delta ,\delta ]$ and a linear loss for the rest
  • For the graph to be smooth (continuously differentiable):
    • The first derivative of linear parts must match at the transition $⟹\delta$ as slope of the abs part since $L_{\text{square}}^{\prime }=r$
    • The y-coordinate must match. $\frac{1}{2}(\pm \delta {)}^2=\frac{1}{2}{\delta }^2$ for the square part, but $\delta |\pm \delta |={\delta }^2$ $⟹$ we have to subtract $\frac{1}{2}{\delta }^2$
  • Advantage: Less sensitive to outliers while still differentiable
• Asymmetric loss (diff than graphic): ${\ell }_{\tau }(r)=\tau \max \{r,0\}+(1-\tau )\max \{-r,0\}$
  • Like absolute loss, but with two different slopes on both sides of the y-axis.
  • Higher $\tau$ correspond to a steeper graph when over-shooting, lower $\tau$ to a steeper graph when under-shooting.

For multiple datapoints: Calculate the average loss (e.g. mean squared error/MSE or mean Huber loss). Usually denoted by $L$.

Classification

Let $f(x)$ be the output of the fitted function.

Metric for Evaluation (0-1-Loss)

$${\ell }_{0-1}(\hat{y},y)={\mathbb{I}}_{\hat{y}\ne y}=\{\begin{array}{ll}1,& \text{if }\hat{y}\ne y;\\ 0,& \text{otherwise.}\end{array}$$

$y,\hat{y}\in \{-1,1\}$ as $\hat{y}=\mathrm{sgn}(f(x))$

Surrogate Loss

The 0-1 loss is not differentiable. Therefore: surrogate loss for training.

• Linear loss
  • ${\ell }_{\text{lin}}=-y\cdot f(x)$
  • Drawback: does not work if data is unbalanced.
• Exponential loss
  • ${\ell }_{\text{exp}}=e^{-y\cdot f(x)}$
  • Drawback: too sensitive to outliers (func values explode)
• Logistic loss
  • ${\ell }_{\log }(\hat{y},y)=\log \left(1+e^{-y\cdot f(x)}\right)$
  • Asymptotes: $0,f(x)$

For unbalanced data, the exp loss (left) works better than the linear loss (right) b/c it does not equally reward correctly predicted datapoints.

The exponential loss (red) is too sensitive to the teal outlier. The logistic loss (blue) is better.