7. NN

z: pre-activation value v: post-activation value

ReLU

• weights going in to neural net decides how inputs are weighted
• bias decides when the linear part starts
• weight going out of neuron decides if the lin part points up or down + how steep it is

$⟹$ combining yields piecewise linear function

Any function in a hypercube like $[0,1]$ can be uniformly approximated this way (diff get arbitrarily small with increasing $n$)

Increasing depth: Each segment of next layer calls previous layer as subroutine

great graphic in slides/script

Backpropagation

Use chain rule to move from back to front, updating weights and biases in direction of gradient

Via chain rule: each stage needs previous ones.

Sigmoid dervative: $\sigma (z)(1-\sigma (z))$

Weight-Space Symmetries

• You can permute a layer’s weight. Leads to same result.

Initialization

• 0 is bad b/c the gradient is 0 everywhere

• Use certain activation functions like ReLU can help avoid $\phi ’(z)=0$

• Scale error signal by $v^{l-1}$

Expected values/Variance proof

Learning Rate

• Often use decaying learning rate scheduler, e.g. piecewise constant
• Monitor ratio of weight change to weight magnitude

Regularization

• Regularization (weight decay):
  • LASSO or Ridge with vectorized weights (write all weights into one huge vector)
  • Multiply weights with constants slightly lower than 1 during each iteration (weight decay)
• Early stopping: Stop once validation error “stops to decrease,” ex. by saving checkpoints

Dropout

• Some units only activate for very specific training examples
• During training: don’t use all units for each mini batch. Only activate each unit by a certain probability $p$
• During inference: use all units, but multiply the weights by $p$

Why it’s allowed:

$$\begin{gathered} \begin{array}{rlrl}\mathbb{E}[z]& =\mathbb{E}\left[\sum _{i=1}{w}_i{v}_i{S}_i\right]& & \text{where}{S}_i\sim \mathrm{Ber}(p)=\text{cases}0\text{ if “dropped out”} \\ 1\text{ otherwise}\\ & =\sum _i{w}_i{v}_i\underset{p}{\underbrace{\mathbb{E}[S_i]}}& & (w_i,v_i\text{ const. }\forall i)\\ & =p\cdot (w^{T}v)& & \\ & =(pw{)}^{T}v& & \end{array} \end{gathered}$$

Batch Normalization

• Normalize inputs in layer according to minibatch statistics, i.e. s.t. the outputs of the mini-batch are

During training:

1. Input: activation values of one unit across minibatch, ex. $v_i\forall i\in S$ in minibatch $S$
2. Learnable parameters: $\gamma ,\beta$
3. Batch normalization: $\bar{v}=\mathrm{BN}(v;\gamma ,\beta )$ by
   1. Mean: ${\mu }_S=\frac{1}{|S|}\sum _{i\in S}{v}_i$
   2. Variance: ${\sigma }_S^2=\frac{1}{|S|}\sum _{i\in S}(v_i-{\mu }_S{)}^2$
   3. Normalize: $\widehat{v_i}=\frac{v_i-{\mu }_S}{\sqrt{{\sigma }_S^2+ϵ}}$ where $ϵ$ is a very small value to avoid div. by zero
   4. Scale and shift: $\overline{v_i}=\gamma \widehat{v_i}+\beta$
4. Output: $\overline{v_i}\forall i\in S$ in the minibatch $S$

During inference:

• Use running averages across training

Benefits:

• Larger step sizes
• Initialization less relevant
• Avoid overfitting