Derivative of Softmax loss function with $\text{one}_\text{hot}(y)$

Question

Given the following:

$h=g\left(w_1\cdot x+b_1\right)$

$ z=w_2\cdot h+b_2$

$y_\text{hat} = \mbox{softmax}\left(z\right)$

$\mbox{Loss}\left(y,y_\text{hat}\right)=-\sum \mbox{onehot}\left(y\right)\cdot \log\left(y_\text{hat}\right)$

How can we derivate the loss function: $\frac{\partial L}{\partial z}$?

Answer 1

From here softmax, you will see that with your notations $$ \frac{\partial \mathrm{Loss}}{\partial \mathbf{z}} = \mathbf{y}_\mathrm{hat} - \mathrm{onehot}(y) $$ which is particularly simple...