This is the question:
Equations 5.127 and 5.128:
I am stuck on proving 5.206. This is my proof, and I am getting the wrong answer:
$\Omega_{n} = \frac{1}{2} \sum_{k} \left( \sum_{i} J_{nki}\tau_{ni} \right)^{2}$
$\Longrightarrow \frac{\partial \Omega_{n}}{\partial w_{rs}} = \frac{\partial }{\partial w_{rs}} \left( \frac{1}{2} \sum_{k} \left( \sum_{i} J_{nki}\tau_{ni} \right)^{2} \right)$
$= \frac{1}{2} \sum_{k} \frac{\partial }{\partial w_{rs}} \left( \sum_{i} J_{nki}\tau_{ni} \right)^{2}$
$= \frac{1}{2} \sum_{k} 2\left( \sum_{i} J_{nki}\tau_{ni} \right) \left( \frac{\partial }{\partial w_{rs}} \left( \sum_{i} J_{nki}\tau_{ni} \right)\right) $
$ = \sum_{k} \left( \sum_{i} J_{nki}\tau_{ni} \right) \left( \sum_{i} \frac{\partial J_{nki}\tau_{ni}}{\partial w_{rs}} \right) $
$ = \sum_{k} \left( \sum_{i} J_{nki}\tau_{ni} \right) \left( \sum_{i} \tau_{ni}\frac{\partial J_{nki}}{\partial w_{rs}} \right) $
$= \sum_{k} \left( \sum_{i} J_{nki}\tau_{ni} \right) \left( \sum_{i} \tau_{ni}\frac{\partial J_{nki}}{\partial a_{r}}\frac{\partial a_{r}}{\partial w_{rs}} \right)$
$= \sum_{k} \left( \sum_{i} J_{nki}\tau_{ni} \right) \left( \sum_{i} \tau_{ni} \left( \frac{\partial^{2} y_{k}}{\partial x_{i} \partial a_{r}} \right) \left(\frac{\partial a_{r}}{\partial w_{rs}} \right)\right)$
$= \sum_{k} \left( \sum_{i} J_{nki}\tau_{ni} \right) \left( \sum_{i} \tau_{ni} \left( \frac{\partial^{2} y_{k}}{\partial x_{i} \partial a_{r}} \right) \left(\frac{\partial \sum_{i'}w_{ri'}z_{i'}}{\partial w_{rs}} \right)\right)$
$= \sum_{k} \left( \sum_{i} J_{nki}\tau_{ni} \right) \left( \sum_{i} \tau_{ni} \left( \frac{\partial^{2} y_{k}}{\partial x_{i} \partial a_{r}} \right)z_{s} \right)$
$= \sum_{k} \left( \sum_{i} J_{nki}\tau_{ni} \right) z_{s} \left( \sum_{i} \tau_{ni} \left( \frac{\partial^{2} y_{k}}{\partial x_{i} \partial a_{r}} \right) \right)$
$= \sum_{k} \left( \sum_{i} J_{nki}\tau_{ni} \right) z_{s} \left( \sum_{i} \tau_{ni} \left( \frac{\partial}{\partial x_{i}} \delta_{kr} \right) \right)$
$= \sum_{k} \left( \sum_{i} J_{nki}\tau_{ni} \right) z_{s} \mathcal{G}\delta_{kr}$
$= \sum_{k} \alpha_{k} z_{s} \phi_{kr}$
I can't figure out where the error is.
The regularization term is the (squared) norm of $R = \frac12 \| \mathbf{\alpha}^L \|^2_2$ where $ \mathbf{\alpha}^L = \frac{\partial \mathbf{z}^{L}}{\partial \mathbf{x}_0} \mathbf{\tau} $.
It holds \begin{eqnarray*} \frac{\partial}{\partial W_{rs}} \left[ \frac{\partial z_k^L}{\partial \mathbf{x}_0} \right] &=& \frac{\partial}{\partial \mathbf{x}_0} \left[ \frac{\partial z_k^L}{\partial W_{rs}} \right]= \frac{\partial}{\partial \mathbf{x}_0} \left[ \frac{\partial z_k^L}{\partial a_r} \frac{\partial a_r}{\partial W_{rs}} \right] \\ &=& \frac{\partial}{\partial \mathbf{x}_0} \left[ \delta_{kr} z_s \right] = \frac{\partial \delta_{kr}}{\partial \mathbf{x}_0} z_s + \delta_{kr} \frac{\partial z_s}{\partial \mathbf{x}_0} \end{eqnarray*} It follows $$ \frac{\partial \alpha_k^L}{\partial W_{rs}} = \phi_{kr} z_s + \delta_{kr} \alpha_s $$
\begin{eqnarray*} dR &=& \sum_k \alpha_k^L d\alpha_k^L = \sum_k \alpha_k^L \left( \frac{\partial \alpha_k^L}{\partial W_{rs}} \right) dW_{rs} \\ \frac{\partial R}{\partial W_{rs}} &=& \sum_k \alpha_k^L \left[ \phi_{kr} z_s + \delta_{kr} \alpha_s \right] \end{eqnarray*}