English isn't my first language, so I'm sorry if I am unable to clearly describe the problem. Recently I have encountered a question in my artificial intelligence course that I am unable to wrap my head around. It involves using backward-propagation algorithm and deep learning to find the derivative of a parameter of a neural network. Here is an image link to an aritficial intelligence question I am stuck on So my first thought in tackling this question is:
- Taking the derivative of the transpose is weird, because I can't write the derivative as a matrix.
- However, I could twist the equation and denote $y = \langle W_1 \sigma(W_2 x), \sigma(W_1\sigma(W_3 x))\rangle $, where $\langle , \rangle $ is the dot product/standard inner product. And note that $\frac{d\langle v, w\rangle }{dw} = v^T$ and $\langle v, w\rangle = \langle w, v\rangle $ so $\frac{d\langle v, w\rangle }{dv} = \frac{d\langle w, v\rangle }{dv} = w^T$
But even then, I am still not sure myself if I am tackling the question the right way.
Write it as follows with intermediate variables (hidden layers)
$$ h_{11} = \sigma(W_2 x)$$
$$ h_{12} = \sigma(W_1 h_{11})$$
$$ h_{21} = \sigma(W_3 x)$$
$$ h_{22} = \sigma(W_1 h_{21})$$
$$ y = h_{12}^T*h_{22} $$