$L=\sum_{i}(y_i-t_i)^2$
I want to differentiate $L$ by $y$. Unfortunately I didn't learn this material, so a wild guess would be:
$(1) \frac{dL}{dy}=\frac{d}{dy} (\vec{y}\cdot \vec{y}^T -2\cdot \vec{y} \cdot \vec{t}^T + \vec{t} \cdot \vec{t})$
From here I got to: $=2\vec{y}-2\vec{t}^T$
But I guess this not ok, because the dimensions are not equal. This means I needed to do:
$(2) \frac{dL}{dy}=\frac{d}{dy} \vec{y}\cdot \vec{y}^T -2\cdot \vec{t} \cdot \vec{y}^T + \vec{t} \cdot \vec{t}$
And then I'll get $=2\vec{y}-2\vec{t}$ which is the correct answer I think.
I switched places because it suited me better, but I don't think that math works that way. I'm sure there is an algorithm for these operations. How I was supposed to know that I should of opened it like in $(2)$ and not like $(1)$ ?
\begin{align}\frac{\partial L}{\partial y_j}&=\frac{\partial }{\partial y_j} \sum_i (y_i-t_i)^2\\&= \frac{\partial }{\partial y_j} \left[(y_j-t_j)^2 + \sum_{i \ne j} (y_i-t_i)^2\right]\\&= \frac{\partial }{\partial y_j} \left[(y_j-t_j)^2\right] + \left[\frac{\partial}{\partial y_j} \sum_{i \ne j} (y_i-t_i)^2\right] \\&=2(y_j-t_j) \frac{\partial (y_j-t_j)}{\partial y_j }+0\\&=2(y_j-t_j) \end{align}
Hence $$\nabla_y L=2(y-t)$$
Note that $y^Tt=t^Ty$, here all my vectors are column vectors.