The gradient of a scalar-valued function is calculated. The $H$ matrix is symmetrical here and there is differentiate to both occurence of $x$ to $x$. My professor choose to write $y$ for the vector we differentiate to.
I found a transpose rule for differentiating here, yet I do not understand how it works in the solutions of the problem in this image.
Why does $x\cdot(Hy)$ become $x^T H y$, instead of $xH$? Why and how do they continue swapping transposed vectors and matrices.
Any answer would be appreciated, a lot.
Well, $x^THy$ is a scalar, and a scalar is equal to its transpose. Hence,
$$ x^THy = (x^THy)^T = y^T H^T x. $$
Also, if the dot-product between vectors $a$ and $b$ is denoted $a\cdot b$. Then we can also write
$$ y^T H^T x = y \cdot (H^T y). $$
See The properties of matrix transpose