Graph Regularized Nonnegative Matrix Factorization for Data Representation | Why second order derivative of $F_{ab}$ is like this?

Question

Graph Regularized Nonnegative Matrix Factorization for Data Representation
[From here]

Doubt : In equation $(28)$. Why the second order derivative of $F_{ab}$ is the form of the equation $(28)$? How to deduce subscripts $aa$ and $bb$?

Answer 1

Basically, they're hedging because they don't want to introduce the notion of 4th order tensors or vectorization to deal with a matrix-by-matrix hessian.

Instead, they take the matrix-valued gradient which they've calculated as $$F'=2(VU^TU-X^TU+\lambda LV)$$ and extract a single scalar element from it by pre/post multiplying by $\{e_k\}$ vectors from the standard basis $$F'_{ab} = e_a^TF'e_b \,=\, 2e_a^T(VU^TU-X^TU+\lambda LV)e_b$$ and then take the gradient of that with respect to $V$ $$F''_{ab} = \frac{\partial F'_{ab}}{\partial V} = 2e_ae_b^TU^TU + 2\lambda Le_ae_b^T$$ By doing things this way, all of the following quantities are matrices with identical dimensions $$V,F',F''_{ab} \in {\mathbb R}^{N\times K}$$ If you want to see a different way to handle things, check out Magnus & Neudecker's book, "Matrix Differential Calculus"