I have a function that that takes a matrix and returns a scalar, $f : \mathbb{R}^{m\times n} \rightarrow \mathbb{R}$. I know how to calculate the derivative of this function with respect to the matrix argument. For example, if $f(X) = -\log \det X$, then $\nabla_X f(X) = -X^{-1}$. If $f(X) = tr(X^{-1} Y)$, $\nabla_X f(X) = -X^{-1} Y X^{-1}$ (assuming $X$ and $Y$ are symmetric for now).
What I don't know how to do is compute the second derivative (Hessian?) of $f$ with respect to $X$. I'm assuming it'll be messy, as the result might require four indices, if I'm not mistaken?
Here's the reason I'm curious: I'm trying to prove a function $f : \mathbb{R}^{m\times n} \rightarrow \mathbb{R}$ (specifically $f(X) = \log \det X + tr(X^{-1} Y)$) is convex; one way to do so is to prove that $\nabla^2_X f(X) \succeq 0$ everywhere. At least, this is true for functions $g : \mathbb{R}^n \rightarrow \mathbb{R}$, I haven't been able to find any generalizations of this to matrices, but I think it must be true for the correct notion of "Hessian."
Many times, convexity for these types of functions can be checked by proving convexity of $f(Z+tV)$ in $t$, for arbitrary $Z,V$ (see Boyd & Vandenberghe "Convex Optimization" p.74).
The general Hessian for $\log\det(X)$ would require 4 indices. See p.644 for an approximation.
(http://www.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf).