I know that for a matrix positive definite variable $X$, $\mathrm{tr}(X^{-1})$ is a convex function. Now I want to consider the following function:
$f(X) = \frac{(\mathrm{tr} (A X^{-1}))^2}{\mathrm{tr} (B X^{-1})}$
where $A$ and $B$ are also positive definite matrices. Could we say anything about its convexity/concavity?
My effort: I tried to take the second derivative of the above function but it seems to become very complicated.
It's pretty easy to construct a counterexample using a symbolic computation package. Let
$A=\left[ \begin{array}{cc} 10 & 0 \\ 0 & 1 \\ \end{array} \right]$
$B=\left[ \begin{array}{cc} 1 & 0 \\ 0 & 10 \\ \end{array} \right]$
And consider matrices
$X=\left[ \begin{array}{cc} X_{1,1} & 0 \\ 0 & X_{2,2} \\ \end{array} \right]$
At $X_{1,1}=1$, $X_{2,2}=1$, the Hessian is
$H=\left[ \begin{array}{cc} 52.73 & -14.72 \\ -14.72 & -1.27 \\ \end{array} \right]$
which is clearly not positive semidefinite and also not negative semidefinite. Thus the function is not convex or concave in general.