For a positive semi-definite $A$, I need to find positive semi-definite $X$ which maximizes the following
$$2\text{Tr}(AX) - \|AX\|_F^2$$
- Is there a geometric or statistical interpretation of this quantity?
- I suspect it's maximized by $X=A^\dagger$, the pseudo-inverse, is there an easy way of showing this?
Motivation: this gives improvement in error norm squared after 1 step of least squares gradient descent to solve $Aw=b$ with preconditioner $X$
Let $\langle X,Y \rangle_F = tr(X^TY)$ be the Frobenius inner product. Then the expression you are trying to maximize is $2\langle AX, I \rangle_F - \langle AX, AX\rangle_F$, but maximizing that is equivalent to minimizing $\langle AX, AX\rangle_F-2 \langle AX, I \rangle_F $, which is turn equivalent to minimizing $\langle AX, AX\rangle_F-2\langle AX, I \rangle_F +\langle I, I \rangle_F=\| AX-I\|^2_F.$
This gives a sort of geometric interpretation of what you are looking for: you are minimizing a distance between a family of matrices and the identity matrix.
By using the definition of the Frobenius norm together with the cyclic shift property of trace, one can show that $\|UZU^T\|_F^2=\|Z\|_F^2$ for any orthogonal matrix $U$, and so one if $A=UDU^T$ is the orthogonal diagonalization of $A$, then $\|AX-I\|_F^2=\|DY-I\|^2_F$ where $UYU^T=X$.
In the diagonal case, it is a straight forward calculation to show that the pseudoinverse a the minimizer (the bottom $\dim \ker D$ rows can be arbitrary, giving some flexibility in the solution), and so the pseudoinverse is a minimizer in general.