I have a question about the convexity of an optimization problem and its solution. Suppose $f(X)=-tr(A^{T}XA)+tr(X)$, $A$ is any matrix with its dimension "matched" with $X$. The optimization problem is trying to minimize $f(X)$ such that $X$ has to be a p.s.d matrix(member of a p.s.d cone). This is a convex optimization problem as the cost function is linear in $X$.
This problem can be solved in projected gradient method. But a lot of projection on p.s.d cone has to be performed through thresholding the eigen value. The computational complexity is huge for high dimensional $X$.
I am trying to avoid projection onto p.s.d cone. First, I want to replace $X$ by $Y^{T}Y$, and the problem becomes minimize $f(Y)=-tr(A^{T}Y^{T}YA)+||Y||^{2}_{F}$ such that $Y^{T}Y$ is a p.s.d matrix. Since $Y^{T}Y$ must be a p.s.d matrix, the constraint is removed. The problem is not convex because of the first term$f(Y)=-tr(A^{T}Y^{T}YA)$. In this case, what kind of method should be applied to solving this problem? Also, what will be the difference in terms of solving the problem between the case when $Y$ is full rank square matrix like $X$ and when $Y^{T}Y$ is a low rank matrix(for example $Y$ is m by n matrix and m is (much)smaller than n. Here rank(Y)<=min{m,n}?
Does what I did for the problem make sense? For original problem, what is the best way to solve it?
Many Thanks
Henry
Sam Burer and co-workers attacked semi-definite programming problems using this nonlinear programming strategy some 10 years ago.