I'm working on a semidefinite program (SDP) in $X \ge 0$, and I want to obtain an optimal solution $X^*$ with rank at most $k$, where $k$ is typically a small integer.
Let $0 \le \lambda_1 \le \lambda_2 \le \cdots \le \lambda_n$ be the eigenvalues of $X$. I've seen many papers in the area of compressive sensing trying to minimize the trace of $X$
$$\mbox{tr}(X) = \sum_{i=1}^n \lambda_i$$
to obtain low-rank solutions. If I want to obtain a solution with rank at most $k$, should I minimize $\sum_{i=1}^k \lambda_i$? Any hints or references will be appreciated! Thanks!
The constraints $\mbox{rank}(X) \leq k$ is non-convex and solving SDP with rank constraints is an NP-Hard problem (if you're not familiar with the terminology, this means that the problem can be extremely hard and effectively impossible to solve.)
The heuristic of minimizing the trace norm (aka nuclear norm) of $X$ can be effective in minimizing the rank of $X$ in much the same way that minimizing the 1-norm of a vector can be effective in finding a sparse solution. You would typically truncate the singular values of $X$ to the $k$ largest to get an approximate solution that is rank-$k$.
Although trace norm minimization has been used in many applications, it is not always practical for applications in which $X$ is very large. There are other approaches to low-rank matrix approximation and related problems that might work better for your particular problem. However, without further information on what you're trying to do, we can't really help you with that.