My problem can be listed as follows:
$$\begin{array}{ll} \text{minimize} & f(X)\\ \text{subject to} & \mbox{rank} (X) = 3\end{array}$$
Is there some efficient algorithm to solve this problem? High complexity is affordable for high performance. Thank you in advance.
The constraint $\mbox{rank}(X)=3$ is a non-convex constraint, so your optimization problem is in general non-convex.
One common approach to dealing with this is reparameterize the rank-3 matrix $X$ as
$X=VV^{T}$
where $V$ is an $n$ by $3$ matrix. When you do this, you'll get rid of the nonconvex constraint, but you'll probably create a situation in which the new function of $V$ is nonconvex.
There are also methods that work directly with the rank constraint to find a local minimum solution. Look for research on optimization on manifolds.