can anyone help me to solve the following optimization problem? thank you very much.
min_{U} F(U)
subject U'U=I_r
where U is a matrix of size m-by-n ,F(U) is a nonlinear function with respect to U. I_r is an r-by-r identity matrix.
Let's put the parameters in order: if $U$ has $m$ rows and $n$ columns, the product $U'U$ has size $n\times n$. So, $r=n$ in your statement.
The condition on $U$ says that its columns are orthonormal. (This only happens if $m\ge n$, by the way). The set of such matrices is a compact manifold; I think it's the product of Grassmannian $G(n,m)$ with the special orthogonal group $SO(n)$. The tangent space at $U$ consists of matrices $M$ such that $(U+tM)'(U+tM)=I_n+o(t)$, which amounts to $U'M+M'U=0$.
Thus, in order for $F$ to have a critical point at $U_0$, the directional derivative $(F(U_0+tM)-F(U_0))/t$ must be equal to $0$ for all $M$ such that $U_0'M+M'U_0=0$. How effective this approach is in locating the critical points of $F$ depends on what your $F$ is.