I have a linear cost function $f(r_{11},r_{12},...r_{33}, t_1, t_2, t_3)$ in terms of the elements of a 4x4 matrix $RT$. It is a rotation cum translation matrix and so the search should obey the equality constraint for $r_{11}$ to $r_{33}$ being elements of a rotation matrix $R$ which is $R^T * R = I$. \begin{equation} RT = \begin{pmatrix} r_{11} & r_{12} & r_{13} & t_1 \\ r_{21} & r_{22} & r_{23} & t_2 \\ r_{31} & r_{32} & r_{33} & t_3 \\ 0 & 0 & 0 & 1 \end{pmatrix} \end{equation}
What is the best optimization method to solve this problem? My cost function is not of the form $f(R)$ and hence I think I cannot use the suggestion given in this post. I found people using Lagrange Multipliers approach for constrained optimization. In that case $R^T * R = I$ will generate 9 constraints, equating to each element of $I$. Is that right?