I frequently need to solve optimization problems which involves a 3D rotation R. Usually, I will just use exponential map to convert R from SO(3) to so(3) , then update directly in so(3) space. In my understanding, the 3 elements of so(3) are the weights for its 3 generators, the 3 generators form a basis in so(3) space. So that 3 elements in so(3) are independent to each other, updating them during the optimization should be optimal.
There's also a way of doing this optimization but using rotation vector, which is also a 3d vector, who's unit vector represents an axis in 3d space, the norm of it represents the angle around that axis. The optimization treats the rotation vector the same as so(3). But I feel that embedding the angle into rotation vector makes its 3 dimension dependent to each other, which means the later updating step in the optimization may not be optimal. But the Rodriguez's formula tells that the rotation vector is also in so(3).
The whole stuff has made me confused for days. I am sure I am wrong above somewhere, can somebody tell me about it ? are they identical by optimizing with so(3) or with rotation vector ? Why ? If you can also give me some books & documents to read, it's also greatly appreciated.
p.s. Most of my problems are non-linear least square problems, So by optimization, mostly I use Gaussian-Newton or LM.