Recently I'm dealing with the 3D rotation group $SO(3)$, which is also a manifold. I'm new to Manifold, so I'm a little bit confused about geodesically convex functions.
The $SO(3)$ are rotation matrices, so it naturally embeds into $\mathbb{R}^9$. In Euclidean space, the function $$f(X) = Tr(A^TX)\quad X\in SO(3)$$ for some matrix $A$ is affine in $X$, so it's concave and convex at the same time.
But we can also see this $f$ as a function from the manifold $SO(3)$ to $\mathbb{R}$. But on Riemannian manifolds, we can only define geodesically convex, which is defined to be the convexity on its geodesic. So I am wondering if this $f$ is geodesically convex/concave? If so how can we prove this? Or if not, are there any easily understood ways to show?
I have tried to follow the definition of geodesically convex and using Rodrigues' rotation formula. But the final form seems to be too complicated for me to reach the conclusion. Any guides or insights will be appreciated! Thanks in advance!
Beware: This is somewhat hand-wavy, and done mostly on-screen with hardly any proof-reading, so there's a huge chance I've made a number of misprints and possibly more critical errors!
As far as I can tell, a geodesic $X_t$ in $SO(3)$ can always be written on the form $$X_t = X_0\cdot\exp(tR)$$ where $R=-R^T$ is the direction and speed of rotation.
We are looking at $\text{tr}(AX_t)=\text{tr}[AX_0\cdot\exp(tR)]$, and $\exp(tR)$ is a constant speed rotation which is what makes the curve $X_t$ a geodesic. We define $$ \exp(M) =I+M+\frac{M^2}{2!}+\frac{M^3}{3!}+\cdots =\lim_{n\rightarrow\infty}\left(I+\frac{M}{n}\right)^n $$ which will come in handy later.
We can always choose a basis $e_1,e_2,e_3$ so that the rotation is in the $e_1, e_2$ plane. If the rotation speed is $\omega$, that makes $$ \exp(tR) = \exp\left[\begin{matrix} 0&t\omega&0\\ -t\omega&0&0\\ 0&0&0 \end{matrix}\right] = \left[\begin{matrix} \cos(t\omega)&\sin(t\omega)&0\\ -\sin(t\omega)&\cos(t\omega)&0\\ 0&0&1 \end{matrix}\right] $$ which we can then plug into the trace: $$ \text{tr}(AX_t) = \text{tr}\left(AX_0\cdot \left[\begin{matrix} \cos(t\omega)&\sin(t\omega)&0\\ -\sin(t\omega)&\cos(t\omega)&0\\ 0&0&1 \end{matrix}\right]\right) = a + b\,\cos(t\omega) + c\,\sin(t\omega) $$ where $a,b,c$ are linear in the coefficients of $AX_0$. Of course, we also have to express $A$ and $X_0$ in the $e_1,e_2,e_3$ basis.
So now we must determine when this expression is convex or concave in $t$.
The answer to that is simply that it is concave whenever $\text{tr}(AX_t)\ge a$ and convex whenever $\text{tr}(AX_t)\le a$. That's just because $\cos$ (or $\sin$) is concave when it is positive and convex when it is negative as we could have rewritten $\text{tr}(AX_t) = a + d\,\cos(t\omega - \nu)$ for $b=d\,\cos\nu$, $c=d\sin\nu$.
For the function $X\mapsto\text{tr}(AX)$ to be geodesically convex (or concave) at a point $X_0\in SO(3)$ is the same as saying that all geodesics passing through $X_0$ are convex (or concave) there. That is, $\text{tr}[AX_0\cdot\exp(tR)]$ is convex (or concave) at $t=0$ for all $R=-R^T$. We can check that by taking the second derivative at $t=0$, which is $\text{tr}(A X_0 R^2)$.
So $X\mapsto\text{tr}(AX)$ is convex (or concave) at $X=X_0$ if $\text{tr}(A X_0 R^2)\ge0$ (or $\le 0$) for all skew-symmetric matrices $R=-R^T$. This will be the case if $AX_0$ is negative definite (or positive definite), but the criterion can be relaxed somewhat.
If $R$ represents rotation an angle $\omega$ around a unit vector $u$, then $-R^2 = RR^T = \omega^2\cdot(I-u^Tu)$. Using this, the criterion $\text{tr}(A X_0 R^2)\ge0$ (or $\le 0$) can be rewritten $u A X_0 u^T\ge\text{tr}(A X_0)$ (or $\le$).
Thus, $X\mapsto\text{tr}(AX)$ is convex (or concave) at $X=X_0$ if $u A X_0 u^T\ge\text{tr}(A X_0)$ (or $\le$) for all unit vectors $u$.
If $A X_0$ has eigenvalues $\lambda_1\le\lambda_2\le\lambda_3$, then $\text{tr}(A X_0)=\sum_i \lambda_i$, while $\lambda_1\le u A X_0 u^T\le\lambda_3$ for unit vectors $u$ with either equality obtainable by entering the corresponding eigenvector as $u$. Thus, we can express the criterion in terms of the eigenvalues: $$ X\mapsto\text{tr}(A X)\quad \begin{cases} \text{is convex at $X=X_0$ if}\quad \lambda_1\ge\text{tr}(A X_0) \iff \lambda_2+\lambda_3\le 0; \\ \text{is concave at $X=X_0$ if}\quad \lambda_3\le\text{tr}(A X_0) \iff \lambda_1+\lambda_2\ge 0; \\ \text{otherwise, it is convex in some directions, concave in other.} \end{cases} $$