Define the set of rotation matrices:
\begin{equation}
\begin{aligned}
SO(n) := \{X\in \textbf{R}^{n\times n}: X^TX=I, \text{det}(X)=1\}
\end{aligned}
\end{equation}
I want to prove that if $X\in SO(n)$, $X$ is an extreme point of conv $SO(n)$.
My work:
- Suppose not. Then $X=(X_a+X_b)/2$ where $X_a,X_b\in $ conv $SO(n)$, $X_a,X_b$ are defined at the bottom.
- By the definition of $SO(n)$, we have the following
\begin{align*} &\big(\frac{X_1+Y_1}{2}\big)^T\big(\frac{X_1+Y_1}{2}\big)=I \\ \Rightarrow \ \ & \frac{1}{4}X_1^TX_1+\frac{1}{4}X_1^TY_1+\frac{1}{4}Y_1^TX_1+\frac{1}{4}Y_1^TY_1=I \\ \Rightarrow \ \ & \frac{1}{4}(Y_1^TX_1 + X_1^TY_1)=\frac{1}{2}I \\ \Rightarrow \ \ & Y_1^TX_1+X_1^TY_1 = 2I \end{align*}
I have no idea how to come up with the contradiction.
The definition of conv $SO(n)$:
http://arxiv.org/pdf/1403.4914v1.pdf (p.1315)
According to @stewbasic suggestion:
If $X_a, X_b\in$ conv $SO(n)$, then $X_a=\sum \theta_i X_i$ and $X_b=\sum \alpha_i Y_i $ where $X_i,Y_i\in SO(n)$. And $\sum \theta_i = 1, \sum \alpha_i=1$ and $\theta_i , \alpha_i \geq 0$. Consider the simplest case, $X_a=X_1$ and $X_b=Y_1$ with $X_1,Y_1\in SO(n)$. And go back to the proof above.