prove of convexity of orthogonal matrix

Question

I don't know how I should formally prove that a set of all n*n orthogonal matrices is not convex? how I can show that the convex combination of two orthogonal matrices is not in the set? I know there is some examples like if we choose I and -I the convex combination is not in the set. But I want to prove it formally by convex definition?

Answer 1

Your example is correct. You can use it formally in a proof as follows: Define $$ S_n = \{ A \in \mathbb{R}^{n \times n} \mid A^T A = I_n \}, $$ where $I_n$ is the identity matrix. Suppose that $S_n$ is convex, then for any $X, Y \in S_n$ we must have for $t \in [0, 1]$ that $tX + (1-t)Y \in S_n$. Let $X = I_n$ and $Y = -I_n$ and $t = 1/2$, then $tX + (1-t)Y = 0$ but $0^T 0 \neq I_n$, it follows that $0 \notin S_n$ and hence by contradiction $S_n$ cannot be convex.