Find $\text{card}(T_n( \mathbb R ) \cap O(n))$.

Question

Let $T_n( \mathbb R )$ be the set of upper triangular matrices of size $n$.

Let $O(n)$ be the set of general orthogonal matrices and $SO(n)$ the set of special orthogonal matrices.

Find the cardinal of $T_n( \mathbb R ) \cap O(n)$ and $T_n( \mathbb R ) \cap SO(n)$.

For the second question, I found that case $n = 2$ is pretty easy, since $\forall M \in SO(2), M$ can be written as $M = \begin{bmatrix} \cos(\theta) & -\sin(\theta) \\ \sin(\theta) & \cos(\theta) \end{bmatrix}$, $\theta \in \mathbb R$.

We can see that $M$ is upper triangular iff $\theta = k \pi$ with $k \in \mathbb Z$.

So that gives us $\text{card}(T_2( \mathbb R ) \cap SO(2)) = 2$.

I have trouble generalizing this result.

Answer 1

Take $M\in T_n(\mathbb{R})\cap O(n)$. Then $M^{-1}=M^T$ (since $M\in O(n)$) and $M^{-1}\in T_n(\mathbb{R})$ (since the inverse of an upper triangular matrix is also upper triangular. So, $M$ and $M^T$ are both upper triangular, which means that $M$ is, in fact, a diagonal matrix. Now, how many diagonal matrices are there in $O(n)$? And in $SO(n)$?