Orthogonal $M$ such that $M^TAM$ is upper triangular?

Question

We triangularise $$A=\begin{bmatrix}2&1&1\\-20&-5&-10\\16&3&8 \end{bmatrix}$$

into $U=S^{-1}AS$ with $$S=\begin{bmatrix}-1&0&0\\0&-1&0\\2&1&1 \end{bmatrix} \ \ \text{and} \ \ U=\begin{bmatrix}0&0&-1\\0&5&10\\0&0&0 \end{bmatrix}$$

where the first two columns of $S$ are eigenvectors of $A$.

The question is if there is an orthogonal $M$, such that $M^TAM$ is upper triangular?

I think no since $(-1,0,2)^T$ and $(0,-1,1)^T$ aren't orthogonal but all the eigenvectors of $A$ would need to show up in such an $M$. Does it make sense? If yes, how does one prove it?

Answer 1

The answer is yes. It's Schur's decomposition for real matrices. It is indeed OK to use Schur here because all of $A$'s eigenvalues are real.

To find an orthogonal matrix which triangularizes $A$, find the $QR$ decomposition of $S$. It will yield an orthogonal matrix $Q$ and an upper triangular matrix $R$ such that $S=QR.$

Then $S^{-1}=R^{-1}Q^{-1}$ and $A=SUS^{-1}=QRUR^{-1}Q^{-1}$. Since $R^{-1}$ is an upper triangular matrix and the product of upper triangular matrices is an upper triangular matrix, the desired decomposition has been gotten.