I want to solve the following matrix equation:
$$ A^TMA=M, where M= \begin{pmatrix}1 & 0& \cdots & 0 & 0\\0 & 1 &\cdots &0 &0\\\vdots & \vdots & \ddots & \vdots & \vdots \\ 0 & 0 & \cdots & 1 & 0 \\ 0 & 0 & \cdots & 0 & -1\end{pmatrix} $$
Is there a closed-form expression for A, or for several matrices that A can be decomposed to?
Moreover, if we not assume A is a square matrix, but a matrix of $m\times n$, then how about $$ A^TM_mA=M_n $$
Here is an infinite class of solutions.
First, let $B$ be any $(n-1) \times (n-1)$ matrix such that $B^TB=I_{n-1}$ and set $$ A=\begin{pmatrix}b_{1,1} & b_{1,2}& \cdots & b_{1,n-1} & 0\\ b_{2,1} & b_{2,2} &\cdots &b_{2,n-1} &0\\\vdots & \vdots & \ddots & \vdots & \vdots \\ b_{n-1,1} & & \cdots & b_{n-1,n-1} & 0 \\ 0 & 0 & \cdots & 0 & -1\end{pmatrix} $$
It is easy to see that $AM=MA$. Moreover, $A^TA=I_n$ since $B^TB=I_{n-1}$.
It follows that $$ A^TMA= A^TAM=I_nM=M \,. $$ Since there are infinitely many choices of $B$, this describes infinitely many solutions.
Morever $\det(A)=-\det(B)$ tells you that picking $B \in O_{n-1}\backslash SO_{n-1}$ gives you $A \in SO_n$.