Matrices are over the field of complex numbers, and $X^t$ means transpose of a matrix $X$.
Consider the group action of $O(4)=\{P\mid PP^t=I\}$ on $SK(4)=\{M\mid M^t=-M\}$ by $(P,M) \rightarrow PMP^t$. Does anyone know what is the stabilizer of a general $M$?
Thank you!
This is only a paritial answer. The major difficulty of this question is that for every eigenvector $v$ corresponding to a nonzero eigenvalue of $M$, we have $v^Tv=0$. This directly conflicts with the conjugation of the form $PMP^T$, where $PP^T=I$. In other words, unlike the real case or the unitary equivalence case, we cannot directly use the eigenvectors of $M$ as columns of $P$ to simplify the structure of $M$.
Yet, when $M$ is diagonalizable and has nonzero distinct eigenvalues, the problem is easily solvable. As $M$ is skew symmetric, all its nonzero eigenvalues must have their negative counterparts. So, suppose the eigenvalues of $M$ are $\lambda,-\lambda,\mu,-\mu$, where $\lambda,\mu\not=0$ and $\lambda\not=\mu$. Let $v_1, v_2, v_3, v_4$ be the four corresponding unit eigenvectors. Note that when $x,y$ are eigenvectors of $M$ that correspond to some two eigenvalues $p,q$ with $p\not=-q$ and $p,q\not=0$, we have $$px^Ty = (y^Tpx)^T = (y^TMx)^T = -x^TMy = -qx^Ty$$ and hence $x^Ty=0$. Therefore, in our case, for $k\le j$, we have $v_k^Tv_j=0$ whenever $(k,j)\not=(1,2),(3,4)$.
Note that $v_1^Tv_2\not=0$ (and similarly $v_3^Tv_4\not=0$), otherwise we would have $\langle v_j,\bar{v}_1\rangle=0$ for $j=1,\ldots,4$, which is impossible. Therefore, if we define $c^2=1/(2v_1^Tv_2),\,w_1=c(v_1+v_2)$ and $w_2=ic(v_1-v_2)$ (here $i=\sqrt{-1}$) and define $w_3,w_4$ with $v_3,v_4$ analogously, it can be verified that $w_k^Tw_j=\delta_{kj}$, i.e. the matrix $W$ containing the $w_i$s as columns is complex orthogonal ($W^TW=I_4$). Hence $$ W^TMW = \begin{pmatrix}i\lambda J_2\\&i\mu J_2\end{pmatrix}; \ J_2 = \begin{pmatrix}0&1\\-1&0\end{pmatrix}. $$ The stabilizers of $J_2$ are "rotation matrices" of the form $R(z)=\begin{pmatrix}\cos z&\sin z\\ -\sin z&\cos z\end{pmatrix}$, where $z\in\mathbb{C}$. Therefore the stabilizers of $M$ are given by $P=W\left(R(z_1)\oplus R(z_2)\right)W^T$.
The other cases, such as $M$ has zero eigenvalues, or some eigenvalues of $M$ have geometric multiplicity $>1$, etc., are more difficult to handle. I will try to cook up something if I can and if I have time.