If A is an arbitrary 3x3 orthogonal matrix with det(A)=1, then how do I show that the eigenvalues are 1, cos(x)+i sin(x), and cos(x)-i sin(X), where cos(x)=(tr(A)-1)/2.
I was thinking that maybe tr(A) equals det(A), and that gives a value for x, but I'm not sure if thats right
I am assuming your matrix is real-valued (since it would not make sense to talk about orthogonality otherwise).
Since matrix is orthogonal, it is a normal operator -> it can be diagonalized ($\mathbf{M}=\mathbf{\Gamma}^\dagger.\mathbf{\Lambda}.\mathbf{\Gamma}$). Using the fact that for real-valued matrix transpose is the same as conjugate transpose:
$$ \mathbf{M}^T=\mathbf{M}^\dagger=\mathbf{M}^{-1}=\mathbf{\Gamma}^\dagger.\mathbf{\Lambda}^{\dagger}.\mathbf{\Gamma}=\mathbf{\Gamma}^\dagger.\mathbf{\Lambda}^{-1}.\mathbf{\Gamma} $$
Since $\mathbf{\Gamma}$ is invertible, it follows that for matrix with eigenvalues in the diagonal:
$$ \mathbf{\Lambda}^\dagger=\left( \begin{array}\\ \lambda_1^* & 0 & 0\\ 0 & \lambda_2^* & 0\\ 0 & 0 & \lambda_3^*\\ \end{array} \right)= \mathbf{\Lambda}^{-1}=\left( \begin{array}\\ \lambda_1^{-1} & 0 & 0\\ 0 & \lambda_2^{-1} & 0\\ 0 & 0 & \lambda_3^{-1}\\ \end{array} \right) $$
So the eigenvalues of the matrix are all of the form $\lambda_s=\exp\left(i\phi_s\right),\,s=1,2,3$
Next $\mbox{det}\left[\mathbf{M}\right]=\Pi_s \exp\left(i\phi_s\right)=\exp\left(i\left[\phi_1+\phi_2+\phi_3\right]\right)=1$, so
$$ \phi_3=-\phi_1-\phi_2+2\pi n,\quad n=0,\pm1,\dots $$
Next, if matrix is real-valued, the eigenvalues will be solutions to a cubic polynomial equation with real-valued coefficients. Therefore at least one of them will be real-valued, let it be the third one, so $\lambda_3=\pm1$. Also the other two eigenvalues will be complex conjugates of each other. So the only possibility is:
$$ \lambda_1=\exp\left(i\phi\right),\,\lambda_2=\exp\left(-i\phi\right),\,\lambda_3=1 $$
Therefore $\mbox{Tr}\left[\mathbf{M}\right]=2\cos\phi+1$