could anyone help me to understand which fatal mistake is there..? I mean, I can understand , that orthogonal matrices can have complex eigenvalues. but then, what fallacy is there in that proof ?
and a related question , can the eigenvalues of an orthogonal matrix be 'zero' !
On
The fallacy is in the fact that the eigenvector has, in general, complex entries. For instance, the matrix $$ Q=\begin{bmatrix} 0 & 1 \\ -1 & 0 \end{bmatrix} $$ is orthogonal, but has no real eigenvalue and no real eigenvector either.
Let's try with the eigenvalue $i$, for which a norm $1$ eigenvector is $x=\left[\begin{smallmatrix}1/\sqrt{2}\\i/\sqrt{2}\end{smallmatrix}\right]$. Then $$ x^TQ^TQx= \begin{bmatrix}i/\sqrt{2}&1/\sqrt{2}\end{bmatrix} \begin{bmatrix}i/\sqrt{2}\\1/\sqrt{2}\end{bmatrix}= 0 $$ and surely not $1$.
Clarification: The "proof" assumed that $x^Tx\neq0$ and hence when we get $(\lambda^2-1)x^Tx=0$ the author of the proof concluded that $\lambda^2=1$.
However, one must remember that the proof is correct when $x$ is a vector with real entries. Otherwise, in general, $x^*x>0$ and not $x^Tx>0$ where $x$ is the eigenvector.
Hence, NO CONCLUSION can be drawn from this "proof". This is where the fallacy lies.
The correct method to proceed would be to consider $Qx=\lambda x$, where $x\neq0$. Then, $(Qx)^*=\bar\lambda x^*$ i.e. $x^*Q^*=\bar\lambda x^*$. Hence we have $x^*Q^*Qx=|\lambda|^2x^*x$.
Now use $Q^*Q=Q^TQ=I$ and the fact that $x^*x>0$, which shows that $|\lambda|^2=1$ and hence $|\lambda|=1$.