For example, say
$A= \left ( \begin{matrix} \cos x & -\sin x & 0 \\ \cos y \sin x & \cos x \cos y & -\sin y \\ \sin x \sin y & \sin y \cos x & \cos y \end{matrix} \right)$.
How do i conpute null space of $A-I$?
Since I don't know whether $\cos x$ , the 1-1 entry, is $0$ or not, I cannot simply take elementary operators on this matrix.
Is there any good strategy to find its null space?
(Note that $A$ is just an example. I'm asking how to compute null space of such matrices with entries $\cos x$, $\sin x$, $\cos y$, $\sin y$...)
Your rotation matrix is given by $YX$, where $$ Y=\pmatrix{1\\ &\cos(y) &-\sin(y)\\ &\sin(y) &\cos(y)}, X=\pmatrix{\cos(x) &-\sin(x)\\ \sin(x) &\cos(x)\\ &&1}. $$ $Y$ and $X$ have the quaternion representations \begin{cases} \mathbf{y} = \cos(\tfrac y2) + \mathbf{i}\sin(\tfrac y2),\\ \\ \mathbf{x} = \cos(\tfrac x2) + \mathbf{k}\sin(\tfrac x2) \end{cases} Hence $YX$ is given by the quaternion \begin{align*} \mathbf{y}\mathbf{x} &=\left[\cos(\tfrac y2) + \mathbf{i}\sin(\tfrac y2)\right] \left[\cos(\tfrac x2) + \mathbf{k}\sin(\tfrac x2)\right]\\ &= \cos(\tfrac y2)\cos(\tfrac x2) + \color{red}{ \mathbf{i}\sin(\tfrac y2)\cos(\tfrac x2) - \mathbf{j}\sin(\tfrac y2)\sin(\tfrac x2) + \mathbf{k}\cos(\tfrac y2)\sin(\tfrac x2)}. \end{align*} The axis of rotation is obtained from the nonreal part, i.e. $$ \color{red}{\mathbf{u} = \left( \sin(\tfrac y2)\cos(\tfrac x2), \ -\sin(\tfrac y2)\sin(\tfrac x2), \ \cos(\tfrac y2)\sin(\tfrac x2) \right)} $$ provided that $\mathbf{u}\neq0$. In this case, $YX\mathbf{v}=\mathbf{v}$ if and only if $\mathbf{v}$ is a scalar multiple of $\mathbf{u}$. If $\mathbf{u}=0$, $YX$ is the identity matrix and every nonzero vector is an eigenvector corresponding to the eigenvalue 1.
The above method can be easily extended to tackle the cases where the rotation is specified by Euler angles.