Given the 'xyz' rotation matrix ($\theta$ - roll, $\phi$ - pitch, $\psi$ - yaw):
$$R = \begin{bmatrix} c_\phi c_\psi & -c_\phi s_\psi & s_\phi \\ c_\theta s_\psi + c_\psi s_\phi s_\theta & c_\psi c_\theta - s_\phi s_\psi s_\theta & -c_\phi s_\theta \\ s_\psi s_\theta - c_\psi c_\theta s_\phi & c_\psi s_\theta + c_\theta s_\phi s_\psi & c_\phi c_\theta \end{bmatrix}$$
where $c_x = \cos x$ and $s_x = \sin x$, we can say that the singularity has occured if $|R(1,3)|-1=0$. In my textbook, the angles are calculated as follows: $$\theta = 0,$$ $$\phi=\arcsin{R(1,3)}$$ and there are two cases for $\psi$:
- If $R(1,3)=\sin\phi=-1$, $\phi=-\pi/2$: $$\psi=-\arctan2(R(2,1), R(3,1))$$
- If $R(1,3)=\sin\phi=1$, $\phi=\pi/2$: $$\psi=\arctan2(R(3,2), R(2,2))$$
What confuses me is the first case. Why is there a minus sign before $\arctan2$?
Your textbook appears to have a typo. That minus sign should not be there.
Firstly, we should check the expression for $R$ you have is correct. It is, assuming our conventions are equivalent to implementing rotations in the following order: $$ R = \begin{pmatrix} 1 & 0 & 0 \\ 0 & c_{\theta } & -s_{\theta} \\ 0 & s_{\theta } & c_{\theta} \end{pmatrix} \cdot \begin{pmatrix} c_{\phi } & 0 & s_{\phi } \\ 0 & 1 & 0 \\ -s_{\phi } & 0 & c_{\phi } \\ \end{pmatrix} \cdot \begin{pmatrix} c_{\psi } & -s_{\psi } & 0 \\ s_{\psi } & c_{\psi } & 0 \\ 0 & 0 & 1 \\ \end{pmatrix} $$ Now clearly, in the special case that $\phi = -\pi/2$ and $\theta = 0$, we obtain $$ R = \begin{pmatrix} 0 & 0 & -1 \\ s_\psi & c_\psi & 0 \\ c_\psi & -s_\psi & 0 \end{pmatrix} $$ Clearly, the correct choice of $\psi$ has sine given by $R_{2,1}$ and cosine given by $R_{3,1}$. Thus $$\psi = \operatorname{atan2}(\sin \psi, \cos \psi) = \operatorname{atan2}(R_{2,1},R_{3,1})$$