Suppose that I want to calculate the closest rotation in $SO(n)$ to an $n \times n$ matrix $M$. It can be shown by geometric arguments that $R = M(M^tM)^{-1/2}$. However, this requires that we can calculate $(M^tM)^{-1/2}$. This is easy when $M^tM$ has a spectral decomposition with non-zero eigenvalues. But what if one of the eigenvalues is $0$?
Edit:
Here's the geometric argument that I mentioned:
Suppose that we are in $\mathbb{R}^{n^2}$. Consider $\mathrm{SO}(n)$ as a subset of this space. We know that the tanget to $\mathrm{SO}(n)$ at $I$ consists of anti-symmetric matrices. We can shift the tangent space to other points by left multiplication. So, given $M$ in the ambient space, I want to project it on $\mathrm{SO}(n)$. On the other hand, we know that anti-symmetric matrices and symmetric matrices are orthogonal complements of each other in the ambient space with respect to the inner product given by $<A,B> = \mathrm{tr}(A^tB)$. Consider the set $\{RA: A\text{ is anti-symmetric. } \}$. For any $A$, $M-R$ must be orthogonal to $RA$:
$$\mathrm{tr}((RA)^t(R-M)) = 0$$ $$\mathrm{tr}(A^tR^tR-A^tR^tM) = 0$$ $$\mathrm{tr}(A^t(I-R^tM)) = 0$$
Hence, $I-R^tM$ is a symmetric matrix. Therefore, $R^tM$ is symmetric and we have $R^tM = S$, or equivalently $M=RS$. Now write the spectral decomposition for $M^tM$. Note that $M^tM = S^tR^tRS = S^2$. Hence, $S = (M^tM)^{1/2}$. Finally, $R = M(M^tM)^{-1/2}$.
You end up with $M^TM=S^TS=S^2$ and $M=RS$. This does not imply $S=(M^TM)^{1/2}$, it could also hold $S=-(M^TM)^{1/2}$.
Now take the singular value decomposition of $M$ $$ M = U\Sigma V^T, $$ which implies the spectral decomposition of $M^TM$: $$ M^TM = V\Sigma^2V^T. $$ Let $W$ be a diagonal matrix with $W^2=\Sigma^2$. Then all matrices $S:=VWV^T$ satisfy $S^2=M^TM$. It remains to compute $R$. The equation $M=RS$ is equivalent to $$ U\Sigma = R VW. $$ Let $i$ be such that the singular value $\sigma_i$ is positive. Then $\sigma_i U_i = w_iR V_i $, i.e., $R$ maps the vector $V_i$ to $\pm U_i$. This specifies $R$ on $\ker M$. If $M$ is not invertible, $R$ has to be completed to be in $SO(n)$. The choice of the square root $W$ is essential to get $\det(R)>0$.
There are many degrees of freedom in the construction: non-uniqueness of the SVD, the construction of $W$ up to even-number of sign-changes, the definition of $R$ on the kernel of $M$.
These considerations seems to compute stationary points only. Hence, the matrices $R$ constructed like this, are candidates. I do not know how to prove optimality for some/all of them.
Uniqueness cannot be expected: If $M=0$ then all matrices of $SO(n)$ are solutions.