Given two orthogonal matrices ${\bf{M}} \in {{\Bbb{R}}^{m \times n}}$ and ${\bf{N}} \in {{\Bbb{R}}^{m \times n}}$, there is an orthogonal transformation matrix ${\bf{T}} \in {{\Bbb{R}}^{n \times n}}$ which closely maps ${\bf{M}}$ and ${\bf{N}}$:
$$\begin{array}{ll} \text{minimize} & {\left\| {{\bf{M}} - {\bf{NT}}} \right\|_F}\\ \text{subject to} & {{\bf{T}}^T}{\bf{T}} = {\bf{I}}\end{array}$$
Is this a convex optimization problem?
It is not convex because the feasible set (the set of matrices $T$ satisfying $T^TT = I$) is not a convex set.
For instance, $I$ and $-I$ are both orthogonal matrices, but their non-trivial convex combinations (that is, every $T = (1 - \theta) I + \theta(-I)$ with $\theta \in (0,1)$) fail to be orthogonal.