The sets of orthogonal matrices with determinant +1 ("rotations") and -1 ("reflections"), respectively, make up distinct sets, so that if we label the sets $R$ and $M$, minimizing
$$ \min_{O\in R}\|O^*-O\|,\quad O^*\in M $$ will generally lead to a poor approximation, as the sets are disjoint.
Now assume that $X$ is column-orthogonal and rectangular. $OX$ will likewise be column-orthogonal for both $O\in R$ and $O\in M$. The idea of a determinant is no longer applicable (as $X$ is rectangular), so for a column-orthogonal $X^*$, is it possible to simultaneously solve $$ \min_{O\in M}\|X^*-OX_M\|, $$ and $$ \min_{O\in R}\|X^*-OX_R\| $$ for fixed column-orthogonal matrices $X_M$ and $X_R$? Can an arbitrary column-orthogonal matrix $Y^*$ be written as the reflection/rotation of an arbitrary column-orthogonal matrix, or do we need to impose conditions on $X$? Are the same sets of column-orthogonal matrices reachable?