Given a matrix $M \in \mathbb{R}^{n \times m}$ with $n > m$ and an arbitrary vector $v \in \mathbb{R}^n$, I am looking for an analytical solution for the orthogonal matrix $R \in \mathbb{R}^{n \times m}$ with $R^T R = I_m$ that minimizes $$ \lVert R - M \rVert_{F}^2 $$ subject to the hard constraint $$ R^T v = 0. $$ In other words, I'm looking for the closest orthogonal matrix $R$ to $M$ whose columns are perpendicular to $v$.
It seems one way to get an orthogonal $R$ from $M$ satisfying this constraint would be to perform Gram-Schmidt with $v$ as the first basis vector and the columns of $M$ as the others. However, this is perhaps not the minimal solution satisfying the constriant.
$ \def\R#1{{\mathbb R}^{#1}} \def\k{\otimes} \def\h{\odot} \def\o{{\tt1}} \def\BR#1{\Big(#1\Big)} \def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\trace#1{\op{Tr}\LR{#1}} \def\frob#1{\left\| #1 \right\|_F} \def\qiq{\quad\implies\quad} \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\c#1{\color{red}{#1}} \def\CLR#1{\c{\LR{#1}}} \def\fracLR#1#2{\LR{\frac{#1}{#2}}} \def\gradLR#1#2{\LR{\grad{#1}{#2}}} $Use the vector $v$ and an unconstrained matrix $A\in\R{n\times m}\,$ to construct $\,B$ $$\eqalign{ B &= \LR{I-\frac{vv^T}{v^Tv}}A \qiq v^TB = 0 \qquad \qquad \quad }$$ Then use $B$ to construct $\,R$ $$\eqalign{ R &= B\LR{B^TB}^{-1/2} \quad\qiq v^TR=0,\quad R^TR = I }$$ Minimizing the objective function is then an unconstrained problem to find $A$.