I want to find the unit vector ($v^*$) which is closest to some other unit vector ($v$) with the constraint that $v^*$ lies within some plane ($n$) that goes through the origin.
Written as a constrained optimization problem:
max $v \cdot v^*$
subject to $v^* \cdot n = 0$
In other words, we want to find a unit vector $v^∗$ in the plane with the smallest angle to the given vector $v$, not in it. In 3D the angle will be $π−ϕ$, where $ϕ$ is the (acute) angle between $v$ and the normal line to the plane, and $v^∗$ will be the normalized orthogonal projection of $v$ onto the plane. One can solve this using Lagrange multipliers, but $v^∗\cdot v^∗=1$ should be added as a constraint.
Or we can see it directly from the law of cosines. Let $p$ be the projection of $v$ to the plane, and $u$ be any vector in the plane of the same length, $\|q\|=\|p\|$. By the law of cosines: $$\|v-q\|^2=\|v\|^2 + \|q\|^2 - 2\|v\|\|q\|\cos\angle(v,u)\\= 1 + \|p\|^2 - 2\|p\|\cos\angle(v,q). $$ The only term on the right that still depends on $q$ is $\cos\angle(v,q)$. Since we want minimal $\angle(v,q)$, and cosine is monotone decreasing on acute angles, we want to make the cosine as large as possible, and hence the right hand side as small as possible. But that is achieved exactly when $q=p$, by the best approximation property of orthogonal projections. It is also geometrically intuitive that rotating $q$ away from $p$ increases its angle with $v$.