Orthogonal vector in a plane

Question

I have two vectors $\boldsymbol{a}$ and $\boldsymbol{b}$. Now I want to find a vector $\boldsymbol{v}$ orthogonal to $\boldsymbol{a}$ that is in the plane spanned by $\boldsymbol{a}$ and $\boldsymbol{b}$. Is it ok if I do $\boldsymbol{v} = \boldsymbol{a} \times (\boldsymbol{a} \times \boldsymbol{b})$?

Answer 1

Yes. $\boldsymbol a\times (\boldsymbol a\times \boldsymbol b)$ is perpendicular to both $\boldsymbol a$ and $\boldsymbol a\times \boldsymbol b$. Being perpendicular to $\boldsymbol a\times \boldsymbol b$ means being on the plane generated by $\boldsymbol a$ and $\boldsymbol b$.

See the other answer for a less expensive computation that outputs such a vector.

Answer 2

As @Tal-Botvinnik says, it's perfectly OK in general. There is a failure case, which is when $a$ and $b$ are linearly dependent; then $a \times b = 0$, and you're in trouble.

A computationally slightly cheaper approach is to take $$ n = b - \frac{b \cdot a}{a \cdot a} a. $$

This also works in the case where $a$ and $b$ are vectors in any inner-product space, not just 3-dimensions.

(BTW, it also fails on the same case as the cross-product approach; that's because the answer isn't unique in that case.)