A vector $\bf P$ is given by $\mathbf{P}= m\mathbf{a}+n\mathbf{b}+ q(\mathbf{a}\times\mathbf{b})$ where $\bf a$ and $\bf b$ are unit vectors inclined to each other at an angle of $60^\circ$.
(a) Show that $|\mathbf P|^2 = m^2+n^2+ mn+ \frac34q^2$.
(b) Express $\bf P.a$ and $\bf P.b$ in terms of $m, n, q$.
(c) Find two sets of values for $m, n, q$ such that $\bf P$ is a unit vector at right angles to $\bf a$ and making an angle of $45^\circ$ with $\bf b$.
On
Call $q(\mathbf a\times\mathbf b)=\mathbf v,m\mathbf a+n\mathbf b=\bf w$. Then clearly $\bf v,w$ are orthogonal since the former is perpendicular to the plane containing $\bf a,b$ while the latter lies in that very plane.
$(a)~|\mathbf P|^2=\mathbf P.\mathbf P=(\mathbf v+\mathbf w).(\mathbf v+\mathbf w)=\mathbf v.\mathbf v+\mathbf w.\mathbf w$, where $\mathbf{v.v}=q^2|\mathbf{a\times b}|^2=q^2\sin^2(60^\circ)$ and $\mathbf{w.w}=m^2+n^2+2mn(\mathbf{a.b})$.
$(b)~\mathbf{P.a}=\mathbf{v.a}+\mathbf{w.a}=\mathbf0+m+n(\mathbf{a.b})$. Similarly try $\mathbf{P.b}$.
$(c)~\bf P$ is orthogonal to $\bf a$, so $\mathbf{P.a}=m+n/2=0$. $\bf P$ is a unit vector inclined at $45^\circ$ to $\bf b$, so $\mathbf{P.b}=n+m/2=\cos45^\circ=\frac1{\sqrt2}$. These equations give $m=-\frac{\sqrt2}3,n=\frac{2\sqrt2}3$.
Now find $q$ by equating $|\mathbf P|^2$ to $1$ in $(a)$.
$$\vec P= m \hat a+ n \hat b+q (\hat a \times \hat b)$$ $$|\vec P|^2=\vec P. \vec P=m^2 (\hat a. \hat a)+mn (\hat b. \hat a)+q m(\hat a \times \hat b). \hat a + ma(\hat a. \hat b)+n^2 (\hat b. \hat b)+qn (\hat a \times \hat b).\hat b+ mq \hat a.(\hat a.\times \hat b) +mq \hat b.(\hat a\times \hat b)+q^2(\hat a \times \hat b).(\hat a\times \hat b)$$ We have $\hat a. \hat a=1, \hat a. \hat b= \cos \theta, (\hat a \times b). \hat a=[\hat a, \hat, a, \hat b]=0$, $(\hat a \times \hat b). (\hat a \times \hat b)=|(\hat a \times \hat b)|^2= \sin ^2 \theta $ So $$|\vec P|^2= m^2+n^2+2mn \cos\theta+q^2 \sin^2 \theta$$ $\theta$ is given to be $60^0$, then $$|\vec P|^2=m^2+n^2+ mn+ \frac{3}{4}q^2.$$