If $\vec{a}\;,\vec{b}$ are $2$ unit vectors and $\vec{c}$ is a vector such that
$\vec{c} = \vec{a}\times \vec{c}+\vec{b}.$ Then $\max$ of $\vec{a}\cdot (\vec{b}\times \vec{c})$ is
Try: Given $\vec{c}=\vec{a}\times \vec{c}+\vec{b}=-(\vec{c}\times \vec{a})+\vec{b}.$
So $\vec{c}\times \vec{a}=\vec{b}-\vec{c}.$
Now taking dot product with $\vec{b}.$
$\vec{b}\cdot \left(\vec{c}\times \vec{a}\right)=\vec{b}\cdot \vec{b}-\vec{b}\cdot \vec{c}.$
So $[\vec{a}\;\vec{b}\;\vec{c}]=1-|\vec{c}|\cos \theta.$
Could some help me how to solve it, Thanks
We have 3 vectors so the problem could be solved in a proper subspace with dimension 3 which contains all the 3 vectors $a,b,c$ . First we solve the problem when $a$ and $b$ are linearly independent i.e. $$a\ne \pm b$$ In this case, we can have the following linear independent basis for solution space $\Bbb R^3$:$$a,b,a\times b$$since $a\times b$ is orthogonal on both $a,b$. Therefore any vector (including $c$) can be expressed as a linear combination of the basis, specially:$$c=k_1a+k_2b+k_3a\times b$$for some $k_1,k_2,k_3\in\Bbb R$. By substituting in the definition ($c=a\times c+b$) we have $$k_1a+k_2b+k_3a\times b=k_2a\times b+k_3a\times(a\times b)+b\\k_1a+k_2b+k_3a\times b=k_2a\times b+k_3(a\cdot b)a-k_3b+b$$which yields to the following system of equations $$\begin{cases}k_1=k_3(a\cdot b)\\k_2=1-k_3\\k_2=k_3\end{cases}$$that has the following answer $$k_1={1\over 2}a\cdot b\\k_2=k_3={1\over 2}$$b substitution we obtain $$c={1\over 2}(a\cdot b)a+{1\over 2}b+{1\over 2}a\times b$$so we have$$a\cdot (b\times c){=a\cdot \left(b\times[{1\over 2}(a\cdot b)a+{1\over 2}b+{1\over 2}a\times b]\right)\\=a\cdot\left({1\over 2}(a\cdot b)b\times a+{1\over 2}a-{1\over 2}b(a\cdot b)\right)\\={1\over 2}\left(1-(a\cdot b)^2\right)}$$This expression is maximum when $a$ and $b$ are orthonormal (which fulfills the assumption of linear independence of $a,b$) and the maximum is $1\over 2$.