Determine the value of $\lambda$ such that the vectors $$5\vec{a}+6\vec{b}+7\vec{c},7\vec{a}+\lambda\vec{b}+9\vec{c},3\vec{a}+20\vec{b}+5\vec{c}$$ are coplanar given that $\vec{a},\vec{b},\vec{c}$ are non coplanar.
For the vectors to be coplanar,there must exist scalars $x_1,x_2,x_3$ not all $0$ such that their linear combination is $0$. Then after rearranging,the condition we get is $$(5x_1+7x_2+3x_3)\vec{a}+(6x_1+\lambda x_2+20x_3)\vec{b}+(7x_1+9x_2+5x_3)\vec{c}=0$$ Since $\vec{a},\vec{b},\vec{c}$ are non coplanar,hence each of the coefficients must be $0$.
But here,we are getting $3$ equations for $4$ variables(the fourth one being $\lambda$). So,how can $\lambda$ be determined? Can I assign any one of $x_1,x_2,x_3$ any value I like,for example can I make $x_1$ or $x_3$ $0$ or $1$ or any other number? I am really confused.
There's another approach which is imposing $$\mathbf u\cdot(\mathbf v\wedge\mathbf w)=0,$$ where $$\left\{\begin{align} &\mathbf u=5\mathbf a+6\mathbf b+7\mathbf c\\ &\mathbf v=7\mathbf a+\lambda\mathbf b+9\mathbf c\\ &\mathbf w=3\mathbf a+20\mathbf b+5\mathbf c \end{align}\right.$$ Let us now substitute and proceed: $$\begin{align} (5\mathbf a+6\mathbf b+7\mathbf c)\cdot((7\mathbf a+\lambda\mathbf b+9\mathbf c)\wedge(3\mathbf a+20\mathbf b+5\mathbf c)) &\overset{(1)}{=}(5\mathbf a+6\mathbf b+7\mathbf c)\cdot((140-3\lambda)\mathbf a\wedge\mathbf b+8\mathbf a\wedge\mathbf c+(5\lambda-180)\mathbf b\wedge\mathbf c)\\ &\overset{(2)}{=}7(140-3\lambda)\mathbf c\cdot(\mathbf a\wedge\mathbf b)+48\mathbf b\cdot(\mathbf a\wedge\mathbf c)+5(5\lambda-180)\mathbf a\cdot(\mathbf b\wedge\mathbf c)\\ &\overset{(3)}{=}[7(140-3\lambda)-48+5(5\lambda-180)]\underbrace{\mathbf a\cdot(\mathbf b\wedge\mathbf c)}_{\neq 0\, (4)}\overset{!}{=}0\\ &\iff 980-21\lambda-48+25\lambda-900=0\\ &\implies 32+4\lambda=0\implies \lambda=-8 \end{align}$$ Where at $(1)$ we used the fact that $\mathbf v_i\wedge\mathbf v_j=0$ if $i=j$, and $\mathbf v_i\wedge\mathbf v_j=-\mathbf v_j\wedge\mathbf v_i$; at $(2)$ that $\mathbf v_i\cdot(\mathbf v_j\wedge \mathbf v_k)=0$ if $i=j$ or $j=k$; at $(3)$ that every time you interchange places of two near vectors $\mathbf v_i\cdot(\mathbf v_j\wedge \mathbf v_k)$ changes its sign, and that at $(4)$ it is told that $\mathbf a$, $\mathbf b$, $\mathbf c$ aren't coplanar so $\mathbf a\cdot(\mathbf b\wedge\mathbf c)\neq 0$.