Unit vector perpendicular to plane containing $\vec{A} =4\hat{i}−\hat{j}−\hat{k}$ and $\vec{B} =4\hat{i}+\hat{j}−4\hat{k}$

Question

I know that one solution is to find the cross product of $\vec{A}$ and $\vec{B}$ and then find the unit vector in that direction.

But why can't we do it in this way:

Let the vector $\vec{U} = x\hat{i}+ y\hat{j}+ z\hat{k}$ be perpendicular to the plane containing $\vec{A}$ and $\vec{B}$. Then $\vec{U}$ must be perpendicular to both $\vec{A}$ and $\vec{B}$ and also to the vector $\vec{A}+\vec{B}$. Therefore:

$$\vec{A}\cdot\vec{U}=0$$ $$\vec{B}\cdot\vec{U}=0$$ $$\big(\vec{A}+\vec{B}\big)\cdot \vec{U}=0$$ So I have three equations with three unknowns but when I solve it I don't get the answer.

Answer 1

We obtain: $$4x-y-z=0$$ and $$4x+y-4z=0,$$ which after summing gives $$8x=5z.$$ We see that $x=5t,$ $z=8t$ and $y=12t,$ which gives $$\frac{1}{\sqrt{5^2t^2+12^2t^2+8^2t^2}}(5ti+12tj+8tk)$$ or $$\pm\frac{1}{\sqrt{233}}(5i+12j+8k)$$

Answer 2

Your third equation is redundant. It adds nothing new because of the distributive law of dot product. What I mean is that if $A.U=0$ and $B.U=0$ then surely it is always true that $A.U+B.U=(A+B).U=0$ holds and it doesn't give you a new independent linear equation.

Therefore, the system you will get requires $3$ independent equations to find the unknowns, but in reality you have provided only $2$ independent equations.

Important Remark:

It is worthy to add that your system, as it has been given in your question, is unsolvable and has infinitely many solutions. However, if you add the new condition $U.U=1$, which is equivalent to saying that $U$ has to be normal, then the system does have two solutions.

The part that Mark Rozenberg divides by $$\frac{1}{\sqrt{5^2t^2+12^2t^2+8^2t^2}}$$ is when he is using the third condition which you hadn't considered originally. But notice that he isn't solving a linear system anymore.

In general, If $A=(a,b,c)$ and $B=(d,e,f)$ then in general you'll have to solve a polynomial system of equations that involves second degree polynomials as follows:

$$ax+by+cz=0$$ $$dx+ey+fz=0$$ $$x^2+y^2+z^2=1$$

The cross product gives a solution to this sytem. Notice that because the degree of this system is $2$, you will always get $2$ solutions. The Right Hand Rule is an agreement among scientists that helps us pick one solution of the system in a consistent manner.