The answer is $\mathbf 0$ but I don't understand how to get to this answer.
If the answer is $\mathbf 0$ then that means $(\mathbf{a}-\mathbf{b})$ is perpendicular to $(\mathbf{a}\times\mathbf{b})$, can someone please explain to me why $(\mathbf{a}-\mathbf{b})$ is perpendicular to $(\mathbf{a}\times\mathbf{b})$
In fact there is an easy reason why this must be true: $a-b$ lies in the plane of $a,b$, while $a \times b$ is perpendicular to that whole plane.