I need to find the point on the intersection of $a_0+a_1x+a_2y+a_3z=0$ and $b_0+b_1x+b_2y+b_3z=0$ which is nearest to the origin.
Here is how I approached. I would like to check if my logic is correct.
Let $f(x,y,z)=x^2+y^2+z^2$. Then the determinant of the matrix $\pmatrix{x&y&z \\ a_1&a_2&a_3 \\ b_1& b_2 & b_3}$ must be zero. Therefore, if $A=a_2b_3-a_3b_2$, $B=a_1b_3-a_3b_1$, $C=a_1b_2-b_1a_2$, I get $Ax-By+Cz=0$.
Finally, considering the two constraints, I conclude that the desired point is the solution of the linear equation $\pmatrix{a_1&a_2&a_3 \\ b_1&b_2&b_3 \\ A& -B & C} \pmatrix{x\\y\\z}=\pmatrix{-a_0\\-b_0\\0}$.
In order to have a unique solution, I think $A^2+B^2+C^2$ should not be zero.
If $A^2+B^2+C^2=0$ then the vectors $(a_1, a_2, a_3)$ and $(b_1, b_2, b_3)$ are linearly dependent. Therefore, the two planes are either equal or have no intersection; if the two planes are equal, I am left with just one constraint and use the Lagrange multiplier method with one constraint to solve the question.
This is my solution. Could anyone please tell me if it is correct?
The equation
$$ \det\pmatrix{x&y&z \\ a_1&a_2&a_3 \\ b_1& b_2 & b_3}=0 $$
represents a plane orthogonal to $\vec n_1 = (a_1,a_2,a_3)$ and $\vec n_2 = (b_1, b_2, b_3)$ passing by the origin of coordinates. Calling it as $c_1 x+ c_2 y + c_3 z = 0$.
The linear system
$$ \pmatrix{a_1&a_2&a_3 \\ b_1&b_2&b_3 \\ c_1& c_2 & c_3} \pmatrix{x\\y\\z}=\pmatrix{-a_0\\-b_0\\0} $$
according to your argument, gives the sough point.