I have ended up with the $5$ equations of $x,y$ and $a$ and $b$. However I'm stuck with the algebra and cannot proceed. Would be grateful if someone could point me to how to solve these.
Here are the equations:
$2x+7a+yb=0$
$2y+bx=0$
$2z+24a=0$
$7x+24z=0$
$xy=6$
On
By AM-GM $$\sqrt{x^2+y^2+z^2}=\sqrt{x^2+\frac{36}{x^2}+\frac{49x^2}{576}}=\sqrt{\frac{625x^2}{576}+\frac{36}{x^2}}\geq\sqrt{2\sqrt{\frac{625x^2}{576}\cdot\frac{36}{x^2}}}=\frac{5}{\sqrt2}.$$ The equality occurs for $$\frac{625x^2}{576}=\frac{36}{x^2}$$ or for example, for $$(x,y,z)=\left(\frac{12}{5},\frac{5}{2},-\frac{7}{10}\right),$$ which says that we got a minimal value.
On
Let $P=(x,y,z)$ be the point with the minimal distance. The normal vectors to the two given surfaces
$$f(x,y,z)=xy-6,\>\>\>\>\>g(x,y,z)=7x+24z$$
are
$$n_f =(f_x,f_y,f_z)=(y,x,0),\>\>\>\>\>n_g=(g_x,g_y,g_z)=(7,0,24)$$
The tangent vector at the intersection of the two surfaces is then $n_f\times n_g = (24x,-24y,-7x)$, which is perpendicular to P. Then, we have
$$P\cdot(n_f\times n_g)=0$$ or,
$$24x^2-24y^2-7xz=0$$
Along with $xy=6,\> 7x+24z=0$, solve for the coordinates of the point P to get
$$x^2=\frac{144}{25},\>\>\>\>\>\>y^2=\frac{25}4,\>\>\>\>\>z^2=\frac{49}{100}$$
Thus, the minimum distance is $$\sqrt{x^2+y^2+z^2}=\frac5{\sqrt2}$$
It is a lot easier to solve this in just one variable.
You must minimise $\frac{36}{x^2}+\frac{625}{576}x^2$. Therefore $$\frac{625}{288}x^4-72=0$$.
So $x=2.4,y=2.5,z=-0.7$.