I have this question :
Find the point that is at distance $1$ from $(0,0,0)$ and at distance $3$ from $(1,2,3)$ that is closest to $(5,-2,4)$.
Here is my failed attempt.
I used Lagrange multipliers and the fact that the determinant of matrix is 0 when we have non trivial solution. the quad equation went bad
You have done great so far. The only thing left is to substitute $x=\dfrac{57z-33}{17}$ and $y=\dfrac{42-54z}{17}$ into $x^2+y^2+z^2=1$. In other words, you have $$\frac{2(3227z^2-4149z+1282)}{289}=0\,.$$ From this horrendous quadratic equation, you solve it using the famous quadratic formula, but not without pain, and obtain $$z=\frac{4149\pm17\sqrt{2305}}{6454}\,.$$ That is, $$x=\frac{3(461\pm19\sqrt{2305})}{6454}$$ and $$y=\frac{3(461\mp9\sqrt{2305})}{3227}\,.$$ Therefore, the distances to $(5,-2,4)$ from these optimizing points are $$\sqrt{\frac{283\mp\sqrt{2305}}{7}}\,.$$ Thus, the minimizing point is $$\begin{align}(x,y,z)&={\small\left(\frac{3(461+19\sqrt{2305})}{6454},\frac{3(461-9\sqrt{2305})}{3227},\frac{4149+17\sqrt{2305}}{6454}\right)}\\&\approx(0.638,0.027,0.769)\,,\end{align}$$ yielding the minimum distance $\sqrt{\dfrac{283-\sqrt{2305}}{7}}\approx5.794$. On the other hand, the maximizing point is $$\begin{align}(x,y,z)&={\small\left(\frac{3(461-19\sqrt{2305})}{6454},\frac{3(461+9\sqrt{2305})}{3227},\frac{4149-17\sqrt{2305}}{6454}\right)}\\&\approx(-0.210,0.830,0.516)\,,\end{align}$$ yielding the maximum distance $\sqrt{\dfrac{283+\sqrt{2305}}{7}}\approx6.877$.