Use Lagrange multiplier to find the distance between the point $(3,4,0)$ and the surface of the cone $$ z^2=x^2+y^2 $$
I wrote the equation of the distance: $$f(x,y,z)=(x-3)^2+(y-4)^2+z^2$$
and ended up with the following equations: \begin{align} 2(x-3)&=2x \lambda,\\ 2(y-4)&=\lambda2y,\\2z&= -2z \lambda, \\z^2&= x^2 + y^2. \end{align} Solving the equations I get $x=\frac{3}{2}$ , $y=2$ and $z=\pm \frac{5}{2}$. Can you help me solve this, because I am confused, I don't know if I should solve the case where $z=0$ and $z \neq 0$?
Look at the third equation $2z=-2z\lambda$. If $z\ne 0$ we get $\lambda=-1$, and that quickly leads to your solutions.
We do have to consider the possibility $z=0$. If $z=0$, that tells us nothing about $\lambda$, but it forces $x=y=0$, because of $x^2+y^2=z^2$. Now check whether $x=0$, $y=0$ satisfies your first two equations. It doesn't. So the only remaining candidates are the ones that you found.