Given a point $\mathbf{a} \in \mathbb{R}^{3}$ and a unit vector $\boldsymbol{n} \in \mathbb{R}^{3}$. A unique plan $P$ goes through $\mathbf{a}$ and possesses a normal $\boldsymbol{n}$. A surface $S$ is parametrized by a function $\boldsymbol{f}(x,y,z):\mathbb{R}^{3}\rightarrow\mathbb{R}^{3}$.
Statement of the problem
I would like to find the point of the surface $S$ that is the closest from $\mathbf{a}$ and that is in the plane $P$. To solve for this, the objective function to minimize is: \begin{equation} \boldsymbol{g}(x,y,z,\lambda)=\left\Vert \boldsymbol{a}-\boldsymbol{f}\right\Vert -\lambda\boldsymbol{c} \end{equation} with $\boldsymbol{c}(x,y,z)=(\boldsymbol{a}-\boldsymbol{f})^{T}\boldsymbol{n}$ the constraint that forces the point of the surface $S$ to be in the plane $P$. I solve this optimization problem numerically using in general a Newton-Raphsson scheme.
If there is an intersection between the plane $P$ and the surface $S$, I can converge to an extremum of the $\left\Vert \boldsymbol{a}-\boldsymbol{f}\right\Vert $ as long as my first guess is sufficiently good.
However, if there is no intersection, the scheme diverges because the constraint in $\boldsymbol{c}$ cannot be satisfied.
Is there any alternative procedure apart from the use of Newton-Raphsson with Lagrange Multipliers that would allow me to know that the solution I am looking for cannot exist because there are no point where the constraint is respected ?
Put more simply, what is the method of choice to employ when one is not sure that a solution of the optimization problem respecting the constraints exists ?
As a side note, I think what you meant to write is that the surface $S$ is defined as the zero set of a function $f:\mathbb{R}^3\to\mathbb{R}$ (i.e. the set of points $(x,y,z)$ such that $f(x,y,z)=0$). Similarly, you can write the plane $P$ as the zero set of some function: $g(x,y,z)=0$. In fact, you even know that $g$ will be of the form $g(x,y,z) = \vec{n}\cdot (x,y,z)- \vec{n}\cdot \vec{a}$.
Now, in words, what you are trying to do is minimize the distance to the point $\vec{a}\in \mathbb{R}^3$, subject to the constraint of being on the surface $S$ and on the plane $P$. Can you think of an objective function that sends $(x,y,z)$ to the distance from $(x,y,z)$ to $\vec{a}$? Hint: it might be easier to minimize the square of this function, which is perfectly fine since the map $\mathbb{R}_{\geq0}\to\mathbb{R}$, $x\mapsto x^2$ is monotone.
As I mentioned above, the constraints are that you would like your point to be on the surface $S$ and on the plane $P$. Thus, your constraint equations should be $f(x,y,z)=0$ and $g(x,y,z)=0$. Now you're all set to use Lagrange multipliers!