Given a vertex with coordinates $\mathbf{x}_0\!=\!(x_0,y_0,z_0)$, I'd like to compute the coordinates of the nearest point to $\mathbf{x}_0$ on the paraboloid whose equation is
\begin{align} z = P(x,y) = c_0 + c_1x + c_2y + c_3xy + c_4x^2 + c_5y^2 \end{align}
How do I analytically minimise the squared-distance from the point to the surface? I'd prefer to not solve this numerically.
On
Below are the steps to find the nearest point on the paraboloid:
1) Calculate the normal vector to the paraboloid at the point $(x,y,z)$ $$\vec{n} = (z_x', z_y', -1) = (c_1+c_3y+2c_4x,\> c_2+c_3x+2c_5y,\>-1)\tag 1$$
2) The nearest point on the paraboloid is on the line parallel to $\vec{n}$ and passing through $(x_0,y_0,z_0)$. The equation of the line is parametrized as $(x_0,y_0,z_0)+t\vec{n}$, or,
$$x=x_0+t(c_1+c_3y+2c_4x)$$ $$y=y_0+t(c_2+c_3x+2c_5y)$$ $$z=z_0-t$$
3) Then, the nearest point is just the intersection between the line and the paraboloid. Solve for $t$ from the system of the equations of the line and the paraboloid.
4) Plug the solution of $t$ into the line equations above to obtain the nearest point coordinates $(x_n,y_n,z_n)$.
Hint: Perhaps you need to consider to minimize $$(x-x_0)^2+(y-y_0)^2 +z-z_0)^2,$$ subject to $$c_1x + c_2y + c_3xy + c_4x^2 + c_5y^2-z=-c_0.$$
For, ones takes the gradient of both relations and makes them proportional, that is $$[2(x-x_0),2(y-y_0),2(z-z_0)]=\lambda [c_1+c_3y+2c_4x,c_2+c_3x+2c_5y,-1]$$ from which you get three equations $$2(x-x_0)=\lambda (c_1+c_3y+2c_4x),$$ $$2(y-y_0)=\lambda (c_2+c_3x+2c_5y),$$ $$2(z-z_0)=-\lambda.$$
Now a strategy is to solve for $\lambda$ in each and from this, the resulting relations, will determine how $x,y,z$ dependent among them.
If this isn't enough to go through let me know.