If we know an ellipsoid and a point $c$ out side of it, and the point lies on one of the major axises of the ellipsoid. Consider the convex hull of the point $c$ and the ellipsoid, given a point $a$ inside the hull, we want to calculate the distance between $a$ and the boundary. We can by far just consider Euclidean space.
Are there any standard procedure to do this?
Thanks!
Here's my suggested procedure.
First, translate the ellipsoid to origin, and rotate it so that $\vec{c}$ (the apex of the elliptical cone) is on the positive $z$ axis, ellipsoid axes aligned with the coordinate axes, and $\vec{a}$ in the positive $xy$ quadrant. The rest of the calculations are done in the transformed coordinates.
(You can construct the rotation matrix from transposed normalized semi-major axis vectors, negating first row if transformed $\vec{a}$ has negative $x$ coordinate, and negating second row if transformed $\vec{a}$ has negative $y$ coordinate. While these are reflections and not rotations, they do not break the symmetries, and distances and angles will be preserved.)
If $\vec{a}$ has negative $z$, then you consider only the ellipsoid, as the cone part is outside the other half of the ellipsoid and therefore always further out.
Next, find the point $\vec{p}_e$ on the surface of the ellipsoid, closest to point $\vec{a}$. This is a well-known problem; see e.g. David Eberly's Distance from Point to an Ellipse, an Ellipsoid, or a Hyperellipsoid (PDF) at the Geometric Tools site.
Also find the point $\vec{p}_c$ on the elliptical cone, closest to point $\vec{a}$. A quick web search didn't yield a good source, so I guess I'll try to outline my own approach.
Edited on 2016-08-28: The key is to remember that point $\vec{a}$ is perpendicular to the surface at the closest point. So, we can find the closest point(s) on the surface of the cone by finding the points on the cone where the surface is perpendicular to point $\vec{a}$ (i.e., a line through the surface point, parallel to the surface normal, will also go through point $\vec{a}$), and if there is more than one, pick the closest one. Since our elliptical cone has an axis parallel to the $z$ axis, and point $\vec{a}$ is in the positive $xy$ quadrant, we only need to consider the positive (nonnegative) quadrant of the cone, though.
If we parametrize our cone as $$\begin{cases} x(s,t) = (1-t) r_x \cos(2\pi s) \\ y(s,t) = (1-t) r_y \sin(2\pi s) \\ z(s,t) = t r_z \end{cases}, \; 0 \le s \le 1, \; 0 \le t \le 1$$ we can examine the surface tangents (two vectors perpendicular to the surface normal at that point): $$\begin{cases} \frac{d x(s,t)}{d s} = 2 \pi r_x (t - 1) \sin(2 \pi s) \\ \frac{d y(s,t)}{d s} = 2 \pi r_y (1 - t) \cos(2 \pi s) \\ \frac{d z(s,t)}{d s} = 0 \end{cases}, \; \; \begin{cases} \frac{d x(s,t)}{d t} = - r_x \cos(2 \pi s) \\ \frac{d y(s,t)}{d t} = - r_y \sin(2 \pi s) \\ \frac{d z(s,t)}{d t} = r_z \end{cases}$$ If $\vec{a}'$ is a vector from the surface point to $\vec{a}$, then dot product between $\vec{a}'$ and the tangent vectors above will yield zero; the two equations are enough to solve for $s$ and $t$ and find the point(s) on the surface of the cone closest to $\vec{a}$, but it may be quite a lot of work in practice.
Their cross product gives us the surface normal vector $\vec{n}(s,t)$ at point $\left(x(s,t), y(s,t), z(s,t)\right)$: $$\begin{align}\vec{n}(s,t) &= \left( 2 \pi r_y r_z (1 - t) \cos(2 \pi s), \; 2 \pi r_x r_z (1 - t) \sin(2 \pi s), \; 2 \pi r_x r_y (1 - t) \right) \\ &= 2 \pi (1-t) \left ( r_y r_z \cos(2 \pi s), \; r_x r_z \sin(2 \pi s), \; r_x r_y \right )\end{align}$$ In other words, the $t$ parameter affects only the length of the normal vector (which is irrelevant for our purposes here). We can therefore find the $s$ first -- i.e. on the $xy$ plane --, then solve their corresponding $t$'s, and the one that is closest to $\vec{a}$ is the closest on the cone surface.
This is essentially a two-step process: first, you find the point(s) on the ellipse closest to $\vec{a}$ considering only the $xy$ plane. (Note that the axes of the ellipse match the elliptical cone axes at $z=0$; otherwise you ignore the $z$ coordinates for this step.) Next, you draw a line from the apex of the ellipse through that point (at $z=0$), and find the point on that 3D line closest to $\vec{a}$. That is the point on the cone surface. (End of edit on 2016-08-28.)
Next, find point $\vec{q}_c$, which has the same $x$ and $y$ coordinates as $\vec{p}_e$, but is on the surface of the cone instead of the ellipse. Whichever point has the larger $z$ coordinate, is on the convex hull. Let's call that point $\vec{s}_1$.
Also find point $\vec{p}_e$, which has the same $x$ and $y$ coordinates as $\vec{p}_c$, but is on the surface of the ellipsoid, instead of the cone. Whichever point has the larger $z$ coordinate, is on the convex hull. Let's call that point $\vec{s}_2$.
The distance from point $\vec{a}$ to the convex hull is then$$\min\left(\left\lvert\vec{a}-\vec{s}_1\right\rvert, \left\lvert\vec{a}-\vec{s}_2\right\rvert\right)$$i.e., the distance to the closer point on the convex hull.