I have two objects:
- A line $L$ defined by one of its points $l_0$ and a direction vector $d$. (I.e. $l_0 + t \cdot d$)
- An infinite cone $C$ defined by its apex $a$, a direction $e$ and a half-angle $\alpha < \frac{\pi}{2} $.
(Geogebra)
I want to find the point on the line that is closest to the cone. In the case that the two objects intersect (in more than one point), I would expect the algorithm to return the point that's furthest "inside" the cone.
So far I failed to figure this out myself. Approaches I thought about/tried:
- The point closest to the cone's axis is not necessarily the one closest to the cone itself. I found out by finding a counter-example.
- I read about ways to calculate the intersection(s) between a line and a cone, but that doesn't help me in the case that there is no intersection: then I have no clue.
- I thought about finding a formula for cone-point-distance and then substitute my line equation for the point, finding a minimum in that function in regards to $t$. But I haven't found anything that looks like I could analytically find a minimum.
With this, I don't know how to proceed, hence this question. Any help is greatly appreciated :)


animated geogebra figure here.
Fig. 1: Initial cone (C) in violet. "Enlarged cone" (C') in pale blue, tangent to given line (L). We have chosen to take point $B$ at the same altitude as apex $A$. The shortest distance point $S$ on $(L)$ is projected as $S'$ on $z$ axis.
The equation of the cone is
$$x^2+y^2=t^2(z-a)^2 \ \ \text{where} \ t:=\tan \alpha \tag{1}$$
The issue is equivalent to find a smallest distance segment between the given line and a certain generatrix $AS''$ of cone $(C)$. It is known that this smallest distance segment is both orthogonal to the line and the generatrix ; if one considers now any cone $(C')$ with the same apex and axis as the initial cone, with half-aperture angle $\alpha'$, the shortest distance segment from line $(L)$ to cone $(C')$ will be contained in the same plane.
The generic equation of such cones is evidently :
$$x^2+y^2=t'^2(z-a)^2 \ \ \text{where} \ t':=\tan (\alpha') \tag{2}$$
Besides, the parametric equations of the line can be taken in this way :
$$\begin{cases}x&=& x_0 + \lambda u \\ y&=& y_0 + \lambda v\\ z&=& a + \lambda w \end{cases}\tag{3}$$
Indeed, one can assume that the origin point on the line has the same ordinate as the apex of the cone (the only case where this is not possible is for horizontal lines ; a case deserving a - simple - separate treatment).
Now, consider among all cones $(C')$ the one which allows to transform the initial problem where the line doesn't cross $(C)$ into an equivalent tangency issue for cone $(C')$ : we are going to look for a value of $t'$ such that the generic point of the line (equations (3)) is a double point of $(C')$.
Let us express that the point $M$ belongs to the cone by plugging equations (3) into (2) :
$$(x_0+\lambda u)^2+(y_0+\lambda v)^2=t'^2(\lambda w)^2 $$
giving rise to the quadratic equation in $\lambda$:
$$\lambda^2(u^2+v^2-t'^2w^2)+2 \lambda(x_0u+y_0v)+(x_0^2+y_0^2)=0\tag{4}$$
A double root means annihilation of the discriminant of (4) :
$$(x_0u+y_0v)^2-(x_0^2+y_0^2)(u^2+v^2-t'^2w^2)=0,$$
a condition giving the looked-for value of $t'$ :
$$t'^2=\frac{1}{w^2}\left(u^2+v^2-\frac{(x_0u+y_0v)^2}{x_0^2+y_0^2}\right).\tag{5}$$
The double root condition gives the unique solution for (4) :
$$\lambda_S=- \frac{x_0u+y_0v}{u^2+v^2-t'^2w^2}=-\frac{x_0^2+y_0^2}{x_0u+y_0v}\tag{6}$$
(by taking (5) into account). It remains to plug this value of $\lambda$ into (4) to get the coordinates of the shortest distance point $S$ on the line (see figure).
Important remark : As asked by Hosam H, the shortest distance from $S$ to the given cone is obtained plainly by computing
$$d=AS \sin(\alpha'-\alpha)$$
where the half-aperture angle of the enlarged cone is computed in this way :
$$\alpha'=\arccos\left(\frac{\vec{AS}.\vec{AO}}{\|\vec{AS}\|\|\vec{AO}\|}\right)$$
Remark : as pointed out by Lukas, there is a case where this method is not applicable : when (4) is not a quadratic equation, i.e. when its leading term zero, i.e., when
$$(u^2+v^2-t'^2w^2)=0 \ \iff \ \text{using (5) : } \ x_0u+y_0v=0$$