I would like to compute analytically the distance between a finite cone and a line. My idea so far is to setup the following minimization problem
$\min_{t_1, t_2, c, s} \|L(t_1) - C(t_2, c,s)\|^2$
s.t. $c^2 + s^2 = 1$ and $t_2 \in [0,1]$
where
$L(t) = p_1 + t \cdot v_1$ (parametric line)
$C(t, c, s) = \begin{bmatrix}c & -s & 0\\s & c & 0 \\ 0 & 0 & 1\end{bmatrix}(p_2 + t \cdot v_2)$ (parametric cone by rotating a line)
and $p_1, p_2, v_1, v_2 \in \mathbb{R}^3$
But I struggled to solve the system of equations arising from the first-order optimality conditions of such an optimization problem. Also the minimization problem does not capture the base of the cone.
When the constraint on $t_2$ is not active, (e.g. the nearest point on the cone is on the interior) and the line and cone do not intersect, I expect that the normal vector $n$ of the cone at the nearest point should be orthogonal to the line. That is, $n \cdot v_1 = 0$, with e.g. n = $[\cos \theta, \sin \theta, h]$. After solving for the normal vector, the point on the cone and the point on the line can be recovered. But this doesn't handle intersection, or the case when the nearest point is at any of the boundaries of the cone where the normal vector is not defined (the tip and rim).
This minimization problem can be stated as
$$ d^* = \arg\min_{c,s,\lambda,\mu}||p_1+\lambda \vec v_1-R(c,s)\cdot(p_2+\mu \vec v_2)||\ \ \mbox{s.t.}\ \ s^2+c^2= 1 $$
with
$$ R(c,s) = \left(\begin{array}{ccc}c&-s&0\\s&c&0\\0&0&1\end{array}\right) $$
developing we have
$$ d^2 = ||p_1+\lambda \vec v_1||^2-2(p_1+\lambda \vec v_1)\cdot(R(c,s)\cdot(p_2+\mu \vec v_2))+||R(c,s)\cdot(p_2+\mu \vec v_2)||^2 $$
here
$$ ||p_1+\lambda \vec v_1||^2=||p_1||^2+2\lambda p_1\cdot\vec v_1+\lambda^2||\vec v_1||^2 $$
etc.
Now the lagrangian reads
$$ L(c,s,\lambda,\mu,\nu) = ||p_1+\lambda \vec v_1||^2-2(p_1+\lambda \vec v_1)\cdot(R(c,s)\cdot(p_2+\mu \vec v_2))+||R(c,s)\cdot(p_2+\mu \vec v_2)||^2+\nu(c^2+s^2-1) $$
and solving for $c,s,\lambda,\mu,\nu$
$$ \nabla L = 0 $$
we will obtain the sought solution.