Suppose you have the cone with its vertex at the origin given by
$ r^T Q r = 0 \tag{1}$
where $r=[x,y,z]^T $, and $Q$ is a $3 \times 3$ symmetric indefinite matrix.
And you have the sphere centered at $C$ with a radius of $a$, whose equation is
$ (r - C)^T (r - C) = a^2 \tag{2}$
I'd like to find the minimum and maximum distance between the origin and the intersection curve between the sphere and the cone. This is the question.
My attempt:
Setting up the Lagrange multiplier problem, the objective function is
$ g(r) = r^T r + \lambda_1 (r^T Q r) + \lambda_2 ( (r - C)^T (r - C) - a^2 ) \tag{3} $
Taking the gradient vector of $g$ with respect to $r$, we get
$ \nabla_r g = 2 r + 2 \lambda_1 \ Q r + 2 \lambda_2 \ (r - C) = \mathbf{0} \tag{4}$
By differentiating with respect to $\lambda_1$ and $\lambda_2$, we get two additional equations:
$ r^T Q r = 0 \tag{5}$
$ (r - C)^T (r - C) - a^2 = 0 \tag{6}$
And the task now is to solve the system of equations $(4), (5), (6)$ for $r$.
Edit:
If $u_1$ and $u_2$ are perpendicular to $C$, then
$u_1^T (\nabla_r g) = 2 \bigg( u_1^T r + \lambda_1 \ u_1^T Q r + \lambda_2 \ u_1^T r \bigg) = 0 $
i.e.
$ (1 + \lambda_2) (u_1^T r) + \lambda_1 \ u_1^T Q r = 0 $
Similarly with $u_2$, we will get,
$ (1 + \lambda_2) (u_2^T r) + \lambda_1 \ u_2^T Q r = 0 $
Since this homogenous system has non-trivial solutions, it follows that its determinant must be zero. That is,
$ (u_1^T r) (u_2^T Q r) - (u_2^T ) (u_1^T Q r) = 0 $
which can be re-arranged, and re-written as,
$ r^T ( u_1 u_2^T - u_2 u_1^T ) r = 0 \tag{3} $
Now we have a system of three quadratic equations $(1),(2),(3)$ in three unknowns which are the coordinates of $r$. This can solved numerically, for example with Sage Math.
Hint.
No need for Lagrange multipliers. The solution points are located on the plane which contains $C$ and the cone revolution axis.
The cone can be represented as
$$ \|r-(r\cdot \vec v)\vec v\|=\alpha |r\cdot \vec v|,\ \ \ \|\vec v\|=1 $$
and this plane can be defined as
$$ \Pi\to (\vec v\times C)\cdot r = 0 $$
and
$$ Q = (I_3-(1+\alpha^2)\vec v\vec v^T)c_0 $$
Now the intersection between $Q$ and $\Pi$, gives two lines ($\mathcal{L}_1,\mathcal{L}_2$). The intersection between the sphere $\mathcal{S}$ and the plane $\Pi$ gives a circle $\mathcal{C}$ and finally the solution points are the intersections of ($\mathcal{L}_1,\mathcal{L}_2$) and $\mathcal{C}$.