Given the ellipsoid
$ (r - C)^T Q (r - C) = 1$
and the plane
$ a^T r = b $
where $ r = [ x,y,z]^T $, $C \in \mathbb{R}^3$ is the center of the ellipsoid, and $Q \in \mathbb{R}^{3 \times 3} $ is a symmetric positive definite matrix. And, $ a \in \mathbb{R}^3 $ is a normal vector to the plane, and finally $b$ is a scalar.
I would like to find the closest and farthest points on the intersection of the plane and the ellipsoid to a point $P$.
My approach:
The distance squared function is
$ f(r) = (r - P)^T (r - P) $
The corresponding Lagrange multiplier function that includes both constraints is
$ g(r) = (r - P)^T (r - P) + \lambda_1 ( (r - C)^T Q (r - C) - 1 ) + \lambda_2 (a^T r - b) $
The gradient of this function is
$ \nabla_r g = 2 (r - P) + 2 \lambda_1 (r - C) + \lambda_2 a = 0 $
Let $u_1, u_2$ be two linearly independent vectors that are perpendicular to $a$. Then
$ u_1^T (r - P) + \lambda_1 u_1^T (r - C) = 0 $
$ u_2^T (r - P) + \lambda_1 u_2^T (r - C) = 0 $
This can be written in vector form as
$ \begin{bmatrix} u_1^T (r - P) && u_1^T (r - C) \\ u_2^T (r - P) && u_2^T (r - C) \end{bmatrix} \begin{bmatrix} 1 \\ \lambda_1 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix} $
This is a homogeneous system, and for it to have non-trivial solutions, we must have the determinant of the coefficient matrix equal to zero. That is,
$ ( u_1^T (r - P) ) (u_2^T (r - C) ) - (u_2^T (r - P) ) (u_1^T( r - C) ) = 0 $
Re-arranging the terms, this becomes
$ (r - P)^T \bigg( u_1 u_2^T - u_2 u_1^T \bigg) (r - C) = 0 $
And this the desired third equation. So now we have one linear equation in $r$ which is $a^T r = b $ and two quadratic equations in $r$ which are $ (r - C)^T Q (r - C) = 1$ and $ (r - P)^T \bigg( u_1 u_2^T - u_2 u_1^T \bigg) ( r - C) =0 $
These three equations can be solved for the three unknown coordinates of $r$. There will be a number of solutions (most likely two), where we can evaluate $f(r)$ at these points and determine which one gives the minimum and which one gives the maximum distance squared.
Having said that, I would like to verify my answer to my question, and would appreciate alternative solutions.
Thank you all.