Given the elliptic cone $r^T Q r = 0 $ , where $ r= [x, y, z]^T $ and
$$ Q = \begin{bmatrix} \dfrac{1}{a^2} && 0 && 0 \\ 0 && \dfrac{1}{b^2} && 0 \\ 0 && 0 && -1 \end{bmatrix} $$
And suppose $ U, V, W$ are three mutually perpendicular vectors. Find the condition on $a$ and $b$, such that if $U$ is contained in the surface of the cone, then so are $V$ and $W$.
My attempt:
From the given structure of matrix $Q$, we can see that a vector contained in the surface of this elliptic cone has the form:
$ U = k [ a \cos s , b \sin s , 1 ]^T $
Since we're talking about angles and not lengths, we can take $ k = 1$.
The plane perpendicular to $U$ (which is to contain $V$ and $W$) has the equation
$ a \cos s \ x + b \sin s \ y + z = 0 $
Intersecting this plane with the cone, gives
$ \dfrac{x^2}{a^2} + \dfrac{y^2}{b^2} = (a \cos s \ x + b \sin s \ y )^2 $
which is a homogeneous equation of $x$ and $y$. Let $m = \dfrac{y}{x} $ , then dividing the above equation by $x^2$ gives
$ \dfrac{1}{a^2} + \dfrac{m^2}{b^2} = ( a \cos s + b \sin s m )^2 $
which is a quadratic equation in $m$. If $m_1$ and $m_2$ are the solution, then vectors $V$ and $W$ are given by
$ V = (x_1, y_1, z_1) = x_1 ( 1 , m_1 , a \cos s + b \sin s \ m_1 ) $
$ W = (x_2, y_2, z_2) = x_2 (1, m_2, a \cos s + b \sin s \ m_2 ) $
Imposing orthogonality between $V$ and $W$ and using Vieta's formulas applied to the above quadratic equation, will lead to the required condition on the parameters $a$ and $b$.
As I mentioned in the question statement, we now have the following quadratic equation in $m = \dfrac{y}{x} $
$ m^2 ( b^2 \sin^2 s - \dfrac{1}{b^2} ) + m (a b \sin(2 s) ) + (a^2 \cos^2 s - \dfrac{1}{a^2} ) = 0 $
The two solutions $m_1 , m_2$ satisfy Vieta's formulas
$ m_1 m_2 = (a^2 C^2 - \dfrac{1}{a^2}) / (b^2 S^2 - \dfrac{1}{b^2} ) $
$ m_1 + m_2 = - a b \sin(2 s) / (b^2 S^2 - \dfrac{1}{b^2} ) $
Orthogonality of the two vectors implies
$1 + m_1 m_2 + (a C + b S m_1) (a C + b S m_2) = 0$
Substituting the above expressions for $m_1m_2$ and $m_1 + m_2$ into this gives us, after simplification
$\boxed{ \dfrac{1}{a^2} + \dfrac{1}{b^2} =1 }$
The following Sagecell.Sagemath.Org page contains an example for such a cone described by
$r^T Q r = 0 $ with $Q = \begin{bmatrix} a_1 && 0 && 0 \\ 0 && 1-a_1 && 0 \\ 0 && 0 && -1 \end{bmatrix} $ with $ 0 \lt a_1 \lt 1 $
and a possible set of three mutually perpendicular vectors embedded in it. You can freely rotate the view, by holding down the left mouse button and dragging, inside the plot window.