I have the Lagrange multiplier problem where the objective function is
$ f(r) = r^T r $
where $ r \in \mathbb{R}^3 $
subject to $r^T Q r = 0 $ where $Q$ is a $3 \times 3$ symmetric indefinite matrix, and also subject to $ a^T r = b $, where $a \in \mathbb{R}^3$ and $b \in \mathbb{R} $. I'd like to find the minimum and the maximum of $f$ with the two given constraints.
So the Lagrange multiplier function will be
$ g(r) = r^T r + \lambda_1 (r^T Q r) + \lambda_2 ( a^T r - b ) $
Taking the gradient vector of $g$ with respect to $r$, we get
$ \nabla_r g = 2 r + 2 \lambda_1 \ Q r + \lambda_2 \ a = \mathbf{0} \tag{1}$
And also by differentiating with respect to $\lambda_1$ and $\lambda_2$
$ r^T Q r = 0 \tag{2}$
$ a^T r - b = 0 \tag{3}$
How can I solve the system of equations $(1),(2),(3)$ for $r$ ?
Working on the idea by 'whpowell96' in his comment above, the second (linear) constraint:
$ a^T r = b \tag{1}$
Leads to the solution
$ r = r_0 + V u \tag{2}$
where we can choose $ r_0 = \dfrac{b}{a^T a} a $ , and $V$ to be a $3 \times 2$ matrix whose columns are unit vectors that are mutually perpendicular and at the same time they are perpendicular to $a$. $u$ is the coordinate vector with respect to $V$.
Substituting $(2)$ into the function, we get
$ f = r^T r = (r_0 + V u)^T (r_0 + V u) = u^T u + 2 u^T V r_0 + r_0^T r_0 \tag{3} $
And the first quadratic constraint becomes
$ r^T Q r = (r_0 + V u)^T Q (r_0 + V u) = u^T V^T Q V u + 2 u^T V^T Q r_0 + r_0^T Q r_0 = 0 \tag{4}$
Thus, we have reduced the order of the problem from $3$ to $2$.
Now we'll define a new Lagrange multiplier function
$ g = u^T u + 2 u^T V r_0 + r_0^T r_0 + \lambda (u^T V^T Q V u + 2 u^T V^T Q r_0 + r_0^T Q r_0 ) \tag{5} $
The gradient of $g$ is
$ \nabla_u g = 2 u + 2 V^T r_0 + \lambda( 2 V^T Q V u + 2 V^T Q r_0 )=\mathbf{0} \tag{6}$
And the derivative of $g$ with respect to $\lambda$ will be
$u^T V^T Q V u + 2 u^T V^T Q r_0 + r_0^T Q r_0 = 0 \tag{7} $
Equation $(6)$ implies
$ u + V^T r_0 = (- \lambda) ( V^T Q V u + V^T Q r_0 ) \tag{8}$
Both the left hand and the right hand sides are two-dimensional vectors and equation $(8)$ says that they are multiples of each other. Vectors $V$ and $W$ that are two dimensional, are multiples of each other, if and only if
$ V_x W_y - V_y V_x = 0 \tag{9} $
Equation $(9)$ can be written as
$ V^T E W = 0 \tag{10} $
where
$E = e_1 e_2^T - e_2 e_1^T = \begin{bmatrix} 0 && 1 \\ -1 && 0 \end{bmatrix} \tag{11} $
Applying this to $(8)$, we deduce that
$ ( u + V^T r_0 )^T E ( V^T Q V u + V^T Q r_0 ) = 0 \tag{12} $
So now the solutions to our optimization problem are the solutions of the system of equations $(7)$ and $(12)$.
Once the solutions $\{u\}$ are found, the corresponding $r$ can be found by $(2)$.