We need to solve the following least square problem \begin{align} Y=X\theta+W, \end{align} with quadratic constraint \begin{align} \theta^TA\theta = 0, \end{align} where the complex matrix $X \in \mathbb{C}^{m\times n}$, the complex vector $Y \in \mathbb{C}^{m\times 1}$, and the real matrix $A \in \mathbb{R}^{n\times n}$ are given, and $W \in \mathbb{C}^{m\times1}$ is the complex additive white Gaussian noise (zero mean and unit variance). $\theta$ is the parameter to estimate, so basically we need to solve
\begin{equation} \begin{array}{rrclcl} \displaystyle \min_{\theta} & {\|Y-X\theta\|^2} \\ \textrm{s.t.} &\theta^TA\theta& = & 0 \\ \end{array} \end{equation} in a closed form.
Indeed we try to have some intuition by starting from working on a special case: $m=n=4$, $A =\begin{pmatrix}0_{2\times2}& a \\ 0_{2\times 2} & 0_{2\times 2}\end{pmatrix}$, where $a = \begin{pmatrix}0& 1 \\ -1 & 0\end{pmatrix}$. By employing Lagrange multipliers we have \begin{align} -2X^HY+2X^HX\theta+\lambda \begin{pmatrix}0_{2\times2}& a \\ -a & 0_{2\times 2}\end{pmatrix}\theta &= 0\nonumber \\ \theta^T\begin{pmatrix}0_{2\times2}& a \\ 0_{2\times 2} & 0_{2\times 2}\end{pmatrix}\theta &= 0 \end{align}
Did we do something wrong? Can anyone help us? Thanks a lot!