Firstly, I want to thank @adam W gives a good clue to solve my homework problem. I have a set of quadratic matrix need to solve(not one equation) according to the following form:
$\alpha_{k}H(2n)\alpha^{T}_{k} = \delta(n),$
where $H(2n)$ is a symmetric matrix and the number of equations depend on the number of $\alpha_{k}$. For example, if $k=2$, the matrix can be extended in to following forms:
$[\alpha_{1} \alpha_{2}] * H(0) * [\alpha_{1} \alpha_{2}]^{T} = 1 ,$
$[\alpha_{1} \alpha_{2}] * H(2) * [\alpha_{1} \alpha_{2}]^{T} = 0 ,$
where $H(2n)$ has $2 \times 2$ matrix form.
Is there any algorithm for solving such kinds of quadratic matrix ?
Look first (using $\mathbf{\alpha} = [\alpha_1,\alpha_2]$) at $$\mathbf{\alpha} H_2\mathbf{\alpha}^T=0$$ With a suitable change of basis, this will give $$\mathbf{x}D\mathbf{x}^T = 0$$ This will lead to two equations representing lines through the origin. The following shows where that comes from, using $D=\pmatrix{d_0 & 0 \\ 0 & d_1}$ \begin{align} d_0x_0^2 + d_1x_1^2 &= 0 \\ x_0^2 &= \frac{-d_1}{d_0}x_1^2 \\ x_0 &= \pm\sqrt{\frac{-d_1}{d_0}}x_1 \\ x_0 &= \pm r x_1 \end{align}
So you have two lines from the equation $x_0 = \pm r x_1$. More conveniently for plugging into the other equation, you will have $$x_0^2 = r^2x_1^2$$ $$x_0x_1 = \pm r x_1^2$$ When this is plugged in, it will be linear in the term $x_1^2$ (since $x_0$ is substituted and removed), though two solutions become possible depending on the sign for $r$, and yet that doubles again in the next square root. This may help to see all the (one) cross terms: $$\pmatrix{x_0 & x_1}\pmatrix{A & B \\ C & D\\}\pmatrix{x_0 \\ x_1} = Ax_0^2 + (B+C)x_0x_1 + Dx_1^2$$ Which using the previous substitution gives $$Ar^2x_1^2 + (B+C)(\pm r) x_1^2 + Dx_1^2$$ and is easily solved for $x_1^2$
With your solutions in hand, just apply the inverse change of basis. Your four solutions are the intersection points of the ellipse described by the first equation, and the two lines described by the second equation.
Now if you want for higher dimensions, $k > 2$, things are not as simple. In that case, $$x_n^2 = \frac{-1}{d_n}\sum_{i\ne n}d_ix_i^2$$ which does not give a simple square in the cross terms: $$x_nx_m = \sqrt{\frac{-1}{d_n}\sum_{i\ne n}d_ix_i^2}\sqrt{\frac{-1}{d_m}\sum_{i\ne m}d_ix_i^2}$$
I would be interested if someone has a better answer for you in the more general case of $k>2$.