I am currently studying a textbook and I have a limited background in complex linear algebra and unfortunately the textbook does not provide background knowledge for this either. As a result I am a bit stuck with something that should be fairly simple. The problem is as follows:
Given known $3\times1$ vectors $\mathbf{h_1}$ and $\mathbf{h_2}$ we want to solve for an unknown matrix $C$ of shape $3\times3$ consisting of real entries. Suppose that we form a constraint with them such that $$(\mathbf{h_1} \pm i\mathbf{h_2})^TC(\mathbf{h_1} \pm i\mathbf{h_2})=0$$ Where $i\mathbf{h_2}$ denotes that vector $\mathbf{h_2}$ is an imaginary component. Then such constraint can be rewritten as two constraints: $$\mathbf{h_1}^TC\mathbf{h_2}=0$$ $$\mathbf{h_1}^TC\mathbf{h_1}=\mathbf{h_2}^TC\mathbf{h_2}$$
Can anyone provide a thorough derivation of this? I suspect it is something to do with splitting up the real and imaginary parts or the plus minus sign?
The second constraint can be easily shown by the separation of the real and imaginary parts (as you actually noted). I can prove the first constraint similarly but only for a symmetric C matrix (it is not clearly stated but I suppose it is symmetric from your comment).
I start by expanding the left-hand side: $$ (\mathbf{h_1} \pm i\mathbf{h_2})^TC(\mathbf{h_1} \pm i\mathbf{h_2})=(\mathbf{h_1}^T \pm i\mathbf{h_2}^T)C(\mathbf{h_1} \pm i\mathbf{h_2}) = \mathbf{h_1}^TC\mathbf{h_1} \pm i\mathbf{h_1}^TC\mathbf{h_2} \pm i\mathbf{h_2}^TC\mathbf{h_1} + i^2\mathbf{h_2}^TC\mathbf{h_2} = \mathbf{h_1}^TC\mathbf{h_1} \pm i\mathbf{h_1}^TC\mathbf{h_2} \pm i\mathbf{h_2}^TC\mathbf{h_1} - \mathbf{h_2}^TC\mathbf{h_2} = 0 $$
You need to realize that $ 0=0+0i $ and you can balance both real and imaginary part separately. Balancing the real part gives you directly the second constraint: $$ \mathbf{h_1}^TC\mathbf{h_1} - \mathbf{h_2}^TC\mathbf{h_2} = 0 \implies \mathbf{h_1}^TC\mathbf{h_1} = \mathbf{h_2}^TC\mathbf{h_2} $$
Balancing the imaginary part is a little bit more tricky: $$ \pm \mathbf{h_1}^TC\mathbf{h_2} \pm \mathbf{h_2}^TC\mathbf{h_1} = 0 $$ By using the property of transpose operation that $ (AB)^T=B^TA^T $ I get: $$ \pm \mathbf{h_1}^TC\mathbf{h_2} \pm \mathbf{h_2}^TC\mathbf{h_1} = \pm \mathbf{h_1}^TC\mathbf{h_2} \pm (\mathbf{h_1}^TC^T\mathbf{h_2})^T = 0 $$ Both terms are scalar, which can be proved simply by dimensionality analysis, and transpose of a scalar is the scalar itself: $$ \pm \mathbf{h_1}^TC\mathbf{h_2} \pm (\mathbf{h_1}^TC^T\mathbf{h_2})^T = \pm \mathbf{h_1}^TC\mathbf{h_2} \pm \mathbf{h_1}^TC^T\mathbf{h_2} = 0 $$ Now, you get finally your other constraint if $C$ is symmetric (i.e. $ C=C^T $): $$ \pm \mathbf{h_1}^TC\mathbf{h_2} \pm \mathbf{h_1}^TC^T\mathbf{h_2} = \pm 2 \mathbf{h_1}^TC\mathbf{h_2} = 0 \implies \mathbf{h_1}^TC\mathbf{h_2} = 0 $$