I have a dot product defined by a matrix $$ A= \begin{pmatrix} 2 & 0 & 1+i \\ 0 & 3 & -1 \\ 1-i & -1 & 3 \\ \end{pmatrix} $$ and I want to find an orthogonal complement of a subspace $$W=span\{(-i,1,1+i),(-1-i,1,3)\}$$
I think I need to find the null space of $$W.A=0$$ where W is a matrix with row vectors $$(-i,1,1+i)(-1-i,1,3)$$.
The thing is, I am not sure whether to conjugate the vectors of subspace first or not, as I tried and my complement ended up being $$(-3-3i,i,1)$$ but it doesn't seem right.
Can someone give me a general idea how to solve otrthogonal complement of complex subspace when the dot product is defined by a orthogonal matrix?
Given an hermitian positive definite matrix , as your $A$, the corresponding inner product on $\mathbb{C}^3$ is the Hermitian form defined as $$ \langle x,y\rangle=y^\dagger A x $$ where $y^\dagger$ is the conjugate transpose of $y$.
So in your case you have to found a vector $y=(y_1,y_2,y_3)^T$ such that:
$$ \begin{pmatrix} \overline{y_1}&\overline{y_2}&\overline{y_3} \end{pmatrix} \begin{pmatrix} 2 & 0 & 1+i \\ 0 & 3 & -1 \\ 1-i & -1 & 3 \\ \end{pmatrix} \begin{pmatrix} -i\\1\\1+i \end{pmatrix}=0 $$ and $$ \begin{pmatrix} \overline{y_1}&\overline{y_2}&\overline{y_3} \end{pmatrix} \begin{pmatrix} 2 & 0 & 1+i \\ 0 & 3 & -1 \\ 1-i & -1 & 3 \\ \end{pmatrix} \begin{pmatrix} -1-i\\1\\3 \end{pmatrix} =0 $$
The searched orthogonal complement subspace is the span of this vector $y$.