Below is a theorem and proof as appears in https://arxiv.org/pdf/quant-ph/9608006.pdf
Notation:
We are working over $GF(4)$
Trace inner product: $u*v=Tr(u.\bar{v})=\sum_{i=1}^{n}Tr(u_{i}\bar{v_{i}})=\sum_{i=1}^{n}(u_{i}v_{i}^{2} + u_{i}^{2}v_{i}) (mod2)$
The conjugate over GF(4): $\bar{x}=x^{2}$
The dual code wrt the Hermitian trace inner product is $C^{\perp}=$ {$u \in GF(4)^{n} | u*c=0 \forall c \in C$}
$C$ is self-orthogonal if $C \subseteq C^{\perp}$
Theorem:
A linear code $C$ is self-orthogonal (with respect to the trace inner product) if and only if it is classically self-orthogonal with respect to the Hermitian inner product.
Proof:
The condition is clearly sufficient. Suppose $C$ is self-orthogonal. For $u,v \in C$ let $u.\bar{v}= \alpha + \beta w$ $\alpha, \beta \in \mathbb{Z}_{2}$.
Then:
$Tr(u.\bar{v})=0$ implies $\beta = 0$
$Tr(u.\bar{w}\bar{v})=0$ implies $\alpha = 0$
So $u.\bar{v}=0$
Question:
I want to check that my reasoning is correct:
$Tr(u.\bar{v})=0$ implies that $\beta=0$.
$Tr(u.\bar{v})=Tr(\alpha + w \beta)=Tr(\alpha) + Tr(\beta w)$. This is a linear code, we can distribute additively over the trace: $Tr(\alpha)=0$ because $\alpha$ is $0$ or $1$ and $Tr(0)=Tr(1)=0$
then $Tr(\beta w)$ has to be $0$ as well, meaning $\beta w$ must be $1$ or $0$, but $\beta w \neq 1$, so $\beta w =0$, meaning $\beta=0$.
Then $Tr(u.\bar{w}\bar{v})=Tr(\bar{w} \alpha + w \beta) = Tr(w^{2}\alpha +w^{3} \beta )=Tr((w+1)\alpha + \beta)=Tr(w\alpha) + Tr(\alpha) + Tr(\beta)=Tr(w \alpha) =0$ so $\alpha=0$
A note on inner products:
First note that in $GF(4)$, $\bar{x} = x^{2}$ and $\bar{u}=\bar{u_{1}},\dots,\bar{u_{n}} = u_{1}^{2}, \dots, u_{n}^{2}$
Hermitian trace inner product of two vectors in $GF(4)$ of length $n$, $u=u_{1},\dots,u_{n}$, $v=v_{1},\dots,v_{n}$ is defined by: $$u*v=Tr(u.\bar{v})=\sum_{i=1}^{n}Tr(u_{i}\bar{v_{i}})=\sum_{i=1}^{n}(u_{i}v_{i}^{2} + u_{i}^{2}v_{i}) (mod2)$$ This incidentally also gives us the number($mod2$) of positions in which $u$ and $v$ have different non-zero values. We describe the dual of $C$, $C^{\perp}$ with respect to this Hermitian trace inner product. A code is self-orthogonal if $C \subseteq C^{\perp}$ (defined wrt Hermitian trace inner product).
Hermitian inner product of two vectors in $GF(4)$ of length $n$ is $$u.\bar{v}$$ A code $C$ is classically self-orthogonal if it is orthogonal wrt the Hermitian inner product.
Symplectic inner product of two vectors $(a|b)$, $(a'|b')$ in $GF(2)$ of length $2n$ is defined as: $$((a|b),(a'|b'))=a.b' + a'.b$$