What is the condition on the $N\times N$ Matrix $\Omega$, So that $Tr\left[{A}^{\dagger}B\Omega\right]$ is a valid inner product in Vector Space of $N\times N$ Complex Matrices?(Where $A$ and $B$ are $N\times N$ Matrix)
I tried by Matrix Multiplication, I took general assumption that to valid the product ${A}^{\dagger}B\Omega=\left(\left[a_{i j}\right]_{n\times m}\right)^\dagger\left[b_{i j}\right]_{n\times p}\left[\omega_{i j}\right]_{p\times q}$
Step : 1 $B\Omega=R=\left[r_{i j}\right]_{n\times q}$ Where $r_{i j}=\sum_{k=0}^{p-1}{b_{ik}\omega_{kj}}$
Step : 2 ${A}^{\dagger}B\Omega = {A}^{\dagger}R={R}^{'}=\left[r_{i j}^{'}\right]_{m\times p}$ where $r_{i j}^{'}=\sum_{l=0}^{n-1}{a_{li}^{*}r_{lj}}=\sum_{l=0}^{n-1}{a_{li}^{*}}\sum_{k=0}^{p-1}{b_{lk}\omega_{kj}}$
step : 3 $Tr\left[{A}^{\dagger}B\Omega\right]=\sum_{i=0}^{s-1}{}\sum_{l=0}^{n-1}{}\sum_{k=0}^{p-1}{a_{li}^{*}b_{lk}\omega_{ki}}$ where $m=p=s\left(say\right)$
I think a moment , it also demand $n=m$ or not ?
But during verification of inner-product definitions $\langle A,B\rangle$,
1- positivity : $\langle A,A\rangle=\sum_{i=0}^{s-1}{}\sum_{l=0}^{n-1}{}\sum_{k=0}^{p-1}{a_{li}^{*}a_{lk}\omega_{ki}} \ge 0$ [I freezes here]
2-skew-symmetry: $\langle A,B\rangle={\langle B,A\rangle}^{*}\implies a_{li}^{*}b_{lk}\omega_{ki} = b_{li}a_{lk}^{*}\omega_{ki}^{*}= k \leftrightarrow i $ on right side$\implies a_{li}^{*}b_{lk}\omega_{ki} = b_{lk}a_{li}^{*}\omega_{ik}^{*}$
$\implies \omega_{ki}=\omega_{ik}^{*} \implies \Omega $ a Hermitian Matrix ($p=q=s$).
3- Distributive: $\langle A+B,C\rangle=\langle A,C\rangle+\langle B,C\rangle$
It also demand $n=m$ or not ?
4- Sesquilinearity: $\langle \alpha A,B\rangle=\alpha^{*}\langle A,B\rangle $ And $ \langle A,\alpha B\rangle=\alpha\langle A,B\rangle$
I also tried another way, $\langle A,B\rangle=Tr\left[{A}^{\dagger}B\Omega\right]$
1- positivity : $\langle A,A\rangle=Tr\left[{A}^{\dagger}A\Omega\right]=Tr\left[{A}^{\dagger}A\Omega\right]$
2-skew-symmetry: $\langle A,B\rangle={\langle B,A\rangle}^{*} \implies\Omega $ a Hermitian Matrix So on.
I think that I am broken in the proof and about their order
This question is more proper of MathSE, but the answer is easy. Suppose that $\Omega$ is a strictly positive matrix, i.e., both conditions are valid
(a) $u^\dagger\Omega u \geq 0$ for every $u \in \mathbb{C}^N$,
(b) $u^\dagger\Omega u = 0$ implies $u=0$.
Since positive matrices are Hermitian, we also have $\Omega = \Omega^\dagger$. Furthermore, $\Omega = P^\dagger P$ for some strictly positive matrix $P$ as it follows from spectral decomposition.
Notice that $P$ is also bijective, it being strictly positive (all eigenvalues are strictly positive and thus the determinant cannot vanish). Therefore $P^{-1}$ exists.
$Tr(A^\dagger B\Omega) = Tr (B \Omega A^\dagger) = Tr (BP^\dagger PA^\dagger) = Tr((PB^\dagger)^\dagger PA^\dagger)$.
From this expression you can easily prove that $(A,B) \mapsto Tr((PB^\dagger)^\dagger PA^\dagger)$ satisfies all the requirements of an Hermitian scalar product just because the initial Hermitian scalar product $(A,B) \mapsto Tr(B A^\dagger)$ satisfies them.
The only interesting fact is the strictly positivity. Evidently
$Tr(A^\dagger A\Omega) = Tr((PA^\dagger)^\dagger PA^\dagger) \geq 0$.
Furthermore
$0 = Tr(A^\dagger A\Omega) = Tr((PA^\dagger)^\dagger PA^\dagger)$
implies (since the initial inner product is strictly positive)
$P A^\dagger =0$
Multiplying both sides for $P^{-1}$, we have $A^\dagger=0$ and thus $A=0$ just by taking the adjoint.
Now, let us assume that $(A,B) \mapsto Tr(A^\dagger B \Omega)= Tr(B \Omega A^\dagger)$ is an Hermitian scalar product. It is clear that we can always assume that $\Omega = \Omega^\dagger$. Indeed, since we are dealing with an Hermitian scalar product,
$\overline{Tr(A\Omega A^\dagger)} = Tr(A\Omega A^\dagger) \in \mathbb{R}$,
but the left hand side is
$Tr((A \Omega A^\dagger)^\dagger)= Tr(A\Omega^\dagger A^\dagger)$.
Hence
$Tr(A \Omega A^\dagger) = Tr \left(A \frac{1}{2}(\Omega+ \Omega^\dagger) A^\dagger \right)$.
By a standard polarization argument this identity extends to
$Tr(B \Omega A^\dagger) = Tr \left(B \frac{1}{2}(\Omega+ \Omega^\dagger) A^\dagger \right)$ where $\frac{1}{2}(\Omega+ \Omega^\dagger)$ is Hermitian by construction.
We finally prove that $\Omega$ (now supposed to be Hermitian) is strictly positive if $(A,B) \mapsto Tr(B \Omega A^\dagger)$ is an Hermitian scalar product.
Just take as $B$ a matrix containing only a non-zero row, the first one, corresponding to a vector $u^\dagger$, and $A^\dagger= B^\dagger$ is a matrix containing only a non-zero column, the first one, corresponding to the vector $u$.
We have
$0\leq Tr(B \Omega B^\dagger) = u^\dagger \Omega u$
and $=0$ is possible only if $B=0$, that is $u=0$.
In summary,
PROPOSITION. Assume that $\Omega \in M(N, \mathbb{C})$. The following facts are true.
(a) The map $$M(N, \mathbb{C})\times M(N, \mathbb{C}) \ni (A,B) \mapsto Tr(A^\dagger B \Omega) \in \mathbb{C}$$ is an Hermitian scalar product on the complex vector space $M(N, \mathbb{C})$ if $\Omega$ is strictly positive (thus Hermitian).
(b) If the map $$M(N, \mathbb{C})\times M(N, \mathbb{C}) \ni (A,B) \mapsto Tr(A^\dagger B \Omega) \in \mathbb{C}$$ is an Hermitian scalar product, we can always assume that $\Omega=\Omega^\dagger$. In this case $\Omega$ is strictly positive.