I am puzzled about the following.
Let $(M,g)$ be a compact, orientable Riemannian manifold without boundary. We define the usual inner product $(,)$ for two r-forms $\alpha,\beta\in\Omega^r(M)$ by the formula $$ (\alpha,\beta) = \int_M \alpha\wedge \star\beta\, . $$ This inner product is positive-definite, i.e. for any $\alpha\in\Omega^r(M)$, we have $$ (\alpha.\alpha)\geq 0\, , $$ and $(\alpha,\alpha)=0$ if and only if $\alpha =0$.
For definiteness I focus now on a ten-dimensional manifold: dim $M$=10.
Let us consider $\gamma$ a self-dual 5-form, i.e. $-i \gamma = \star \gamma$ (I am using the convention where when acting on a r-form, $\star^2=(-1)^{r(m-r)}$ for dim $M$=m. Moreover, I could have considered anti-self-dual forms, by replacing $-i$ by $i$ in $-i \gamma = \star \gamma$). Then we have
$$ \gamma \wedge \star \gamma = - \gamma \wedge \star \gamma = 0\, , $$ where I used the self-duality condition as well as the fact that for a $r$-form $\omega$ and a $p$-form $\eta$, we have $\omega \wedge \eta = (-1)^{rp}\eta\wedge \omega$. But then we have $(\gamma,\gamma)=0$ and thus $\gamma=0$.
My question is: is there a flaw in the above argument?
My puzzle originally comes from the fact that in physics, we often encounter non-trivial 5-forms on 10-dimensional Riemannian manifold. The example that I have in mind is a case where the manifold is not compact ($AdS_5\times S^5$), so strictly speaking there is no contradiction with the above argument. But as far as I can see, the compactness is only necessary to define the integral of the inner product. But in the present case the product $(\gamma,\gamma)=0$, so there is no integration problem (there is no convergence problem).
I admit that the last argument is very hand-waving so I guess this is the place where a there's flaw but I'd like to know what you think.
Thank you for you answer(s).
Edit: Let me add one comment that might sharpen my puzzle.
In the example I have in mind, where $F\in\Omega^5(AdS_5 \times S^5)$ is such that $\star F = -iF$, we have moreover that $$ \int_{S^5} F $$ exist. Moreover, it is independent of the radial coordinate (if using the Poincaré coordinates of $AdS_5$). This suggests that there is really nothing wrong with $F$ at infinity, hence the above integrals for the inner product $(,)$ still make sense for this 5-form.
You're mixing up the real and complex as well as Riemannian and pseudo-Riemannian.
Let $X$ be a $2n$-dimensional Riemannian manifold. Then a self-dual $n$-form is a $+1$-eigenvector of the Hodge star map $$\star: \Omega^n(X; \Bbb R) \longrightarrow \Omega^n(X; \Bbb R).$$ We have the well-known formula $$\star^2 = (-1)^{n(2n - n)} \mathrm{Id} = (-1)^{n^2} \mathrm{Id}.$$ Now if $n$ is odd, as it is in your case, Then $\star^2 = -\mathrm{Id}$, so it cannot have $\pm 1$ as eigenvalues, and hence there are no self-dual or anti-self-dual $n$-forms. $\star$ does, however, have $\pm i$ as eigenvalues in this case, so you can define (anti-)self-dual $n$-forms as $\pm i$-eigenvectors of $\star$, as you did in your original post. Notice, however, that when you do this you are now working with complex $n$-forms and you should not be using the real inner product any longer.
To complicate matters further, you're really interested in Lorentzian manifolds. One way to define the Hodge star is by $$\alpha \wedge \star \beta = \langle\alpha, \beta\rangle\, \mathrm{vol},$$ where $\langle \cdot, \cdot \rangle$ is the inner product induced by the metric $g$. This is the same inner product you integrate to get the global inner product, and when $X$ is a Lorentzian manifold, $g$ is indefinite and hence the inner product on forms is indefinite as well. As you know, there is nothing wrong with having vectors of length zero in an indefinite inner product space (e.g. lightlike vectors in Minkowski space).
As a final remark, on a $2n$-dimensional Lorentzian manifold, we actually have $$\star^2 = (-1)^{n(2n - n) + (2n - 1)}\mathrm{Id} = (-1)^{n^2 + 2n - 1}\mathrm{Id}$$ (the signature of the metric affects things now). When $n$ is odd (as in your $10$-dimensional case), we then see that $\star^2 = \mathrm{Id}$ on a Lorentzian manifold (of dimension $4k + 2$), so we actually use the usual definition of self-duality (without any $i$'s). When $n$ is even (e.g. in Minkowski space), then $\star^2 = -\mathrm{Id}$ and we have to use the alternate definition of self-duality and work with complex inner product spaces.