I would like to know if there exists some criteria to test whether a given modular form, of level N with integral or half-integral weight, has non negative coefficients. I am also interested in results saying when certain coefficients are necessarily non negative.
Any research paper or reference is welcome !
On
Any cusp form will have infinitely many positive coefficients and infinitely many negative coefficients. This is pointed out in the answer from reuns when the multiplier system is trivial and there is trivial central character.
For holomorphic cuspidal Hecke eigenforms of integer weight $k \geq 2$, this result is also implied by the Sato-Tate distributions of (normalized) coefficients. This is a much, much stronger result than what you ask for.
More generally, fix some half-plane passing through the origin. Any cusp form, with any weight and any character, will have infinitely many coefficients inside each half-plane. This follows from the pair of facts that $$ \sum_{n \leq X} a(n) = o(X) $$ and $$ \sum_{n \leq X} \lvert a(n) \rvert^2 \asymp X. $$ Here, the coefficients $a(n)$ are normalized to be $1$ on average. The first is the analogue of the circle problem in this case, and the second is essentially work due to Rankin and Selberg (separately). This applies also to half-integral weight forms.
The same is not necessarily true of non-cuspidal forms. For example, the classical theta function $$ \sum_{n \in \mathbb{Z}} e^{2 \pi i n^2 z} $$ is a modular form of weight $1/2$ with nonnegative coefficients. More generally, to any positive semidefinite quadratic form, one can associate a theta function with nonnegative coefficients.
I do not see an easy way to determine if a generic given modular form will have nonnegative coefficients if it is not cuspidal.
Let $f\in S_k(\Gamma(N))$, ie. $f(z)=\sum_{n\ge 1} a_n e^{2i\pi nz}$ vanishes at all the cusps.
Then $ f(-1/z)z^{-k}=O(e^{-2\pi\Im(z)})$ as $\Im(z)\to \infty$ automatically implies that the $a_n$ aren't non-negative.