Smallest prime p with N a quadratic residue mod p

Question

Let $N$ be a square-free number equal to 2 or 3 $(\mod 4)$.
Let $P(N)$ be the first odd prime, not a factor of $N$, for which $N$ is a quadratic residue.
On average, $N$ would be a non-residue for all the first $k$ primes once in $2^k$, so I would expect $P(N)=O( \log^m N)$ for a small number $m$.
Is there a logarithmic upper bound for $P(N)$?
This is related to an earlier question showing that 436 numbers are not of the form $x^2+My^2$.