On sum of two squares

Question

Can a prime of form $3\bmod 4$ ever divide an integer represented by form $a^2+b^2$ with $a,b$ being coprime and $b$ being even?

Answer 1

No, it can't. If $p\equiv 3\pmod 4$, $\gcd(a,b)=1$ and $a^2+b^2\equiv 0\pmod p$ then $b\not\equiv 0\pmod p$, so let $c\equiv ab^{-1}\pmod p$, and this makes $c^2\equiv -1\pmod p$.

Then $(c+i)(c-i)=c^2+1=kp$ for some $k\in\Bbb Z$ (where $i^2=-1$). Since $\Bbb Z[i]$ is an Unique Factorization Domain, and $p$ is a prime in $\Bbb Z[i]$ (proof below), $p$ divides $c+i$ or $c-i$. Contradiction.

Proof that $p$ is prime in $\Bbb Z[i]$: if $p$ is not a prime in $\Bbb Z[i]$, then $p=rs$, where $r$ and $s$ are in $\Bbb Z[i]$, $|r|>1$ and $|s|>1$. Also, $p^2=|r|^2\cdot|s|^2$, so $|r|^2=|s|^2=p$. This implies that $p$ is the sum of two squares (of integers), but the sum of two squares is $0$, $1$ or $2 \pmod 4$. Contradiction.

Answer 2

If $p \mid a^2+b^2$ and $a,b$ are coprime then we must have $b \not\equiv 0 \pmod p$, for otherwise $p$ would divide both $b$ and $a$. Thus, as pointed out in ajotatxe's answer, this implies $c^2 \equiv -1 \pmod p$.

Assuming that $p>2$ (so that $-1$ is distinct from $1$ mod $p$), this means the order of $c$ modulo $p$ is exactly $4$. So $4$ must divide the order of the multiplicative group $\mathbb Z^\times_p$, which is $p-1$; hence $p \equiv 1 \pmod 4$. The only possible primes dividing $a^2+b^2$ with $a,b$ coprime are $2$ and primes of the form $1$ mod $4$.