For a prime $p>3$ the sum of the squares $\pmod p$ is $0$. Indeed, we have $$\sum_{k=1}^{p-1} k^2=\dfrac{p(p-1)(2p-1)}6\equiv 0\pmod p$$
But for $p\equiv 1\pmod 4$ we have that $-1$ is a square $\pmod p$, so the result also follows. (If $a$ is a square then $-a$ is also a different square, so the sum is $0$).
Question: Is there any 'more modular' proof for $p\equiv 3\pmod 4$?
If $p$ is an odd prime, the non-zero squares are exactly the roots of $x^{(p-1)/2}-1,$ and so, when $(p-1)/2>1,$ the sum of the roots is zero. Your sum is twice the sum.
More generally, if $j$ is not a multiple of $p-1,$ then $$\sum_{k=1}^{p-1} k^j\equiv 0\pmod p$$
This is because the non-zero perfect $j$-th powers modulo $p$ are the roots of $x^{(p-1)/\gcd(j,p-1)}-1,$ and the sum is some multiple of the sum of the roots of this polynomial.
Another proof can just take any $1\leq a\leq p-1,$ multiplication by $a$ modulo $p$ is a permutation, and you get:
$$\sum_k k^2\equiv\sum_k (ak)^2=a^2\sum k^2\pmod p$$
So if $p>3$ there is a non-zero $a^2$ which is not $1,$ and thus the sum must be zero, modulo $p.$
That same works for the generalization - if there is an $a$ with $a^j\not\equiv0,1\pmod p,$ the the sum of the $j$th powers must be zero, modulo $p.$