This is a direct follow-up to this question, which I think is interesting enough to be a separate problem. Please check that post out too in case there are ideas there that might transfer here.
In the post linked, they found numerically that for odd primes $p \equiv 1 \pmod{4}$, the sum of all $n$ such that $(n^2+2n+1 \ \mathrm{mod} p) < (n^2 \ \mathrm{mod} p)$ is exactly $\frac{7}{24}(p^2-1)$. It is natural to generalise this for $p \equiv 3 \pmod{4}$, and you won't believe what I found! Here's some Sage code:
x = var('x')
for p in prime_range(6, 1001):
if p % 4 != 3: continue
ns = [n for n in range(p) if (n^2 + 2*n + 1) % p < (n^2) % p] # your condition
R.<t> = NumberField(x^2 + p) # setup Q[sqrt(-p)]
assert sum(ns) == (p^2 - 1) * 7 / 24 - R.class_number() # observed = ???
As annotated above, I conjecture that for odd primes $p > 3$ satisfying $p \equiv 3 \pmod{4}$, we have
$$ \sum_{0 \leq n < p} n \cdot [(n+1)^2\,\mathrm{mod}\, p \leq n^2\,\mathrm{mod}\, p] = \frac{7}{24}(p^2-1)-h(\sqrt{-p}). $$
Feel free to play with the Sage script above. I suspect that there might be some "symmetry" or relation with quadratic residues, since it is well known that the class number is (-1 / p) times sum of n * (n | p).
I will set a bounty when I can.