Let $p$ be an odd prime. I need to find for which $0\leq a < p^2$, $(2p-1)!\equiv a\mod{p^2}$.
If $a\equiv (2p-1)!\mod{p^2}$, then we have that $a = kp^2 + (2p-1)!$, and therefore $p\mid a$, which means $a$ is an integer multiple of $p$, and necessarily not zero, since we have only one instance of $p$ in $(2p-1)!$
How can I proceed from here?
Your have $(2p-1)! = \prod_{i=1}^{2p-1} j =p \prod_{j=1}^{p-1}j(j+p)$. Thus modulo $p^2$ this is $p \prod_{j=1}^{p-1}j^2= p ((p-1)!)^2$.
Now one knows that $(p-1)!$ is congruent $-1$ modulo $p$; this is not hard to to show, and you can find it as part of a proof of Wilson's theorem.
Thus in total we get that the result is $p$.