Consider:$$7! = 7\cdot 6\cdot 5\cdot 4 \cdot 3 \cdot 2\cdot 1 = 5040$$ We also have: $$71^2 = 5041$$
How often does it happen that a factorial is right next to ($\pm 1$) a prime power (exponent $>1$)? Since factorials grow so quickly we can consider how many percent of $n$ s.t. $\exists p,k:$ $$n! = p^k\pm1, \cases{n\in \mathbb Z\\k\in \{2,3,\cdots\}\ \\p \text{ prime}}$$
Own work: So far checked $n<10$ by hand:
$$\begin{align*}5^2 &= 4!+1\\11^2&=5!+1\\71^2&=7!+1\end{align*}$$
I suppose some language with "big-int" types would be very helpful as $n!$ grows very fast.
The number of prime powers (exponent > 1) up to $x$ is roughly $\sqrt{x}/(2\log x)$ so I'd estimate the "probability" that a natural number $n$ has the property that $n!\pm 1=$prime power to be roughly $1/(\sqrt{n!}\log n!)$. This suggests that there are only finitely many such $n$.