I am in a 3rd year university course in elementary number theory, and one of the problems I am given to solve is as follows.
If $p$ is prime and $x \in \mathbb{N}$ then $a^2+a = p^{2x}(b^2+b)$ has no solutions for $a,b \in \mathbb{N}$.
I am trying to show this Diophantine equation has no solutions though contradiction. So, assume there exist $a,b \in \mathbb{N}$ that satisfy the equation. Then we can factor $a^2+a=a(a+1)$. Because $a$ and $a+1$ are coprime, $p^{2x} \mid a$ or $p^{2x} \mid (a+1)$. Can splitting the equation like this lead to some sort of contradiction? I'll take any advice or hints!
Yes, splitting the equation like you did can lead to a contradiction. With your first case, for some positive integer $c$, we have
$$p^{2x} \mid a \; \; \to \; \; a = cp^{2x} \tag{1}\label{eq1A}$$
The given equation then becomes
$$\begin{equation}\begin{aligned} (cp^{2x})(cp^{2x} + 1) & = p^{2x}(b^2 + b) \\ c(cp^{2x} + 1) & = b^2 + b \\ b^2 + b - c(cp^{2x} + 1) & = 0 \end{aligned}\end{equation}\tag{2}\label{eq2A}$$
Treating \eqref{eq2A} as a quadratic equation in $b$ then, from the quadratic formula, the discriminant must be a perfect square. This means that, for some non-negative integer $e$, we have
$$\begin{equation}\begin{aligned} 1 - 4(-c(cp^{2x} + 1)) & = e^2 \\ 1 + 4c^2p^{2x} + 4c & = e^2 \\ 1 + 4c & = e^2 - (2cp^{x})^2 \\ 1 + 4c & = (e - 2cp^{x})(e + 2cp^{x}) \end{aligned}\end{equation}\tag{3}\label{eq3A}$$
Since $1 + 4c \gt 0$ and $e + 2cp^{x} \gt 0$, then $e - 2cp^{x} \gt 0$. Thus, on the RHS, we have $e - 2cp^{x} \ge 1$ and, since $e \gt 1$, $p \ge 2$ and $x \ge 1$, then $e + 2cp^{x} \gt 1 + 2c(2^1) = 1 + 4c$. This means the right side is $\gt 1 + 4c$, so \eqref{eq3A} can't be true. This contradiction shows there are no natural numbers $a$ and $b$ which work.
Using a similar procedure (which I'm leaving to the reader to do) with $p^{2x} \mid a + 1$ also leads to a contradiction.
Interestingly, apart from your initial step of $p^{2x} \mid a$ or $p^{2x} \mid a + 1$, the rest of the proof doesn't require that $p$ be prime, just simply that $p \ge 2$.