I'm struggling to evaluate this at $x = 2$ (or, really, any number).
To me, the fact that we have the $f(x)$ on the right hand side of the equation makes this a recursive function, so the calculation will go on indefinitely.
What is wrong with my approach?
On
Write your equation in the form $$f(x+1)-f(x)=2x+1\ .\tag{1}$$ This is a linear inhomogeneous equation for an unknown function $f$. A particular solution $f_p$ can be guessed, namely $f_p(x)=x^2$. In addition we need the general solution $f_{\rm hom}$ of the associated homogeneous problem $$f(x+1)-f(x)=0\ .\tag{2}$$ Now $(2)$ just says that $f_{\rm hom}$ has to be a periodic function of period $1$. It follows that the general solution of $(1)$ is $$f(x)=x^2 + g(x)\ ,$$ whereby $g$ is an arbitrary function of period $1$.
One solution is $$f(x)=x^2$$ since $$f(x+1)=x^2+2x+1=f(x)+2x+1$$, so $$f(2)=4$$