The Bolza surface is the smooth completion of the affine curve described by the equation:
$$y^2 = x^5 - x$$
This surface has genus 2, however I'm unable to get to that result.
The polynomial has degree 5 and no singularities, and according to the genus-degree formula it should have genus $\frac12(5 - 1)(5 - 2) = 6$, which is obviously not true. What mistake am I making?
The genus-degree formula that you've used is valid in $\Bbb P^2$. The projective completion of the affine curve $y^2=x^5-x$ in $\Bbb P^2$ is cut out by $y^2z^3=x^5-xz^4$, which has projective Jacobian $$\begin{pmatrix} -5x^4+z^4 & 2yz^3 & 4y^2z^2 + 4xz^3 \end{pmatrix}$$ which is of rank $0$ at $[0:1:0]$. So the projective closure has a singularity at $[0:1:0]$, and cannot be the smooth completion. You need to do a little more work here to find the genus.
One option is to analyze the singularity at $[0:1:0]$: if $X$ is the projective curve and $\widetilde{X}$ is the normalization (= the smooth compactification you're interested in), then $p_a(X)=p_a(\widetilde{X})+\sum_{P\in X_{sing}} \delta_P$, where $\delta_P$ is the delta invariant at $P$, the dimension of the integral closure $\widetilde{\mathcal{O}_{X,P}}$ of $\mathcal{O}_{X,P}$ modulo $\mathcal{O}_{X,P}$ (see for instance Hartshorne exercise IV.1.8).
First, note that the local ring at the singularity is $k[x,z]_{(x,z)}/(z^3-x^5+xz^4)$. The equation suggests adjoining an element like $\frac{x^a}{z^b}$ or $\frac{z^a}{x^b}$ to obtain the integral closure - it turns out that $w=\frac{x^2}{z}$ is the correct element, and $w$ satisfies $wz=x^2$, $w^2x=xz^2+z$, and $w^3=x^2z+x$. It turns out that the elements $w$, $w^2$, $wx$, and $wx^2$ are a basis for the quotient, showing $\delta=4$ and so the genus of the smooth compactification is 2.