$\int\frac1rdl$ where $r$ is the position vector from each element $dl$ to the center of the loop with radius $R$.
Then I need to take the Curl of that to calculate the magnetic field. I'm having trouble setting up the integral to get anything meaningful.
The vector potential generated by an infinitesimal current element is proportional to $\mathrm d\vec l/r$, where $\mathrm d\vec l$ is the infinitesimal displacement along the wire and $r$ is the distance to the point at which the vector potential is evaluated. Note that you got this the wrong way around in the question; you described $r$ and not $\mathrm d\vec l$ as a vector.
The magnetic field is the curl of the vector potential. Since only $r$ and not $\mathrm d\vec l$ depends on the point at which the vector potential is evaluated, the magnetic field generated by the infinitesimal current element is proportional to
$$\vec\nabla\times\frac{\mathrm d\vec l}{r}=\left(\vec\nabla\frac{1}{r}\right)\times\mathrm d\vec l=\frac{\vec r}{r^3}\times\mathrm d\vec l\;.$$
If you want to calculate the magnetic field at the centre of a circular current loop, the vector product $\vec r\times\mathrm d\vec l$ is the same everywhere and points out of the plane of the loop, since $\vec r$ and $\mathrm d\vec l$ are always at right angles to each other, so the magnetic field points out of the plane of the loop and its magnitude is proportional to $\frac R{R^3}(2\pi R)=\frac{2\pi}R$.
If you want to read up on this in more detail, this set of notes might be helpful.