In my textbooks and lectures (I'm a second year physics student) I often come across statements such as $$``dV=4\pi r^2dr\text{ can be thought of as the spherical volume element between }r\text{ and }r+dr."$$
Is this meant only symbolically, or is it a precise statement? For instance, I get \begin{align}\int_r^{r+dr}4\pi r^2dr&=\frac{4}{3}\pi\left[r^3\right]_r^{r+dr}\\&=\frac{4}{3}\pi(r^3+dr^3+3rdr^2+3r^2dr-r^3)\\&=\frac{4}{3}\pi(dr^3+3rdr^2+3r^2dr)\\&=4\pi r^2dr+\text{increasingly smaller terms.}\end{align}
Moreover, a Taylor expansion of the spherical volume function at a point $r'$ around a general point $r$ gives $$V(r')\approx\frac{4}{3}\pi r^3+4\pi r^2(r'-r)+\cdots.$$ Since the volume of a spherical shell between $r$ and $r+dr$ is generally (?) the volume of a sphere with radius $r+dr$ minus the volume of a sphere with radius $r$, we get $$\text{volume between }r\text{ and }r+dr\approx\frac{4}{3}\pi r^3+4\pi r^2dr-\frac{4}{3}r^3\approx4\pi r^2dr.$$
This leads me to think that what $4\pi r^2dr$ really is, is a good approximation (in fact, a first order Taylor expansion) of the volume between $r$ and $r+dr$, which can furthermore be assumed to be perfectly valid in the limit $dr\to 0$.
Maybe ... I'm just being pedantic?
You aren't being pedantic: you're just attaching a precise mathematical meaning to the informal statement you quoted.
The same can be said in general. Let $f(x)$ be a differentiable real-valued function of one real variable.
Then $df=f'(x)dx$ can be thought of as the "differential element of $f$" between $x$ and $x+dx$. (Indeed, in advanced calculus, $df$ is called a "differential," as in "differential form.")
This just means that $$f(x+dx)-f(x)\approx f'(x)dx$$
to first order, as you wrote.