Reading back through an old text on E&M today, I realized that while it makes intuitive sense to me that upon defining density as $$\rho(x,y,z) = \frac{dm}{dV}$$ we get that $$M = \int_V \rho dV$$ I don't know how to prove it. (Note: in E&M it would be $dQ/dV$ but it doesn't really matter which type of density we're talking about so I just went with mass density here.)
Firstly, I'm not sure how the definition of $\rho$ really works as a derivative. What function is it the derivative of? It seems like what we really mean to define is $$\rho(P) = \lim_{V\to \{P\}} \frac{m(V)}{|V|}$$ where $V$ is some subset of $\Bbb R^3$, $m$ is a function of type $m:\mathcal P(\Bbb R^3) \to \Bbb R$ that takes regions of $\Bbb R^3$ to numbers, $|V|$ is the notation I decided on for the volume of the region $V$, and $P$ is some point within the volume $V$.
This doesn't really fit the definition of the derivative I'm familiar with (from Calc I-III back in college). But OK, let's go with that. Then we should be able to show that $$M := m(V) = \int_V\lim_{V'\to \{P\}} \frac{m(V')}{|V'|} dV$$ However, at this point, I'm at a loss for how to show this. It certainly seems like it must be the case physically, but I'm not quite sure how to handle the integral of a limit.
How can I prove this formula (or what am I doing incorrectly)?
Your question can be worded as follows:
Note that once this is proved, we have $$\int_V\lim_{V'\to \{P\}} \frac{m(V')}{|V'|} d\lambda = \int_V\rho \, d\lambda = m(V)$$ proving your claim.
Now, the lemma above is highly non-trivial. It is known as the Lebesgue Differentiation Theorem, and a standard proof using the Hardy-Littlewood Maximal Function can be found on the linked Wikipedia page.
Technically the result holds only almost-everywhere (it could fail on a set of measure zero), but in physical applications this might be meaningless. Additionally, the family of sets $\{V\}$ must be of bounded eccentricity for the limit to converge, and the limit is taken such that the diameters of the sets decrease