I'm running into a problem when trying to show the mass continuity equation for a fluid, which says
$$\frac{\partial \rho}{\partial t} + \left(\nabla \cdot \rho \textbf{u}\right) = 0$$
Where $\rho=\rho(x,y,z,t)$ is the density of the fluid and $\textbf{u} = \textbf{u}(x,y,z,t)$ is the velocity vector of the infinitesimal unit of volume [or mass].
I start by noting that $m=\iiint_V\rho dV$. Differentiating that (with chain rule) gives us
$$\begin{align*} \frac{\partial m}{\partial t} &= \frac{\partial}{\partial t}\iiint_V \rho dV\\ &= \iiint_V \frac{\partial \rho}{\partial t} dV\\ &= \iiint_V \left(\frac{\partial \rho}{\partial x}x'(t)+\frac{\partial \rho}{\partial y}y'(t)+\frac{\partial \rho}{\partial z}z'(t)+\rho_t\right)dV\\ &=\iiint_V(\nabla \rho \cdot \textbf{u} + \rho_t)dV\\ \end{align*} $$
And as you can see, that doesn't really match up with the originally stated continuity equation. I have $(\nabla \rho \cdot \textbf{u})$ instead of $(\nabla \cdot \rho \textbf{u})$. Where have I gone wrong? What needs to be corrected?
What we want is $\displaystyle \frac{dm}{dt} = \iiint_V \frac{\partial \rho}{\partial t} dV$. But the integrand here is not equal to the expression in the next line as you've written it; that would be the case if were writing out the complete derivative$\displaystyle \frac{d\rho}{dt}$.
Instead, the instantaneous change in the mass moving into the volume is equal to flow into the closed volume $V$ over its boundary $\partial V$. If $\vec{u}$ is the velocity vector of the fluid, then
$$\frac{dm}{dt} = -\iint_{\partial V} \rho\vec{u}.d\vec{A}$$
There is a negative sign here because we want mass moving in have a positive effect, not a negative effect and the convention with a closed surface is the to have the normal vectors $d\vec{A}$ pointing out.
Applying now Stokes' theorem, this last integral equal to $$\frac{dm}{dt} = \iiint_V -\nabla(\rho\vec{u}) \ dV $$
Equating now the two expressions for $dm/dt$ we have the desired equation:
$$\frac{\partial \rho}{\partial t} + \nabla(\rho\vec{u}) = 0$$
I've never seen the derivation written down this way, but now that I have, I like it as much as the usual approach using infinitesimal cubes, of size $dx \ dy \ dz$, which in effect rederive Stokes' theorem for this case.