Suppose I am working in $\Bbb R^3$. Suppose I have a pond and I drop some dust on the surface. If the materials spread out, I have positive divergence, usually.
Let $\mathbf{v}(\mathbf{x})$ denote the velocity of a dust particle. I assume my divergence is a measure of the magnitude of this particle's tendency to move away. So if I want to find the magnitude the particle will move, why wouldn't $||\mathbf{v}(\mathbf{x})||$ denote my divergence?
The divergence is not just a single particle's tendency to move, but the tendency of a small ball of test particles to move away from each other. If the test particles stay together, they can move as fast as you like, but they don't diverge from each other.