In this Lecture, in the subsection Evolution of Volumes tell us:
Let $M \subset D$ be a compat subset of phase space. We can define its volume by a usual Riemann integral: $$ Vol(M) = \int_{M}dx, \hspace{3mm} dx = dx_{1} \dots dx_{n} $$ We can define the sets $M_{k} = F^{k}(M)$, we assume that $F$ is continuously differentiable: $\frac{\partial F}{\partial x}(x)$, the Jacobian matrix of $F$ at $x$, which is the $n \times n$ matrix.
Assume $F$ is diffeomorphism and let $V_{k} = Vol(M_{k})$. Then $$ V_{k+1} = \int_{M_{k}} \Bigg\vert det(\frac{\partial y}{\partial x})\Bigg\vert dx $$
My question is: What does it mean take the determinant of the Jacobian and what is the proof about it?, What happen if instead I take the Hessian, what change?
Jacobian at a point is a matrix (with entries from $\mathbb{R}$), you can take a determinant of it, what puzzles you? If you take Hessian instead you will get something completely different -- it is easy to see if you take $F$ to be a linear map -- then the Hessian is just zero, whereas Jacobian is not.
To the formula it you basically split your $M$ into many small pieces, such that in every one of them your map is almost linear (looks line a Jacobian around at a point), for linear maps it is very easy to understand how does the volume change under the map, then just take a sum. Proof with details you can find in any standard calculus book.