Let $u\in C^3(\mathbb{R}^3)$ and $u$ is zero outside some bounded domain. I need to find the value of $$ \int\limits_{ \large\mathbb{R}^3}\det \mathrm D^2 u(\mathbf{r}) \, \mathrm d\mathbf{r} $$ I suspect it equals zero. I can show this for some specific cases.
Here $\mathrm D u$ is a gradient of $u$, while $\mathrm D^2u$ is a Hessian.
Note that the Hessian matrix for $u: \mathbb{R}^3\to \mathbb{R}$ is the Jacobian of the gradient,
$$\text D^2u = \text D(\text Du) = \text J(\nabla u )$$
Let $\text{D}u = v = (v_x,v_y,v_z)$. Then, since $u$ and $v$ are continuous on $\mathbb{R}^3$ and vanish outside of some bounded domain $\Omega$, they must also vanish on the boundary. In particular, $v|_{\partial\Omega}\equiv 0$.
Recall that the Jacobian determinant is used to change coordinates in integration, since
$$\det \text Dv \ \mathrm{d}x\mathrm{d}y\mathrm{d}z = \mathrm{d}v_x\mathrm{d}v_y\mathrm{d}v_z$$
for any $v:\mathbb{R}^3\to\mathbb{R}^3$ (and in particular, $v=\text Du$), and so
$$\int_{\Omega}\det \text{D}^2u \ \mathrm{d}x\mathrm{d}y\mathrm{d}z = \int_{\Omega}\det \text D(\text{D}u) \ \mathrm{d}x\mathrm{d}y\mathrm{d}z = \int_{\Omega}\mathrm{d}v_x\mathrm{d}v_y\mathrm{d}v_z$$
By Gauss-Ostrogradsky Theorem,
$$\int_{\partial\Omega}F(r(s,t))\|\partial_s r\times \partial_t r\| \ \mathrm{d}s\mathrm{d}t = \int_{\partial \Omega}F \ \mathrm{d}S = \int_\Omega \text{tr}\text DF \ \mathrm{d}V$$
where $F: \mathbb{R}^3\to\mathbb{R}^3$ is any continuously differentiable function and $r: I_s \times I_t \to\mathbb{R}^3$ is a parameterization of $\partial \Omega$.
Let $F(v)=\frac{1}{3}v$ so that $\text DF = \frac{1}{3}\mathbf 1$ and so $\text{tr} \text DF = 1$. Then, for any $r$,
$$F(r(s,t)) = \frac{1}{3}v|_{\partial\Omega} \equiv 0$$
which gives us the result
$$\int_\Omega \det\text{D}^2u \ \mathrm{d}x\mathrm{d}y\mathrm{d}z = \int_{\Omega} \det \text D(\text{D}u) \ \mathrm{d}x\mathrm{d}y\mathrm{d}z = \int_\Omega \mathrm{d}v_x\mathrm{d}v_y\mathrm{d}v_z = \int_{\partial\Omega} 0\cdot\|\partial_s r\times \partial_t r\| \ \mathrm{d}s\mathrm{d}t = 0$$