Let $g:\mathbb{R}^n\mapsto\mathbb{R}^n$ be a smooth function, $M$ be an $m$-dimensional manifold embedded in $\mathbb{R}^n$ ($m<n$) with a parameterization $f:U\mapsto M$, where $U\subset\mathbb{R}^m$. Let $h:\mathbb{R}^n\mapsto \mathbb{R}$ be a smooth function. Let $Df$ and $Dg$ be the Jacobian matrices and assume them to be nonsingular. I would like to understand how to correctly apply the area formula for $\int_{g(M)} h(x)dx$.
(a) Does the form of integration on $M$ exist for $\int_{g(M)} h(x)dx$ in general? I do not quite believe $\int_{g(M)} h(x)dx = \int_{M} h(g(x))|\text{det}[Dg(x)]|dx$ is correct.
(b) What about the integration on $U$? It seems $Dg(f(x))Df(x)$ is the Jacobian for $g\circ f$, so do we have $\int_{g(M)} h(x)dx = \int_{U} h(g(f(x)))\sqrt{|\text{det}[Df(x)^TDg(f(x))^TDg(f(x))Df(x)}]|dx$?
Welcome to MSE. Let me see if I can help get towards your answer.
First, integration is defined (for the kinds of things I'm guessing you're studying -- roughly "manifolds" at the level of Spivak's "Calculus on Manifolds," or maybe Guillemin and Pollack's book) only over manifolds. That is to say $$ int_B w $$ is defined only if $B$ is a manifold. And if $B$ is an $m$-dimensional manifold, then $w$ should be an $m$-form. Assuming $B$ is an orientable manifold with a volume-form $\omega$, I think that what you're trying to do is integrate $h\omega$. It's just that you've called $\omega$ by the name $dx$.
But the main point is that this all makes sense only if $B$ is a manifold. When you write $$ \int_{g(M)} h(x) ~dx, $$ you need to know that $g(M)$ is indeed a manifold, and has a volume form. For that, you probably need a restriction on $g$. You might, for instance, say that $g$ is a diffeomorphism, in which case $g(M)$ is sure to be a manifold.
Once you do that, things are relatively easy: if you have a parameterization $f: U \to M$ (one that resembles, say, latitude-longitude coordinates on a sphere, and hence is a diffeomorphism almost everywhere, and is surjective), then $g \circ f$ will be a parameterization of $g(M)$. And now your approach, in part "b", says how to compute the integral over the manifold by instead integrating over a nice open subset of $\Bbb R^n$.
I can never remember the exact formula for the part b thing, but it sure looks as if you have it right.
Just to go back to the earlier "is g(M) a manifold" issue, think of something like a nice mobius-band shaped thing, sitting on the $xy$ plane in 3-space. But instead of a half-twist, put in a full twist, so that this is really just an embedding of a cylinder in 3-space. Call that $M$.
Now let's let $g: \Bbb R^3 \to \Bbb R^3: (x, y, z) \to (x, y, 0)$. That's a smooth function, but $g(M)$ is not a smooth manifold: depending on how you oriented your Mobius band, it looks line an annulus pinched at a couple of points, or like a circle that fattens out at a couple of points. In either case, it's not a manifold, so integrating over it doesn't really make sense.