Let $g : \mathbb{R}^2\to \mathbb{R}^2$ be given by $$g(x,y)=(2ye^{2x},xe^y).$$ Let $f: \mathbb{R}^2\to \mathbb{R}^3$ be given by $$f(x,y)=(3x-y^2, 2x+y, xy+y^3).$$
(a) Show that there is a neighborhood of $(0,1)$ that $g$ carries in a one-to-one fashion onto a neighborhood of $(2,0)$.
(b) Find $D(f\circ g^{-1})$ at $(2,0)$.
For (a), I will use the inverse function theorem, I must prove that $Dg(0,1)$ is non-singular, we know that $$Dg(x,y)=\begin{bmatrix}4ye^{2x} & 2e^{2x}\\ e^y & xe^y\end{bmatrix},$$ so $$Dg(0,1)=\begin{bmatrix}4 & 2\\ e & 0\end{bmatrix}$$ and $$\det Dg(0,1)=-2e\neq 0,$$ we conclude then, by the inverse function theorem that there is a neighborhood of $(0,1)$ that $g$ carries in a one-to-one fashion onto a neighborhood of $g(0,1)=(2,0)$.
(b) $$D(f\circ g^{-1}(t))=Df(g^{-1}(t))[Dg(h(t))]^{-1}=Df(h(t))[Dg(h(t))]^{-1},$$where $h$ is the inverse function of $g$, how can I calculate $h(2,0)$?
Are these arguments right? Thank you very much.
This problem is the "analysis on manifolds" by Munkres
Your argument for (a) is correct and I've got nothing to comment on it (except maybe that the determinant is $-2e$ instead of $2e$, whatever).
For (b), you do $${\sf D}(f\circ g^{-1})(2,0) = {\sf D}f(g^{-1}(2,0))\circ {\sf D}g^{-1}(2,0) = {\sf D}f(0,1) \circ {\sf D}g(0,1)^{-1}. $$
You compute ${\sf D}f(0,1)$ directly like in (a), and you invert the result of (a) to obtain ${\sf D}g(0,1)^{-1}$. This all holds in the neighbourhood given by the Inverse Function Theorem.