Let $g:\mathbb{R}^2 \rightarrow \mathbb{R}^2$ and $f:\mathbb{R}^2 \rightarrow \mathbb{R}^3$ be the functions defined by: $$g(x,y):=(2ye^{2x},xe^y)$$ $$f(x,y):=(3x-y^2,2x+y,xy+y^3)$$
$a)$Show that there exists an open neighborhood $U$of $(0,1)\in \mathbb{R}^2$ and an open neighborhood $V$ of $(2,0)\in \mathbb{R}^2$ such that $g:U \rightarrow V$ is a bijection.
$b$) Using standard Euclidean basis, express $D_{(2,0)}(f \circ g^{-1})$ as a matrix.
My attempt
$a$)I was going to attempt to do a typical one-to-one proof and onto proof to show bijection but I don't think it's relevant in this case.
However, it seems like this problems involve the inverse function theorem. I know that if a function is bijective then its inverse exists (all though I am not sure if it is also true in the other direction).
What we know is that $(0,1)\in U$ and $(2,0)\in V$ and that $g$ is a function from $U$ to $V$.
I am not sure how to move on from here. Maybe find it's differential and see if the determinant $\neq 0$? Then what?
$b$) Not sure how to find $g^{-1}$.
For part (a), you can definitely use the inverse function theorem! This states that, if $g$ is differentiable with continuous derivatives, and the derivative $D g$ is an invertible matrix at a point $a$, then there exist open neighbourhoods $U$ of $a$ and $V$ of $g(a)$ such that $g: U \to V$ is invertible. Moreover, $g^{-1} : V \to U$ is also continuously differentiable, and the derivative of $g^{-1}$ at $g(a)$ is the matrix inverse of the derivative of $g$ at $a$:
$$ Dg^{-1}|_{g(a)} = (D g|_a)^{-1}.$$
If you compute $Dg|_a$ in your example, you will hopefully see that it is invertible at $a = (0,1)$. For example, you can check that its determinant is non-zero. So the inverse function theorem applies, providing the solution to part (a).
[Warning: I believe $g(a) = (2,0)$, not $(0,2)$, as you have written.]
For part (b), you can evaluate $D(f \circ g^{-1})|_{g(a)}$ using the chain rule: $$ D(f \circ g^{-1})|_{g(a)} = Df|_{a} . Dg^{-1}|_{g(a)},$$ where the dot denotes matrix multiplication. You already know $Dg^{-1}|_{g(a)}$ from the inverse function theorem, so you just need to calculate $Df|_a$ and multiply the two matrices.