The problem I'm trying to solve is the following:
Let $F \colon \mathbb{R}^2 \to \mathbb{R}$ a function of class $C^1$ and $G(x,y)=(F(F(x,y),y),F(x,y))$. Give conditions on $F$ such that $G$ is a local diffeomorphism and calculate $dG^{-1}$.
My idea is to give conditions on $F$ such that the hypothesis of the inverse function theorem (specifically $\det J G (a,b) \neq 0$ for some $(a,b) \in \mathbb{R}^2$) hold. However, this specific condition is giving me trouble. I tried calculating the partial derivatives of $F$ through the definition but it's not possible to solve the involved limit without knowing $F$ explicitly. Instead, I think one could apply the chain rule to calculate $\det J G (a)$ in a nice way.
However, I'm completely lost in the notation. For example, if we say $G=(G_1,F)$ (where $G_1=F \circ G_2$ and $G_2=(F,\pi_2)$), then what $\frac{\partial G_1}{\partial x} (a,b)$ is? Something like $\frac{\partial G_1}{\partial F}(a,b) \frac{\partial F}{\partial G_2} (a,b) \frac{\partial G_2}{\partial x} (a,b)$? Well, I know it's not that, but I really have no idea what it's supposed to be.
Recall that if $f(x,y)=g(u(x,y),v(x,y))$, then: $$\begin{align}\partial_xf(x,y)&=\partial_xg(u(x,y),v(x,y))\partial_xu(x,y)+\partial_yg(u(x,y),v(x,y))\partial_xv(x,y),\\\partial_yf(x,y)&=\partial_yg(u(x,y),v(x,y))\partial_yv(x,y)+\partial_xg(u(x,y),v(x,y))\partial_yu(x,y).\end{align}$$ It is also easily checked that: $$\begin{align}\partial_xG(x,y)&=(\partial_xG_1(x,y),\partial_xG_2(x,y)),\\\partial_yG(x,y)&=(\partial_yG_1(x,y),\partial_yG_2(x,y)),\end{align}$$ where $G=(G_1,G_2)$.
Applying the first formula for $f=G_1$, $g=F$, $u=F$ and $v(x,y)=y$, one has: $$\partial_xG_1(x,y)=\partial_xF(F(x,y),y)\partial_xF(x,y).$$ Directly, one has: $$\partial_xG_2(x,y)=\partial_xF(x,y).$$ Therefore, one gets: $$\partial _xG(x,y)=(\partial_xF(F(x,y),y)\partial_xF(x,y),\partial_xF(x,y)).$$ Similarly, one has: $$\partial_yG(x,y)=(\partial_yF(x,y)+\partial_yF(F(x,y),y)\partial_yF(x,y),\partial_yF(x,y)).$$ From there, one finally has: $$\det(\textrm{Jac}_{(x,y)}G)=\partial_xF(x,y)\partial_yF(x,y)(\partial_xF(F(x,y),y)-\partial_yF(F(x,y),y)-1).$$ Here we have a nasty expression and it becomes hard to say things in all generality. Nonetheless, it is necessary that the partial derivatives of $F$ does not vanish to ensure that $G$ is a local diffeomorphism everywhere. In particular, $F$ must be itself locally invertible everywhere! Even though it is not that satisfying, one can probably conclude that $G$ is a local diffeomorphism everywhere if and only if: $$\forall (x,y),\partial_xF(x,y)\partial_yF(x,y)(\partial_xF(F(x,y),y)-\partial_yF(F(x,y),y)-1)\neq 0.$$ The expression is even factorized. Regarding, when it exists, the differentiel of the inverse, one has: $$\mathrm{d}_{G(x,y)}G^{-1}=(\mathrm{d}_{(x,y)}G)^{-1}=\frac{1}{\det(\textrm{Jac}_{(x,y)}G)}\begin{pmatrix}\partial_yF(x,y)&-\partial_yF(x,y)-\partial_yF(F(x,y),y)\\-\partial_xF(x,y)&\partial_xF(F(x,y),y)\partial_xF(x,y)\end{pmatrix}.$$ As the inverse of $2\times 2$ matrix is easily expressed.