In my textbook, the author states that $$ \iint_R f(r\cos\theta,r\sin \theta)r\,dr\,d\theta =\iint_{G(R)} f(x,y)\,dy\,dx, $$ where $G(r,\theta)=(r\cos\theta, r\sin \theta)$ and he says that this is the formula for the change of variables between rectangular coordinates and polar ones.
But for me, it doesn't look right, if we want to change coordinates then the formula should be like this $$ \iint_R f(x,y)\,dy\,dx=\iint_{G(R)} f(r\cos\theta,r\sin \theta)r\,dr\,d\theta. $$
Am I right? or are both formulas the same thing?
This is a picture from the textbook, it says that the map $G$ takes rectangular to polar

No, of course both cannot be right. The formula in the textbook is right. The way I "remember" the formula is that if $(r,\theta)\in R$ then this is my "polar coordinate space" so $(x,y)=(r\cos\theta,r\sin\theta)=G(r,\theta)$ are the corresponding cartesian coordinates so they have to lie in $G(R)$.
In general (under appropriate hypotheses) the change of variables formula is $$ \int_{G(R)} f(\xi)\,d\xi = \int_{R}f(G(\zeta))\cdot \left|\det DG(\zeta)\right|\,d\zeta $$ Here, we interpret it as $\zeta\in R$ and the "substitution" is $\xi=G(\zeta)$ so we must have $\xi\in G(R)$.