Consider the following double-integral:
$$ \iint_{[0,a]\times[0,a]} (x-y)^2 dxdy \stackrel{*}{=} \int_0^a \int_0^a (x-y)^2 dxdy = \frac{a^4}{6} $$
where $*$ follows by iterated integration. Suppose instead we consider the substitution $w=x-y, z=x+y$. The Jacobian of this transformation is $1/2$, and so
$$ \iint_{[0,a]\times[0,a]} (x-y)^2 dx dy = \frac{1}{2} \iint_{R_2} w^2 dw dz =\frac{1}{2} \int_{Y_1}^{Y_2} \left (\int_{X_1}^{X_2} dz\right )w^2 dw. $$ My question is: how do I go about solving this via iterated integration (with respect to $z$ first, then with respect to $w$), or more specifically, what should the limits of integration $Y_1,Y_2,X_1,X_2$ be?
If we consider the transformation $(x,y)\mapsto (w,z)$ as a linear map
$$ A = \begin{bmatrix}1 & -1 \\ 1 & 1\end{bmatrix} $$
then this transforms the square $[0,a]\times [0,a]$ into the parallelogram with vertices: $(0,0), (a,a),(0,2a), (-a,a)$, which is the region bounded by the lines: $$ y=x, \quad y=x+2a, \quad y=-x+2a, \quad y=-x, $$
or equivalently by the lines $$ w=0, \quad w=-2a, \quad z=0, \quad z=2a, $$ and so $$ R_2 = \{(w,z) : -2a \le w \le 0, ~ 0\le z \le 2a \}. $$ This is not a rectangular region in $(w,z)$-space, but clearly it is not correct to take $Y_1 = -2a, Y_2=0, X_1 = 0, X_2=2a$, since it does not lead to the answer found by direct integration mentioned in the first display. My confusion is about how to relate $z$ and $w$ and choose the inner integral limits properly.
So, $w$ can take any value from $-a$ to $a$ and
(see the picture below).
So, your integral is$$\frac12\left(\int_{-a}^0\int_{-w}^{w+2a}w^2\,\mathrm dz\,\mathrm dw+\int_0^a\int_w^{2a-w}w^2\,\mathrm dz\,\mathrm dw\right),$$which turns out to be equal to $\dfrac{a^4}6$ (as it should be).