When I substitute $x^2-x y+y^2$ with $u$ and $v$, I get $2u+2v$. This implies that the bounds of integration become $2u+2v=2$, which is a line. We haven't been taught that the jacobian is allowed to transform $\mathbb{R}^2$ to $\mathbb{R}^1$, so I think the problem is not as intended, and I don't think it would make sense to integrate this because the improper integral likely wouldn't converge.
Here's the question:
Evaluate $$ \iint_R (x^2 - xy + y^2) \, dA $$ by using the transformation $$ x = \sqrt{2u} - \sqrt{\frac{2}{3} v} , \quad y = \sqrt{2u} + \sqrt{\frac{2}{3} v} $$ where (R) is the region bounded by the ellipse (x^2 - xy + y^2 = 2).
$$ \begin{align*} 1. & \ -\frac{4\sqrt{3}}{3} \\ 2. & \ 0 \\ 3. & \ 1 \\ 4. & \ \frac{4\sqrt{3}}{3} \\ 5. & \ \frac{16\sqrt{3}}{9} \end{align*} $$