Recently I gave an exam for Physics masters entrance where they asked the following question
Let $(x, y)$ denote the coordinates in a rectangular Cartesian coordinate system $\mathcal{C}$. Let $(x', y')$ denote the coordinates in another coordinate system $\mathcal{C'}$ defined by: $$\begin{align}x' &= 2x + 3y\\ y' &= -3x + 4y\end{align}$$ The area element in $\mathcal{C'}$ is: $$(\text{A})\ \frac{1}{17}\ dx'dy'\quad\quad(\text{B})\ 12\ dx'dy'\quad\quad(\text{C})\ dx'dy'\quad\quad(\text{D})\ x'\ dx'dy'$$
To solve it, I first calculated the Jacobian and found it to be 17. But, then I saw that the question asked for the area element in the C' coordinate system, so I thought that the answer would simply be $dx'dy'$ which I gave as my answer.
But, today the answer keys came out and it says that the answer is $\frac 1{17}dx'dy'$
Where did I go wrong/Is the answer key wrong?
The answer key is not wrong (the answer is not necessarily $dx'~dy'$ because the transformed coordinate system is not rectangular). The change of variables formula for a transformation $x=g(x',y')$ and $y=h(x',y')$ is given by $$\iint_R f(x,y)~dx~dy=\iint_S f(g(x',y'),h(x',y'))\left|\frac{\partial(x,y)}{\partial(x',y')}\right|~dx'~dy'.$$ Therefore you have to invert your coordinate transformation to get the correct answer. With this, one has that $$x=\frac{4x'-3y'}{17},\quad y=\frac{3x'+2y'}{17},$$ which results in $$\left|\frac{\partial(x,y)}{\partial(x',y')}\right|=\frac{1}{17}.$$ But in your simple case, you could of course use the fact that $$\left|\frac{\partial(x,y)}{\partial(x',y')}\right|=1/\left|\frac{\partial(x',y')}{\partial(x,y)}\right|,$$ which follows from the fact that the determinant of the inverse of a matrix is the reciprocal of the determinant and the inverse function theorem.