I don't know how to do the change of variables:
$$\int_D {e^{f(x, y)} \left(g(x, y)\right)^2 \text{d}x \, \text{d}y},$$where $f,g:\mathbb {R^2}\to \mathbb {R}$ are two linear transformations such that $$\det\left(\frac{\nabla f(x,y)}{\nabla g(x,y)}\right)=3$$and $$D = \lbrace (x, y) \in \mathbb{R}^2 : 5 \leq f(x, y) \leq 8, -1 \leq g(x, y) \leq 2 \rbrace.$$
The hint is given in the domain $D$. Call $u=f(x,y)$ and $v=g(x,y)$. We have $${\rm d}u\,{\rm d}v = \frac{\partial (u,v)}{\partial (x,y)}\,{\rm d}x\,{\rm d}y =3 \,{\rm d}x\,{\rm d}y.$$ Since the change of variables is linear, $D$ corresponds to a rectangle in coordinates $(u,v)$. Hence $$\begin{align} \iint_D e^{f(x,y)}(g(x,y))^2\,{\rm d}x\,{\rm d}y &= \frac{1}{3}\int_{-1}^2\int_5^8 e^u v^2\,{\rm d}u\,{\rm d}v \\ &= \frac{1}{3} e^u\bigg|_5^8 \frac{v^3}{3}\bigg|_{-1}^2 \\ &= \frac{1}{3} (e^8-e^5)\left(\frac{8}{3}+\frac{1}{3}\right)\\ &= e^8-e^5.\end{align}$$