Compute $$\iint_D(-2x-y)\cos((x-y)(-2x-y)) \ dxdy$$
Where $D$ has vertices at the points
$$P_1=\left(-1 + \frac{\pi}{6} , 2+\frac{\pi}{6} \right)$$
$$P_2=\left(-2 + \frac{\pi}{6} , 4+\frac{\pi}{6} \right)$$
$$P_3=\left(-2 + \frac{\pi}{4} , 4+\frac{\pi}{4} \right)$$
$$P_4=\left(-1 + \frac{\pi}{4} , 2+\frac{\pi}{4} \right)$$
I think I'm very close, but for the life of me I can't figure why I get the incorrect answer. Here is what I've done:
Plotting the points I get
The short lines, from left to right are $y=x+6$ and $y=x-3$. The long lines are $y=-2x+\pi/2$ and $y=-2x+3\pi/4.$ So I have that
\begin{array}{lcl} -6 \leq &x-y& \leq 3 \\ -\frac{3\pi}{4} \leq &-2x-y& \leq -\frac{\pi}{2} \end{array}
Setting $u=x-y$ and $v=-2x-y$ and solving the system I get
$$\left\{ \begin{array}{rcr} x & = & \frac{1}{3}u-\frac{1}{3}v \\ y & = & -\frac{2}{3}u-\frac{1}{3}v \\ \end{array} \right.$$
$\implies |J(u,v)|=1/3.$ Thus:
$$\iint_D(-2x-y)\cos((x-y)(-2x-y)) \ dxdy=\iint_Ev\cos{(uv)}|J(u,v)|\ du dv$$
$$= \frac{1}{3}\int_{-6}^{3}v\left(\int_{-3\pi/4}^{-\pi/2}\cos{(uv)} \ du\right) \ dv=\frac{2(3-\sqrt{2})}{9\pi}.$$
Where is the error?
(Edited in response to OP's comment.)
Exact evaluation of the edited integral
You have correctly set up the relationship between $x,y$ and $u,v$.
\begin{align} \begin{bmatrix}u \\ v\end{bmatrix} &= \begin{bmatrix}1 & -1 \\ -2 & -1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} \\ \begin{bmatrix} x \\ y \end{bmatrix} &= \begin{bmatrix}1 & -1 \\ -2 & -1 \end{bmatrix}^{-1} \begin{bmatrix} u \\ v \end{bmatrix} = \begin{bmatrix}1/3 & -1/3 \\ -2/3 & -1/3 \end{bmatrix} \begin{bmatrix} u \\ v \end{bmatrix} \end{align}
$|J(u,v)|=1/3$ is correct.
This major mistake is that you're integrating in much larger domain because there's a sign error in the boundary of $D$. In fact,
\begin{align} D &= \left\{(x,y) \,\Bigg|\, -6 \leq x-y \leq \color{red}{-3}, -\frac{3\pi}{4} \leq -2x-y \leq -\frac{\pi}{2} \right\} \\ &= \left\{(u,v) \mid -6 \le u \le -3, -\frac{3\pi}{4} \leq v \leq -\frac{\pi}{2} \right\} \\ &\quad \iint_D(-2x-y)\cos((x-y)(-2x-y)) \ dxdy \\ &= \iint_D v\cos(uv) |J(u,v)| \ du dv \\ &= \int_{-3\pi/4}^{-\pi/2} \left( \int_{-6}^{-3} \frac13 v\cos(uv) \ du \right) dv \\ &= \int_{-3\pi/4}^{-\pi/2} \frac13 \Big[ \sin(uv) \Big]_{u=-6}^{u=-3} \ dv \\ &= \frac13 \int_{-3\pi/4}^{-\pi/2} (\sin(6v) - \sin(3v)) \ dv \\ &= \frac13 \Big[ \frac{\cos(3v)}{3} - \frac{\cos(6v)}{6} \Big]_{-3\pi/4}^{-\pi/2} \\ &= \frac{1}{18} \Big[ 2\cos(3v) - \cos(6v) \Big]_{-3\pi/4}^{-\pi/2} \\ &= \frac{1}{18} \Big[ -2 \cos \left( 3 \cdot \frac{3\pi}{4} \right) - \cos \left( 6 \cdot \frac\pi2 \right) \Big] \\ &= -\frac{\sqrt2-1}{18} \end{align}
Exact evaluation of the original integral
\begin{align} D &= \left\{(x,y) \,\Bigg|\, -6 \leq x-y \leq \color{red}{-3}, -\frac{3\pi}{4} \leq -2x-y \leq -\frac{\pi}{2} \right\} \\ &= \left\{(u,v) \mid -6 \le u \le -3, -\frac{3\pi}{4} \leq v \leq -\frac{\pi}{2} \right\} \\ &\quad \iint_D(-2x-y)\cos((-x-y)(-2x-y)) \ dxdy \\ &= \iint_D v\cos(\color{red}{(\frac{u}{3} + \frac{2v}{3})}v) |J(u,v)| \ du dv \\ &= \int_{-3\pi/4}^{-\pi/2} \left( \int_{-6}^{-3} \frac13 v\cos((\frac{u}{3} + \frac{2v}{3})v) \ du \right) dv \\ &= \int_{-3\pi/4}^{-\pi/2} \left[ \sin((\frac{u}{3} + \frac{2v}{3})v) \right]_{u=-6}^{u=-3} \ dv \\ &= \int_{-3\pi/4}^{-\pi/2} \left( \sin((\frac{2v}{3}-1)v) - \sin((\frac{2v}{3}-2)v) \right) \ dv \end{align}
Note that you make a substitution error when rewriting $\cos(\dots)$.
Compare the results from Wolfram Alpha