The solutions for this question can be seen here: https://i.stack.imgur.com/3NvXC.jpg
I don't understand how the solutions knew that the transformation matrix was what it has stated above. How did they get \begin{bmatrix} 3 & 4 \\ -2 & 1 \end{bmatrix}
The determinant of the matrix is $11$. That is the number you need to multiply the original area by to get the area of the new triangle. So the area of the original triangle is $165/11= 15$. On the other hand it is $5\cdot k/2$ ($1/2$ base times hight). so $k=6$.