Let $M = \begin{bmatrix} 5 & 3 \\ 3 & -6 \end{bmatrix}$ and let $T : \mathbb{R}^{2} \rightarrow \mathbb{R}^{2}$ be a linear transformation defined by $T(x) = Mx$. Let $\mathcal{C}$ be the unit circle in $\mathbb{R}^{2}$ and $T(\mathcal{C}$) be the resulting figure when $T$ is applied to $\mathcal{C}$. What is the area of $T(\mathcal{C})$?
This is what I did:
$$M^{-1} = -\frac{1}{39} \begin{bmatrix} -6 & -3 \\ -3 & 5 \end{bmatrix},$$
so $M$ maps the unit circle to
$$\left\{\begin{bmatrix} u \\v \end{bmatrix} = M\left(\begin{bmatrix} x \\ y \end{bmatrix} \right) \mid x^2 + y^2 = 1\right\}.$$
The equation of this set in the $u, v$ space is
$$\left|\left|M^{-1} \begin{bmatrix}u\\v \end{bmatrix} \right|\right|^{2} = 1. $$
I don't really know how to go on from here. I plugged in $M^{-1}$ and got nowhere to finding the area. Maybe I'm doing the wrong approach.
The area of the image is the area of the original object, multiplied by $|\det(M)|$, so in your case, that would be $|\det(M)|$.
See this answer for a detailed proof