The triangle $ABC$, right angled at $C$, has median $AD$, $BE$ and $CF$. $AD$ lies along the line $y = x + 3$, $BE$ lies along the line $y = 2x + 4$. If the length of the hypotenuse is $60$, find the area of the triangle $ABC$.
Let $AC=b$, $BC=a$
$$a^2+b^2=60^2$$
$$\triangle_{ABC}=\dfrac{1}{2}ab\tag{1}$$
Let's take a look at $\angle AGB$
Case $1$: $\angle AGB=\theta$ is acute
$$\tan\theta=\left|\dfrac{1-2}{1+2}\right|$$ $$\tan\theta=\dfrac{1}{3}$$
Applying cosine rule in $\triangle_{AGB}$
$$\cos\theta=\dfrac{AG^2+GB^2-AB^2}{2AG\cdot GB}\tag{2}$$
As $G$ is centroid, so $AG=\dfrac{2}{3}AD$, $GB=\dfrac{2}{3}BE$
$$AD=\dfrac{1}{2}\sqrt{2b^2+2c^2-a^2}$$ $$AD=\dfrac{1}{2}\sqrt{4c^2-3a^2}\tag{3}$$
$$BE=\dfrac{1}{2}\sqrt{4c^2-3b^2}\tag{4}$$
$$\cos\theta=\dfrac{\dfrac{1}{9}\left(4c^2-3b^2+4c^2-3a^2\right)-c^2}{\dfrac{2}{9}\sqrt{4c^2-3a^2}\sqrt{4c^2-3b^2}}$$
$$\dfrac{6}{\sqrt{10}}=\dfrac{-4c^2}{\sqrt{4c^2-3a^2}\sqrt{4c^2-3b^2}}$$
This is not possible as $L.H.S$ is positive and $R.H.S$ is negative, so $\theta$ must have been obtuse. So $\cos\theta=\dfrac{-3}{\sqrt{10}}$
Hence above expression can be written as
$$\dfrac{3}{\sqrt{10}}=\dfrac{2c^2}{\sqrt{4c^2-3a^2}\sqrt{4c^2-3b^2}}$$
Squaring both sides
$$9(4c^2-3a^2)(4c^2-3b^2)=40c^4$$ $$9(16c^4-12b^2c^2-12a^2c^2+9a^2b^2)=40c^4$$ $$144c^4-108c^4+9a^2b^2=40c^4$$ $$9a^2b^2=4c^4$$ $$ab=\pm\dfrac{2}{3}c^2$$
As $a$ and $b$ are sides, so $ab$ will be positive, hence $ab=2400$
Hence, area of triangle is $\dfrac{1}{2}ab=1200$, but actual answer is $400$
You went wrong near the end. You got $$9(16c^4-12b^2c^2-12a^2c^2+9a^2b^2)=40c^2$$ We have $a^2+b^2=c^2$ so $$9(16c^4-12c^4+9a^2b^2)=40c^4$$ and hence $$81a^2b^2=(40-36)c^4$$ So $ab=\frac{2}{9}c^2=$ and the area is $60^2/9=400$.