$$\int _\frac{3}{4}^{\frac{3}{2}} \sqrt{9 - 4x^2}$$
First I set $x = \frac{3}{2}\sin\theta, dx = \frac{3}{2}\cos \theta d\theta$:
$$\int \sqrt{9 - 4x^2}dx = \int \sqrt{4\left(\frac{9}{4} - x^2\right)}dx = 2\int \sqrt{\left(\frac{3}{2}\right)^2 - \left(\frac{3}{2}\right)^2\sin^2\theta}*\frac{3}{2}\cos \theta$$
$$3\int \frac{3}{2}\cos^2\theta d\theta = \frac{9}{2}\int \frac{\cos 2\theta + 1}{2}$$
$$\frac{9}{2}\left(\int \frac{1}{2} + \int \cos 2\theta \right) = \frac{9}{2}\left(\frac{1}{2}\theta + \frac{1}{4}\sin2\theta\right) = \frac{9}{2}\left(\frac{1}{2}\arcsin\left(\frac{2}{3}x\right) + \frac{1}{4}\sin\left(2\arcsin\left(\frac{2}{3}x\right)\right)\right)$$
The problem is evaluating the definite integral, it's getting way too messy. How do I go about this?
When you substituted $x = \dfrac 32 \sin \theta$, you could have changed the limits of integral then, according to your new variable, i.e. $\theta$.
The new limits will be $$ \text{Upper limit :} \quad x = \frac{3}{4} =\frac 32 \sin \theta \implies \theta =\dfrac{\pi}{6}$$ and $$\text{Lower limit :} \quad x = \frac{3}{2} =\frac 32 \sin \theta \implies \theta =\dfrac{\pi}{2}$$
Now use these limits in the following expression you got, to evaluate the integral $$\rm I=\frac{9}{2}\left(\frac{1}{2}\theta + \frac{1}{4}\sin2\theta\right) \Bigg |_{\pi/6}^{\pi/2}$$