I tried to rewrite the integral as $$\int (\cos^2x)(\cos^2x)dx$$ which in turn I used the trig identity of $\cos^2x = 1 - \sin^2x$ and therefore changed $$\int (\cos^2x)(\cos^2x)dx = \int (1 - \sin^2x)(1 - \sin^2x)dx = \int ( \sin^4x - \sin^2x + 1) dx$$
I then tried to do u substitution where $u=\sin x$ and $du=\cos x dx$
But my answer was completely wrong...I ended up getting $$\frac{\sin^5x}{5}+\sin x-\frac{2\cos^3x}{3}+K, \quad K\in \mathbb R$$
Am I doing $u$ substitution wrong or is there a better way to solve this integral?
If I were grading your work, I'd deduct a point for leaving the $dx$ and $du$ out of the integral. When you replace $dx$ by $du/\cos x$, you'll have a much different integrand. I don't think there's a $u$-substitution way to work this problem.
The usual way is to use the identity
$$\cos^2x = \frac{1+\cos 2x}{2}.$$
Replace each of your $cos^2 x$ and multiply. Then you'll have a $\cos^2 2x$ which you can replace using the same identity.