$$(\cos^2x+y\sin2x)y'+y^2=0 \\ (\cos^2x+y\sin2x)dy+y^2dx=0 \\ \frac{\partial}{\partial x}(\cos^2x+y\sin2x)=-2\sin2x+2y\cos2x \\ \frac{\partial}{\partial y}y^2=2y $$
This equation is inexact, so I need to find an integrating factor to make it exact.
Let's try an integrating factor $\mu(x)$. We have the following equation (the condition for an exact differential). $$ \frac{\partial}{\partial x}[(\cos^2x+y\sin2x)\mu]=\frac{\partial}{\partial y}[y^2\mu] $$ After some rearraning we arrive at: $$\frac{1}{\mu}\frac{d\mu}{dx}=\frac{2y+2\sin2x-2y\cos2x}{\cos^2x+y\sin2x}$$ The right hand side of this equation is not a function of $x$ only - it also contains $y$ terms that won't cancel out. My textbook states that the integrating factor is $\sec^2(x)$, but the mutlivariable expression for $\frac{1}{\mu}\frac{d\mu}{dx}$ above would show otherwise. However, this integrating factor does work, so where am I going wrong?
Note that $$ \frac{\partial}{\partial x} \cos^2 x = 2 \cos x (-\sin x) = -\sin 2x $$ and not $-2 \sin 2x$. After correcting this calculation error, you will get $$ \frac{1}{\mu}\frac{d\mu}{dx} = 2 \tan x. $$ After integration, you'll get the right answer.