I need to find the integral of $\int x\sec(2x)dx $, but if I use integration by parts, I get an answer of $\int x\sec(2x)dx =0 $, which I know isn't correct.
First I let $u=\sec(2x)$ and $v'=x$. This gave me $$\frac{2x\sec(2x)}{2}−\int x\sec(2x)\tan(2x)dx$$ Then I let $u=x^2$ and $v'=\sec(2x)\tan(2x)$ which gave $$\frac{2x\sec(2x)}{2}−\int x\sec(2x)dx$$ Putting both together yields $$\int x\sec(2x)=\frac{2x\sec(2x)}{2}−\frac{2x\sec(2x)}{2}−\int x\sec(2x)$$ As you can see, everything cancels out, and I'm left with $0$.
I have a deep feeling this is a typo from wherever you got this problem from. I am guessing what was meant was $$ \int x \sec^2 x \;dx, $$ which you should not have too much problem with given your work on the given problem. The given problem is a hot nightmare to integrate, as you can see at this link.