I followed what I was taught and ended up somewhere not nice. Where did I go wrong?
$$ \int \tan^2(4x)\sec(4x)\,dx = I $$
\begin{align} u &= \tan(4x) & dv &= \tan(4x)\sec(4x)\,dx \\ du &= 4\sec^2(4x)\,dx & v &= \frac{\sec(4x)}{4} \end{align}
$$ \frac{\tan(4x)\sec(4x)}{4} - \int \sec^3(4x)\,dx $$
We've also only been doing trig sub and integration by parts so that's the only 'methods' we're allowed to use. We did a question like this in class and we ended with a "$\cdots - I$" on the right side so when we cancel it out, it would become $2I$ on the left side. But in this case if we cancel out the $I$ on the left, it would cancel the $I$ on the right and it would just fail.
I can't seem to figure out where I went wrong.

You basically undid your integration by parts if you closely inspect your two IBPs. This is a very real danger with integrating by parts when it is not immediately resolved.
Let's continue from your third line. Let us let $\sec^2(4x) = 1+\tan^2(4x)$, then we get
$$ I = \frac{\tan(4x)\sec(4x)}{4} - \int \sec(4x)(1+\tan^2(4x))\,dx$$
which then becomes
\begin{align} I &= \frac{\tan(4x)\sec(4x)}{4} - \int \sec(4x)\,dx - \int \tan^2(4x)\sec(4x)\,dx \\ &= \frac{\tan(4x)\sec(4x)}{4} - \int \sec(4x)\,dx - I. \end{align}
Can you take it from here?