How can we evaluate this difficult integral:
$$\int\frac{\tan^2x}{1-x\tan^2x}dx=\int\frac{1}{\cot^2x-x}dx$$
I tried using integration by parts but I suppose it doesn't work with this.
In the worst case, if this integral doesn't have a closed formula, how can we $prove$ that it cannot be found?
It is interesting to note that your integral telescopes, starting with
$$\int\frac{\tan^2x}{1-x\tan^2x}dx=\int\frac{\tan^2x}{1-x\tan^2x}\frac{(1+x\tan^2x)}{(1+x\tan^2x)}dx$$
Next multiply top and bottom by $1+x^2\tan^4x$ and so on. In this process the denominator tends to 1 (assuming the integral is between the limits of $x=0$ and $x=\pi/4$) and the numerator to the infinite series $S$
$$S=\tan^2 x+x\tan^4 x+x^2\tan^6x+x^3\tan^8x+x^4\tan^{10}x+...$$
Now anybody who claims there is a nice closed form to be found can be given the task of integrating these terms one by one until all faith in this proposition runs dry.
I understand that this is not a proof, but all you asked for was a way of demonstrating the difficulty with this integral.