To solve the integrals of the form $(\sin^n(x)\cos^m(x))$, my book uses the sum of $m$ and $n$ to make appropriate substitution.
Eg:
$$\int \sec^3(x)\csc(x) \, dx$$
Sol:
sum of powers=$-4$ {even and negative}, co substitute $t=\tan(x)$
And the rest just becomes easy.
How did they decide to substitute $\tan x$? What is the intuition behind this?
On
Working Rule:
1.) If one of them is odd , then substitute for the term of even Power .
2.) if both are odd, substitute either of them .
3.) if both are even , use trigonometric identities only.
4.) if m and n are rational numbers and m+n-2/2 is a negative integer , then substitute cot X=P or tan X=P whichever is found suitable.
As we know that integrations of certain functions cannot be obtained directly if they are not in the standard form, but they may be reduced to standard forms by proper substitution.
Here, the integral is such that we cannot directly make a substitution rather it requires further more simplification in order to substitute any function. If the integral was such that $m, n \in \mathbb{N}$ then we would have substituted either $t=\sin x$ or $\cos x$ depending upon $m$ or $n$ is odd. See this illustration.
But if $m, n \in \mathbb{Q}$ and $m+n\in \mathbb{Z^-}$, we need to find another substitution and so we move on to $\tan x$-$\sec^2x$ pair. In order to change the integrand in terms of $\tan x$ and $\sec^2x$, we've to divide numerator and denominator by $\cos^kx$, where $k=-(m+n) $ and then substitute $\tan x=t$. See this* illustration.
In the first illustration it is clear that substituting $\tan x=t$ is of no use as $\sec^2x$ is not generated in the numerator efficiently by doing so. The substitution which we're making should be relevant and it should make the integrand more simplified rather than complicating it. That's why in the second case it is preferred the substitution of $\tan x=t$ over $\sin$ or $\cos$.
*Typo in the image. It should be "[Dividing $N^r$ and $D^r$ by $\cos^4 x$]".