In my book, under the topic "Evaluation of Integrals by using Trigonometric Substitutions", it is given, in order to simplify integrals containing the expressions $\sqrt{\frac{x-\alpha}{\beta-x}}$ and $\sqrt{(x-\alpha)(\beta-x)}$, the substitution $x=\alpha\cos^2\theta+\beta\sin^2\theta$ must be used. If I remember this form and the substitution, then it definitely helps simplify the integrand. The first expression gets simplified to $\tan\theta$ and the second to $\sin\theta\cos\theta(\alpha-\beta)$.
I understand that if we do this kind of substitution we greatly simplify the expression. But how do we determine what to substitute in the first place or in other words, if I forget the substitution, is there any way to determine which substitution works well to simplify the integrand? How did the author figure out this substitution is the best fit for this kind of expression? Was it a guess or is there any mathematical reasoning behind it?
The following is intended as an elaboration of Yves Daoust's excellent answer. Given the form $$\sqrt{\frac{x-\alpha}{\beta-x}}$$ to be integrated, let's see what happens under the substitution $x=au+b$ (this is what is meant by a linear transform). We get the expression $$\sqrt{\frac{au+(b-\alpha)}{-au+(\beta-b)}}=\sqrt{\frac{u+(b-\alpha)/a}{-u+(\beta-b)/a}}.$$ In particular, if we choose the constants $a,b$ such that $b-\alpha=a$ and $\beta-b=a$, this transforms the integrand into $$\sqrt{\frac{1+u}{1-u}}=\frac{1+u}{\sqrt{1-u^2}}.$$ At this point, it becomes natural to substitute $u=\cos\theta$, since the Pythagorean identity $\sin^2\theta+\cos^2\theta=1$ now implies that the denominator $\sqrt{1-u^2}$ simply becomes $\sin\theta$, which simplifies the integrand greatly. Similarly, substituting $x=au+b$ into the integrand of the form $\sqrt{(x-\alpha)(\beta-x)}$ provides us with $$\sqrt{(-b^2+(\alpha+\beta)b-\alpha\beta)+u(2ab+a\alpha+a\beta)-a^2u^2}$$ which is the same as $$ a\sqrt{(-b^2+(\alpha+\beta)b-\alpha\beta)/a^2+u(2b+\alpha+\beta)/a-u^2}. $$ In substance this is just $\sqrt{A+Bu-u^2}$ for constants $A,B$, which after completing the square further becomes something that looks like $\sqrt{1-v^2}$ for an appropriate choice of $v$. Again, this makes the substitution $v=\cos\theta$ natural.
In both cases, if you work carefully through each step in the calculation you will find that it all amounts to the substitution $$x=\frac{\alpha+\beta+(\beta-\alpha)\cos\theta}{2},$$ which as Yves shows can also be written as $$x=\alpha\sin^2\frac\theta2+\beta\cos^2\frac\theta2.$$ (Note that the fact that the argument of the functions is off by a factor of $2$ from your substitution should not bother you, since it is just a constant and does not affect the efficiency of the substitution.)