How to evaluate $$\int_{0}^{\frac{\pi}{2}} \frac{\cos(x)}{(1 + \sqrt{\sin(2x)})^n} \,dx$$
My attempt
The transformation of $x \rightarrow \frac{\pi}{2}-x$ yields
$$ \int_{0}^{\frac{\pi}{2}} \frac{\cos(x)}{(1 + \sqrt{\sin(2x)})^n} \,dx = \int_{0}^{\frac{\pi}{2}} \frac{\sin(x)}{(1 + \sqrt{\sin(2x)})^n} \,dx$$
$$ \int_{0}^{\frac{\pi}{2}} \frac{\cos(x)}{(1 + \sqrt{\sin(2x)})^n} \,dx = \frac{1}{2} \int_{0}^{\frac{\pi}{2}} \frac{\cos(x) + \sin(x)}{(1 + \sqrt{\sin(2x)})^n} \,dx = \frac{1}{2} \int_{0}^{\frac{\pi}{2}} \frac{\sqrt{(\cos(x) + \sin(x))^2}}{(1 + \sqrt{\sin(2x)})^n} \,dx $$
$$= \frac{1}{2} \int_{0}^{\pi/2} \frac{\sqrt{1 + \sin(2x)}}{\left(1 + \sqrt{\sin(2x)}\right)^n} \,dx = \int_{0}^{\frac{\pi}{4}} \frac{\sqrt{1 + \sin(2x)}}{\left(1 + \sqrt{\sin(2x)}\right)^n} \,dx = \frac{1}{2} \int_{0}^{\frac{\pi}{2}} \frac{\sqrt{1 + \sin(x)}}{\left(1 + \sqrt{\sin(x)}\right)^n} \,dx$$
Mathematica 14 gives, for $n > 1$,
\begin{align} I_n&=\int_0^\frac{\pi}{2} \frac{\cos(x)}{(1+\sqrt{\sin(2x)})^n}\,dx\\ &= \frac{2 n \, {_2}F{_1}\left(1,n+\frac{1}{2};\frac{3}{2};-1\right)-1}{n-1} \end{align}
in which ${_2}F{_1}$ is the Gauss hypergeometric function, and for $n=1$,
$$ I_1=\frac{\pi}{2}-1$$
This can be further simplified as follows. Using Euler's transformation [Olver et al. NIST Handbook 2010 edition, eq. 15.8.1, p.390]:
$${_2}F{_1}\left(1,n+\frac{1}{2};\frac{3}{2};-1\right) = 2^{-n}\;{_2}F{_1}\left(\frac{1}{2},1-n;\frac{3}{2};-1\right)$$
Gauss hypergeometric functions that have a negative integer as one of the first two parameters are polynomials. The integral is therefore compactly represented this way, but computationally very simple.
If you want to use more common functions (which I think is seldom the best thing to do), just notice that
$$\left(z \frac{d}{dz}z\right)^n (z^{c-a-1}(1-z)^{a+b-c}\,{_2}F{_1}(a,b;c; z)) = (c-a)_n z^{c-a+n-1}(1-z)^{a-n+b-c}\,{_2}F{_1}(b, a-n;c;z) \tag{2}$$
where $(d)_n=\frac{\Gamma(d+n)}{\Gamma(d)}$ is the ascending Pochhammer symbol.
Setting $a=1, \, b=\frac{1}{2}, c=\frac{3}{2}$ in $(2)$ and noting that (NIST 2010, eq. 15.4.3)
$$ {_2}F{_1}(1, \frac{1}{2}; \frac{3}{2}; z) = \frac{\tanh ^{-1}\left(\sqrt{z}\right)}{\sqrt{z}}$$
we therefore obtain:
$${_2}F{_1}\left(1,n+\frac{1}{2};\frac{3}{2};-1\right) = i\frac{(-1)^{n}}{2^n\left(\frac{1}{2}\right)_n} \left(z \frac{d}{dz}z\right)^n_{z=-1} \left( z^{-1}\, \tanh ^{-1}\left(\sqrt{z}\right)\right)$$
Numerical verifications
Except for $n=1$ the proof does not look trivial. Here is my take in the general case. I will just give a sketch of a possible proof, which looks feasable even though it has not been completed to the very end (and calculations remain to be checked).
Setting:
$$J_n(a, x) = \frac{\cos(x)}{(\sqrt{\sin(2x)}-a)^n }$$
and taking the Mellin transform with respect to $a$ at $v$ gives for $n > 1$:
$$\mathscr{M}(J_n(.,x); v) = \frac{(-1)^v}{\Gamma (n)}\Gamma (v) \Gamma (n-v) \cos (x)\sin^{\frac{v-n}{2}}(2 x) \tag{MT}$$
Integrating this Mellin transform between $0$ and $\pi/2$ with respect to $x$ is a much easier issue (granting the interchange of integrals, this having to be checked).
Using formula 2.5.12.36 of Prudnikov et al., Integral and Series, Vol. 1, p.403:
$$ \int_0^\pi \cos(bx)\sin^{\mu-1}(x) \,dx = \frac{2^{1-\mu}\pi }{\mu B((\mu+b+1)/2, (\mu-b+1)/2)}\cos(b\pi/2),$$
in which $B$ is the beta function. With $b=\frac{1}{2}, \mu = \frac{v-n}{2}+1$, and a change of variables $y=2x$, this formula yields:
$$ \int_0^{\pi/2} \cos (x)\sin^{\frac{v-n}{2}}(2 x)\,dx =\frac{\sqrt{\pi } \Gamma \left(\frac{1}{2} (-n+v+2)\right)}{2\Gamma \left(\frac{1}{2} (-n+v+3)\right)}\qquad \text{ (if }n<\Re(v)+2 \text{)}$$
Positing
$$\Theta_n(v)=\frac{\sqrt{\pi }}{2(n-1)!} \frac{ \Gamma (v) \Gamma \left(1-\frac{n}{2} + \frac{v}{2}\right) \Gamma (n-v)}{ \Gamma \left(\frac{3-n}{2} + \frac{v}{2}\right)}$$
we have:
$$\mathscr{M}(J_n(.,x); v) = \Theta_n(v) (-1)^{-v} \tag{3}$$
and taking the inverse Mellin transform of $(3)$ in the variable $v$ at point $a$ to recognize the Mellin-Barnes expression $I_n(a)$ (for some vertical countour path $L$ adequately chosen)
$$I_n(a) = \frac{1}{2\pi\,i} \int_L \Theta_n(v) (-a)^{-v} dv$$
of a Fox-Wright H function evaluated at $-a$, with an adequate factor only depending on $n$:
\begin{align}I_n(a) &= \frac{\sqrt{\pi }}{2(n-1)!} H^{2,\,1}_{2,\,2}\left[ ^{(1-n,\, 1),\, (\frac{3-n}{2},\,\frac{1}{2})} _{(0,\,1),\, (1-\frac{n}{2},\,\frac{1}{2})}; -a\right] \\ &= \frac{\sqrt{\pi }}{2(-a)^n (n-1)!} H^{2,\,1}_{2,\,2}\left[ ^{(1,\, 1),\, (\frac{3}{2},\,\frac{1}{2})} _{(n,\,1),\, (1,\,\frac{1}{2})}; -a\right]\\ &=\frac{\sqrt{\pi }}{2(-a)^n (n-1)!} H^{1\,2}_{2,\,2}\left[ _{(0,\, 1),\, (-\frac{1}{2},\,\frac{1}{2})} ^{(1-n,\,1),\, (0,\,\frac{1}{2})}; -a^{-1}\right] \tag{4}\end{align}
(Mathay, Saxena, Haubold, The H-Function, eqs. 1.58, 1.60, p. 12, Springer, 2010)
Convergence conditions mentioned in this reference book are met (ibid. p.3), except when $a \in \mathbb{R}^+$.
We now recognize in $(4)$ the form of a generalized Wright Psi function, which is computationally simpler than the standard H-function (ibid. eqs. 1.90, 1.91, 1.140);
\begin{align} I_n(a) &= \frac{\sqrt{\pi }}{2(-a)^n (n-1)!} {_2}\Psi{_1} \left[ _{ (\frac{3}{2},\,\frac{1}{2})} ^{(n,\,1),\, (1,\,\frac{1}{2})}; a^{-1}\right] \end{align}
The Wright Psi function should be understood as defined by the Mellin-Barnes integral (MBI) and not by its alternative series definition
$$\sum_{k=0}^\infty \frac{\Gamma(n+k)\Gamma\left(1+\frac{k}{2}\right)}{\Gamma\left(\frac{3}{2}+\frac{k}{2}\right)}\frac{a^{-k}}{k!}$$
which unfortunately does not absolutely converge for $|a|=1$ (it would do so for $|a| > 1$, which points to the tricky nature of the posted question).
The case at hand is a corner case that only happens to be defined by MBI magic and using a specially fine-tuned integration contour ($L_{i\gamma\infty}$ of ibid. eq. 1.206, p.32) and, with this method, for $a \notin \mathbb{R}^+$, which happens to be the case ($a=-1$).
Numerical verification (using the
FoxHimplementation of Mathematica 14)[Edit] Reduction of the results to simpler forms
It suffices to apply Legendre's duplication formula to the numerator Gamma factors with integral multiplicative coefficient in $(4)$ as follows:
\begin{align} H^{1\,2}_{2,\,2}\left[ _{(0,\, 1),\, (-\frac{1}{2},\,\frac{1}{2})} ^{(1-n,\,1),\, (0,\,\frac{1}{2})}; -a^{-1}\right] &= \frac{1}{2\pi i}\int_L \frac{\Gamma(s)\Gamma(n-s)\Gamma(1-\frac{s}{2})}{\Gamma(\frac{3}{2}-\frac{s}{2})} (-a)^s ds \\ &= \frac{2^{n-3}}{\pi^2 i}\int_L \frac{\Gamma(\frac{s}{2})\Gamma(\frac{1}{2}+\frac{s}{2}) \Gamma(\frac{n}{2}-\frac{s}{2}) \Gamma(\frac{n+1}{2}-\frac{s}{2}) \Gamma(1-\frac{s}{2})}{\Gamma(\frac{3}{2}-\frac{s}{2})} (-a)^s ds \\ &= \frac{2^{n-2}}{\pi}\,H^{2,\,3}_{3,\,3}\left[ _{(0\,\frac{1}{2}),\, (\frac{1}{2},\,\frac{1}{2}), (-\frac{1}{2},\,\frac{1}{2})} ^{(1-\frac{n}{2},\,\frac{1}{2}),\, (\frac{1-n}{2},\,\frac{1}{2}),\, (0,\,\frac{1}{2})}; -a^{-1}\right] \\ &=\frac{2^{n-1}}{\pi}\,G^{2,\,3}_{3,\,3}\left[ _{0,\frac{1}{2},-\frac{1}{2}} ^{1-\frac{n}{2},\frac{1-n}{2},\, 0}; \frac{1}{a^2}\right]\tag{6} \end{align} (as $\Re a < 0$ see reference; and applying Legendre's duplication formula.) So that from $(4)$ and $(6)$, the representation of $I_n(a)$ is markedly reduced as a first step to:
$$ I_n(a) = \frac{2^{n-2}}{\sqrt{\pi}(-a)^n (n-1)!} \,G^{2,\,3}_{3,\,3}\left[ _{0,\frac{1}{2},-\frac{1}{2}} ^{1-\frac{n}{2},\frac{1-n}{2},\, 0}; \frac{1}{a^2}\right]$$
This formula is defined for $a=- 1$. For $\Re a < -1$ interesting simplifications obtain using this reduction formula, so that, after some algebra (and verifying that the conditions are met that the first two coefficients of each line do not differ by an integer; note: we now have supposed $\Re a<0$):
$$ I_n(a) = \frac{1}{4} (-a)^{-n} \left(\frac{\pi n}{a} \; _2F_1\left(\frac{n+1}{2},\frac{n+2}{2};2;\frac{1}{a^2}\right) + 4 \, _3F_2\left(1,\frac{n}{2}+\frac{1}{2},\frac{n}{2};\frac{1}{2},\frac{3}{2};\frac{1}{a^2}\right)\right)\tag{7}$$
This already nice result can be further improved noting that the first Gauss function follows a quadratic transformation (Olver et al. NIST, 2010, eq. 15.8.13 p.291):
\begin{align} _2F_1\left(\frac{n+1}{2},\frac{n+2}{2};2;\frac{1}{a^2}\right) &= \left(\frac{a-1}{a+1}\right)^\frac{3}{2}\,\left(\frac{a}{a-1}\right)^{n+1} {_2}F_1\left(2-n,\frac{3}{2}; 3;\frac{2}{1+a}\right) \tag{7'} \end{align}
For $\Re a < -1, n \geq 2$, the last Gaussian function is therefore zero or a polynomial of degree $n-2$ in the variable $\frac{2}{1+a}$. Similar considerations can be invoked to trim down computation times of the $_3F_2$ function.
(Contrary to what was stated in previous edits of this answer, the above results apply to the particular case $n=1$.)
Note that for $n=2$ or $n=3$ the $_3F_2$ part is actually a Gauss hypergeometric function. This can be generalized as follows, using the known finite expansion of the ${_3}F{_2}$ function (Prudnikov et al., Integral and Series, Vol. 3, p.497, eq. 7.4.1.2):
\begin{align} {_3}F{_2}(a,b,c;a-p,d;z) &=\frac{1}{(1-a)_p}\, \sum_{k=0}^p (-1)^k \binom{p}{k} (1-a)_{p-k} \frac{(b)_k(c)_k}{(d)_k} {_2}F_{1}(b+k, c+k; d+k; z) z^k \\ &= \frac{1}{(1-a)_p}(1-z)^{d-b-c}\, \sum_{k=0}^p (-1)^k \binom{p}{k} (1-a)_{p-k} \frac{(b)_k(c)_k}{(d)_k}\,\times {_2}F_{1}(d-b, d-c; d+k; z) \left(\frac{z}{1-z}\right)^k, \end{align} by Euler's hypergeometric transformation. For $n$ even, setting $p=\frac{n}{2}$, $a=\frac{n+1}{2}$, $b=\frac{n}{2}$, $c=1$, $d=\frac{3}{2}$ and inserting in the above equation gives:
\begin{align} _3F_2\left(1,\frac{n}{2}+\frac{1}{2},\frac{n}{2};\frac{1}{2},\frac{3}{2};\frac{1}{a^2}\right) &= \frac{\left(\frac{n}{2}\right)!}{\sqrt{\pi}}\left(\frac{a}{\sqrt{a^2-1}}\right)^{n-1}\, \sum_{k=0}^\frac{n}{2} (-1)^k \frac{\Gamma\left(\frac{1}{2}-k\right)\, (\frac{n}{2})_k}{\left(\frac{n}{2}-{k}\right)!(\frac{3}{2})_k}\,\times {_2}F_{1}(\frac{3-n}{2}, \frac{1}{2}; \frac{3}{2}+k; \frac{1}{a^2}) \left(\frac{1}{a^2-1}\right)^k \tag{8} \end{align} A similar finite expansion can be obtained when $n$ is odd. Inserting in $(7)$ now yields $I_n(a)$ as a finite sum of Gauss hypergeometric functions with at most $[n/2] + 2$ terms.
The limiting property when $a\to 1^+$ can be obtained as follows. Each Gauss function coefficient in the above expansion will tend to its Gauss theorem value:
$$ {_2}F_{1}(\frac{3-n}{2}, \frac{1}{2}; \frac{3}{2}+k; \frac{1}{a^2}) \to \frac{\Gamma\left(\frac{3}{2}+k\right)\Gamma(k+\frac{n-1}{2})}{\Gamma\left(k+\frac{n}{2}\right)\,k!} = \frac{\left(\frac{3}{2}\right)_k \Gamma(k+\frac{n-1}{2})}{\left(\frac{n}{2}\right)_k k!} \frac{\Gamma\left(\frac{3}{2}\right)}{\Gamma\left(\frac{n}{2}\right)}\tag{9}$$
Simplifications ensue in the limit by inserting $(9)$ in $(8)$ so that from $(8)$ and $(9)$, we get when $z=a^{-2} \to 1$:
\begin{align} _3F_2\left(1,\frac{n}{2}+\frac{1}{2},\frac{n}{2};\frac{1}{2},\frac{3}{2};z\right) &\sim \frac{n}{4} (1-z)^{\frac{1-n}{2}} \sum _{k=0}^{\frac{n}{2}} \frac{(-1)^k \Gamma \left(\frac{1}{2}-k\right) \Gamma \left(\frac{n-1}{2}+k\right)}{k!\, \Gamma \left(-k+\frac{n}{2}+1\right)}\,\left(\frac{z}{1-z}\right)^k \\ &= \frac{\sqrt{\pi } n (1-z)^{\frac{1-n}{2}} \Gamma \left(\frac{n-1}{2}\right) \, _2F_1\left(-\frac{n}{2},\frac{n-1}{2};\frac{1}{2};\frac{z}{z-1}\right)}{4 \Gamma \left(\frac{n}{2}+1\right)} \\ &=\frac{\sqrt{\pi } n (1-z)^{\frac{1}{2}-n} \Gamma \left(\frac{n-1}{2}\right) \, _2F_1\left(-\frac{n}{2},1-\frac{n}{2};\frac{1}{2};z\right)}{4 \Gamma \left(\frac{n}{2}+1\right)} \tag{10} \end{align} in which the hypergeometric function is clearly a polynomial.
Note that the rational character of solutions (aptly shown by elementary means by another anwer) is an immediate consequence of the definition of hypergeometric functions.
Numerical verifications