Given for $\alpha>1$ $$\int^{\pi}_0 \frac{dx}{\alpha-\cos x}=\frac{\pi}{\sqrt{\alpha^2-1}}$$
Prove that $$\int^{\pi}_0 \ln \left(\frac{b-\cos x}{a- \cos x}\right)dx=\pi \left(\frac{b+\sqrt{b^2-1}}{a+\sqrt{a^2-1}}\right)$$ if $a, b>1$
Given for $\alpha>1$ $$\int^{\pi}_0 \frac{dx}{\alpha-\cos x}=\frac{\pi}{\sqrt{\alpha^2-1}}$$
Prove that $$\int^{\pi}_0 \ln \left(\frac{b-\cos x}{a- \cos x}\right)dx=\pi \left(\frac{b+\sqrt{b^2-1}}{a+\sqrt{a^2-1}}\right)$$ if $a, b>1$
On
Since one posted solution relied on, and one comment discussed the real analysis methodology of differentiating under the integral, I thought it might be instructive to present a way forward that relies on contour integration. To that end, we proceed.
Let $I(a,b)$ be the integral given by
$$\begin{align} I(a,b)&=\int_0^\pi \log\left(\frac{b-\cos(x)}{a-\cos(x)}\right)\,dx\\\\ &=\frac12 \int_0^{2\pi} \log\left(\frac{b-\cos(x)}{a-\cos(x)}\right)\,dx \\\\ &\frac12 \int_0^{2\pi }\left(\log(b-\cos(x))-\log(a-\cos(x))\right)\,dx\tag 1 \end{align}$$
We will evaluate the integral in $(1)$ by moving to the complex plane. We let $z=e^{ix}$ in $(1)$ to arrive at
$$\begin{align} I(a,b)&=\frac12 \oint_{|z|=|}\frac {1}{iz} \left(\log(z-r_{b+})-\log(z-r_{a+})\right)\,dz \\\\ &+\frac12\oint_{|z|=1}\frac {1}{iz} \left(\log(z-r_{b-})-\log(z-r_{a-})\right)\,dz \tag 2 \end{align}$$
where $r_{a\pm}=a\pm\sqrt{a^2-1}$ and $r_{b\pm}=b\pm\sqrt{b^2-1}$.
Note that the branch points at $r_{a+}$ and $r_{b+}$ are not enclosed by the unit circle. Hence, the integrand of the first integral on the right-hand side of $(2)$ is meromorphic for $|z|\le 1$, with a simple pole at $z=0$.
Applying the residue theorem to the first integral on the right-hand side of $(2)$, we obtain
$$\begin{align} \frac12 \oint_{|z|=1}\frac {1}{iz} \left(\log(z-r_{b+})-\log(z-r_{a+})\right)\,dz&=\frac12 \left(2\pi i \text{Res}\left(\frac {1}{iz} \left(\log(z-r_{b+})-\log(z-r_{a+})\right),z=0\right)\right) \\\\ &=\frac12\left(2\pi i \frac1i \log\left(\frac{r_{b+}}{r_{a+}}\right) \right)\\\\ &=\pi \log\left(\frac{b+\sqrt{b^2-1}}{a+\sqrt{a^2-1}}\right) \tag 3 \end{align}$$
On the other hand, the branch points at $z=r_{a-}$ and $z=r_{b-}$ are enclosed by the unit circle. We can cut the plane so that a suitable branch cut is the segment that adjoins $r_{a-}$ and $r_{b-}$. Then, to evaluate the second integral on the right-hand side of $(2)$, we note that the integrand is analytic in the annular region $1\le |z|\le R$ for any $R>1$.
Therefore, Cauchy's Integral Theorem guarantees that
$$\begin{align} \oint_{|z|=1}\frac iz \left(\log(z-r_{b-})-\log(z-r_{a-})\right)\,dz&=\lim_{R\to \infty}\oint_{|z|=R}\frac iz \left(\log(z-r_{b-})-\log(z-r_{a-})\right)\,dz\\\\ &=\lim_{R\to \infty}\int_0^{2\pi}\frac{i}{Re^{i\phi}}\left(\log\left(1-\frac{r_{b-}}{Re^{i\phi}}\right)-\log\left(1-\frac{r_{a-}}{Re^{i\phi}}\right) \right)\,iRe^{i\phi}\,d\phi\\\\ &=0 \tag 4 \end{align}$$
Finally, putting together $(3)$ and $(4)$ yields the coveted result
$$\bbox[5px,border:2px solid #C0A000]{\int_0^\pi \log\left(\frac{b-\cos(x)}{a-\cos(x)}\right)\,dx=\pi \log\left(\frac{b+\sqrt{b^2-1}}{a+\sqrt{a^2-1}}\right)}$$
First, notice the integrand can be written as $$ \begin{align} \ln (\frac{b-\cos x}{a- \cos x}) &=\ln(b-\cos x)-\ln(a-\cos x)\\ & =\ln(y-\cos x)|^{y=b}_{y=a}\\ & =\int^b_a\frac{1}{y-\cos x}dy \end{align} $$ So the integration equals $$ \begin{align} \int^{\pi}_0 \ln (\frac{b-\cos x}{a- \cos x})dx &=\int^{\pi}_0 [ \int^b_a\frac{1}{y-\cos x}dy]dx \\ &=\int^b_a [\int^{\pi}_0 \frac{1}{y-\cos x}dx ] dy\\ & =\int^b_a(\frac{\pi}{\sqrt{y^2-1}})dy \end{align} $$ Let $$y=\sec t$$ so $$dy=\sec t\tan t dt$$ So the integrantion becomes $$ \begin{align} \pi \int^{acrsec b}_{arcsec a}\frac{\sec t\tan t}{\tan t}dt &=\pi \int^{acrsec b}_{arcsec a}\sec t dt\\ & = \pi \ln |\sec t+\tan t|^{arcsec b}_{arcsec a}\\ & = \pi [\ln(b+\sqrt{b^2-1})-\ln(a+\sqrt{a^2-1})] \\ & =\pi \ln (\frac{b+\sqrt{b^2-1}}{a+\sqrt{a^2-1}}) \end{align} $$