Using differentiation with respect to the parameter,show that for $|r|<1$
$$\mathbf{F}(r)=\int_{0}^{\pi}\ln\left(1-2r\cos x +r^2 \right)dx =0$$
my attempt is $$\mathbf{F}'(r)=\int_{0}^{\pi}{-2\cos x + 2r \over1-2r\cos x +r^2}dx$$
let $u=\tan{x \over2}$ then $\cos x = {1-u^2 \over 1+u^2} $ and $dx={2du \over 1+u^2}$
Now $$\mathbf{F}'(r)=\int_{0}^{\infty}{-2{1-u^2 \over 1+u^2} + 2r \over1-2r{1-u^2 \over 1+u^2} +r^2}{2du \over 1+u^2}$$ $$\mathbf{F}'(r)= 4\int_{0}^{\infty}{r+ru^2+u^2-1 \over 1+u^2-2r+2ru^2+r^2}{du \over 1+u^2}$$ but I do not know how to continue, I tried with partial fractions but it becomes too tedious.
A slightly different approach.
It is clear that $F'(0)=0$. We assume $r\neq 0$ from now on. Notice that for $|r|<1$ one has: \begin{align} \sum_{n=0}^\infty r^n \cos(nx)& = \frac{1-r\cos(x)}{1-2r\cos(x)+r^2}\\\sum_{n=0}^\infty r^n \sin(nx)&= \frac{r\sin(x)}{1-2r\cos(x)+r^2} \end{align} One can use that to write $F'(r)$ as: \begin{align} F'(r) = \int^\pi_0\left[ \frac{2}{r}\sum_{n=0}^\infty r^n \cos(nx) + 2\left( 1-\frac 1 {r^2}\right)\sum_{n=0}^\infty r^n \frac{\sin(nx)}{\sin(x)} \right]\,dx \end{align} Both series converge uniformly on $x\in [0,\pi]$ so one can interchange the order of integration and summation to obtain: \begin{align} F'(r) = \frac{2}{r}\sum_{n=0}^\infty r^n \int^\pi_0\cos(nx)\,dx + 2\left( 1-\frac 1 {r^2}\right)\sum_{n=0}^\infty r^n \int^\pi_0\frac{\sin(nx)}{\sin(x)} \,dx \end{align} Notice that \begin{align}\int^\pi_0 \cos(nx) = \begin{cases} \pi & \text{ if } n =0\\ 0 & \text{ if } n\neq 0 \end{cases} \end{align} and: \begin{align}\int^\pi_0 \frac{\sin(nx)}{\sin(x)} = \begin{cases} \pi & \text{ if } n \text{ is odd} \\ 0 & \text{ if } n \text{ is even} \end{cases} \end{align} So: \begin{align*} F'(r) = \frac {2 \pi} r + 2 \left( 1- \frac 1{r^2}\right) \sum_{n=0}^\infty \pi r^{2n+1} =\frac {2 \pi} r + 2\pi\left( 1 -\frac 1{ r^2}\right) \frac{r}{1-r^2} = 0 \end{align*} This implies that $F(r)=C$ for some $C\in\mathbb R$. Set $r=0$ to find that $C=0$.