I want find real $a$, $b$ such that $$F(a,b)=\int_0^{2\pi} (\sin x-ax-b)^2 dx$$ has minimun. Let $f(x,a,b)=(\sin x-ax-b)^2$.
I know I can directly find compute the integral in terms of $a$, $b$, then set the partial derivative to zero. Or use Leibniz rule to simplify the computation
$$\frac{\partial}{\partial a}F(a,b) =\int_0^{2\pi}\frac{\partial}{\partial a} f(x,a,b) \space dx,$$
$$\frac{\partial}{\partial b}F(a,b) =\int_0^{2\pi}\frac{\partial}{\partial b} f(x,a,b) \space dx.$$
I just have some doubts on the condition to exchange limit in this case:
My understanding for the condition in this case (not improper integral) is that $f(x,a,b)$ and $\frac{\partial}{\partial a} f(x,a,b)$ has to be continuous on $[0,2 \pi]\times[ a_1,a_2]\times[ b_1,b_2]$ for some fixed $a_1,a_2,b_1,b_2$, namely in some closed interval.
But in the question, my $a$, $b$ is just any real number. Why is it that I can exchange the differentiation and integral sign? Can I somehow bound $a$, $b$? or is there a more general condition to apply Leibniz rule? Thank you!
Proceeding as you want, consider $f(a, b)=\int^{2\pi}_0(\sin x -a x -b)^2\,dx$
There are several arguments that will justify the exchange of order between differentiation and integration. One argument that works is dominated convergence, although this requires a little of modern integration theory. Other arguments are based on the continuity and differentiability of the integrand $F(x,a,b)=(\sin x -a x -b)^2$. See for example Apostol, T. Mathematical Analysis, 2nd Edition, section 7.23.
$$\begin{align}\partial_af &=\partial_a\int^{2\pi}_0F(x,a,b)\,dx=\int^{2\pi}_0\partial_aF(x,a,b)\,dx\\ \partial_bf &=\partial_b\int^{2\pi}_0F(x,a,b)\,dx=\int^{2\pi}_0\partial_bF(x,a,b)\,dx \end{align} $$ and
Then you have to find critical points $\partial_af(alb)=0=\partial_bf(a, b)$ and determine (differentiate again and use the Hessian criteria) to determine whether a (local) minima is obtained and so on. The equations in this case are $$\begin{align} -2\int^{2\pi}_0(\sin x -ax-b)x\,dx&=0\\ -2\int^{2\pi}_0(\sin x -ax -b)\,dx&=0 \end{align} $$
Another more geometric approach is to notice the the minimization problem you are trying to solve is equivalent to finding the best approximation in $L_2(0,2\pi)$ (space of square integrable functions) of the function $f(x)=\sin x$ to the linear subspace generated by the functions $f_0(x)=1$ and $f_1(x)=x$. This problem is related to the problem in linear algebra of finding the orthogonal projection of a vector $\mathbf{u}$ to a subspace generated by two vectors $\{\mathbf{v},\mathbf{w}\}$ in a inner product space.
In the case of the OPs problem, consider the inner product $$\langle \phi,\psi\rangle =\frac1{2\pi}\int^{2\pi}_0\phi(x)\psi(x)\,dx$$ An orthonormal basis for the two-dimensional linear space generated by $\{f_0,f_1\}$ is $$u_0(x)=f_0(x)=1$$ and $$ u_1(x)=\frac{1}{\|f_1-\langle f_1,u_0\rangle u_0\|_2}\big(f_1(x)-\langle f_1,u_0\rangle u_0(x)\big) $$ A simple calculation yields $$ f_1(x)-\langle f_1,u_0\rangle u_0(x)=x-\pi$$ Then $$\|f_1-\langle f_1,u_0\rangle u_0\|_2=\sqrt{\frac{1}{2\pi}\int^{2\pi}_0(x-\pi)^2\,dx}=\sqrt{\frac{1}{2\pi}\Big(\frac{2\pi^3}{3}\Big)}=\frac{\pi}{\sqrt{3}} $$ Hence $u_1(x)=\frac{\sqrt{3}}{\pi}(x-\pi)$
The best approximation of $f(x)=\sin x$ to the linear space generated by $\{u_0,u_1\}$ is then $$ \langle f,u_0\rangle u_0 + \langle f,u_1\rangle u_1$$ $$\langle f, u_0\rangle =\frac{1}{2\pi}\int^{2\pi}_0\sin x\,dx=0$$ $$\langle f,u_1\rangle =\frac{\sqrt{3}}{2\pi^2}\int^{2\pi}_0(x-\pi)\sin x\,dx=\ldots$$