Given $f:\mathbb{R}\rightarrow\mathbb{R}$ periodic with $2\pi$ period from $[-\pi,\pi)$ and $f(x)=\sqrt{|x|^{3}}$ for every $n\in\mathbb{N}$ $$\displaystyle \lambda_n=\underset{a,b\, \in\, \mathbb{C}}{\min}\int_{-\pi} ^{\pi}|f(x)-ae^{i(n+1)x}-b\cos(nx)|^{2}dx$$ Calculate $\underset{n\rightarrow\infty}{\lim}\lambda_n.$
I've tried to set $g(x)=f(x)-ae^{i(n+1)x}-b\cos(nx)$ and i found the fourier coefficients:
$$
\hat{g}(k) = \left\{
\begin{array}{@{}l@{\thinspace}l}
\text{$\hat{f}(k)-a$} &: k=n+1\\
\text{$\hat{f}(k)-b/2$} &: k=n,-n\\
\text{$\hat{f}(k)$} &: \text{else}\\
\end{array}
\right.
$$
I concluded that $a$ and $b$ are $0$ but I wasn't sure how can I continue from here, I thought maybe to do parsebal for $f$.
I will believe your relation between $\hat{g}(k)$ and $\hat{f}(k)$ correct.
Notations: $\,\,<f,g> = \frac{1}{2\pi}\int_{-\pi}^{\pi} {f \bar{g}} dx\quad$ and $\quad\hat{f}(n) = <f,e^{inx}> = \frac{1}{2\pi}\int_{-\pi}^{\pi} {f e^{-inx}} dx $
Also, $\|f\|_{L^2} = \sqrt{\frac{1}{2\pi}\int_{-\pi}^{\pi} {|f|^2} dx}\quad$ and $\quad\|\vec{a}\|_{l^2} = $$\sqrt{\sum_{-\infty}^{\infty} |a(n)|^2} $.
Parseval: $ \|f\|_{L^2} = \|\hat{f}\|_{l^2} \\$
$$(2\pi)^{-1}\lambda_n=\underset{a,b\in\mathbb{C}}{\min}\|g\|_{L^2}^2 =\underset{a,b\in\mathbb{C}}{\min}\|\hat{g}\|_{l^2}^2 = \\ \underset{a,b\in\mathbb{C}}{\min}\|\hat{f}\|_2^2 - |\hat{f}(n+1)| -|\hat{f}(n)|^2 -|\hat{f}(-n)|^2 \\+ |\hat{f}(n+1) - a|^2 +|\hat{f}(n) - b/2|^2 +|\hat{f}(-n) - b/2|^2 $$
Since $f$ is even function, $\hat{f}(n) =\hat{f}(-n) $. Hence minimum attained at $a = \hat{f}(n+1)$, $b = \hat{f}(n)$.
Thus $$(2\pi)^{-1}\lambda_n = \|\hat{f}\|_2^2 - |\hat{f}(n+1)|^2 -|\hat{f}(n)|^2 -|\hat{f}(-n)|^2 .$$
By Riemann - Lebesgue lemma (for $L^1$ functions), $\underset{n\rightarrow\infty}{\lim}|\hat{f}(n)|= 0 $.
Finally we get $$\underset{n\rightarrow\infty}{\lim}\lambda_n= 2\pi\|\hat{f}\|_2^2 = 2\pi\|{f}\|_2^2 = \int_{-\pi}^{\pi} {|x|^{3}} = \frac{\pi^4}{2}$$