Let $f:\mathbb R \to \mathbb R$ be the periodic continuation of the function $\sqrt{\left|x \right|^3}$ on the interval $[-\pi,\pi)$.
For every $n \in \mathbb N$ let us denote: $$\lambda_n = \min_{a,b \in \mathbb C}\int_{-\pi}^\pi\left|f(x)-ae^{i(n+1)x}-b\cos(nx) \right|^2dx$$
what is $\lim_{n\to \infty}\lambda_n$?
Well from the fact that we are on a $2\pi$ interval and the fact that we have $\cos(nx),e^{inx}$ this question obviously referring to Fourier series. I tried to define $g(x) = f(x)-ae^{i(n+1)x}-b\cos(nx)$ and extracting its coefficients and using them but I didn't know how to continue.
The mix of exponential and trigonometric notation is (perhaps intentionally) confusing. I would convert both to complex exponential. Let's say $$ f(x) = \sum_{k\in\mathbb Z} c_k e^{ikx} $$ and we want the best approximation to $f$ of the form $\sum_{k\in A}b_k e^{ikx}$ where $A$ is some finite set like $\{-n,n,n+1\}$. Since the quality of approximation is measured in $L^2$ norm, the answer falls out immediately from Parseval's identity: we are minimizing $2\pi\sum |c_k-b_k|^2$ so the best we can do is to let $b_k=c_k$ for $k\in A$ (and there's nothing we can do about $k\notin A$).
The gain is very modest: since $n\to\infty$, we have $\sum_{k\in A} |c_k|^2\to 0$, which means the $L^2$ norm of the best approximation tends to zero, and the rest of $L^2$ norm of $f$ is the residual. Thus, $$ \lambda_n \to \int_{-\pi}^\pi |f(x)|^2\,dx = \int_{-\pi}^\pi |x|^3\,dx $$