Let $$A=\left\{\int_0^1(t^2 - at-b)^2 dt\, : \,a,b \in \mathbb{R}\right\}\,.$$ Find the minimum of $A$.
$\textbf{My attempt:}$
Well, we have
$ 0 \leq\int_0^1(t^2 - at-b)^2 dt = \frac{1}{5} - \frac{a}{2} + \frac{a^2-b}{3} + ab+b^2$.
Ok, I can see that like a funtion $f: \mathbb{R}^2 \to \mathbb{R}$ given by $f(a,b) = \frac{1}{5} - \frac{a}{2} + \frac{a^2-b}{3} + ab+b^2$, then I need to find $(a,b)$ s.t $f(a,b)$ is minimun.
But I don't know how can I do that...Can you give me a hint?
Your calculation is incorrect. For $a,b\in\mathbb{R}$, if $f(a,b):=\displaystyle\int_0^1\,(t^2-at-b)^2\,\text{d}t$, then $$f(a,b)=\frac{a^2}{3}+ab+b^2-\frac{a}{2}-\frac{{\color{red}2}b}{3}+\frac{1}{5}\,.$$ Thus, $$f(a,b)=\frac{1}{3}\,\left(a+\frac{3(2b-1)}{4}\right)^2+\frac{1}{4}\left(b+\frac{1}{6}\right)^2+\frac{1}{180}$$ for all $a,b\in\mathbb{R}$. This shows that $f(a,b)\geq \dfrac1{180}$ for each pair $(a,b)\in\mathbb{R}^2$. The inequality becomes an equality if and only if $$a+\frac{3(2b-1)}{4}=0\text{ and }b+\frac{1}{6}=0\,,$$ which is equivalent to $$(a,b)=\left(1,-\frac16\right)\,.$$