use the least squares to find the best linear approximation to $f(x)=x^2$ on [-1,1]. that is find the line $y=a_0+a_1x$ that minimizes $\int_{-1}^1|f(x)-y(x)|^2$
solution I used the theory of approximation in hilbert spaces: so the inner product is $\int_{-1}^1f(x)g(x)$. f(x)=1 and f(x)=x ae orthogonal polynomials. hence I can find $a_0 = <1,f>=\int_{-1}^1x^2dx=2/3$
$a_1 = <x,f>=\int_{-1}^1x^3dx=0$
so the best linear approximation is $y=2/3$
but what stopped is that the theorem requires f to belong in a hilbert space. Is the space of continuous functions on the mentioned inner product a hilbert space?
You are correct: The space of continuous functions on $[-1,1]$ is not a Hilbert space under the inner product $(f,g)=\int_{-1}^{1}f(x)g(x)dx$.
However, for any inner product space $X$, and fixed $f \in X$, there is always a minimum of $\|f-m\|$, where $m$ varies over all elements of a finite-dimensional subspace $\mathcal{M}$. That minimum is the unique $m\in \mathcal{M}$ such that $(f-m,m')=0$ for all $m' \in \mathcal{M}$. Indeed, if there is some $m\in\mathcal{M}$ for which $(f-m,m')=0$ for all $m'\in \mathcal{M}$, then $(f-m,m-m')=0$ as well, which gives $$ \|f-m'\|^{2}=\|f-m+m-m'\|^{2}=\|f-m\|^{2}+\|m-m'\|^{2}\ge \|f-m\|^{2}, $$ and equality holds iff $m-m'=0$. So $m$ is the unique minimum of $\|f-m'\|$ for $m'\in\mathcal{M}$.
To see that such $m$ exists, let $\{ m_1,m_2,\cdots,m_n\}$ be a basis of $\mathcal{M}$. Apply Gram-Schmidt orthogonalization process to obtain an equivalent orthonormal basis $\{ e_1,e_2,\cdots,e_n\}$. Then $$ m = \sum_{j=1}^{n}(f,e_n)e_n. $$ It's easy to check that $(f-m,e_j)=0$ for all $j$ and, hence, $(f-m,m')=0$ for all $m'\in\mathcal{M}$.
Your Example: The function $y=a_0 x+ b_0$ that you want is the unique solution of $$ (f-y,1)=0,\;\;\; (f-y,x)=0, \\ \int_{-1}^{1}(x^{2}-a_0 x-b_0)dx = 0,\;\;\; \int_{-1}^{1}(x^{2}-a_0x -b_0)x dx =0, \\ \frac{2}{3}-2b_0 = 0,\;\;\; -\frac{2}{3}a_0 = 0. $$ Answer: $y= \frac{2}{3}$.