Consider the inner product space $C[0,1]$ with inner product $$\langle f,g\rangle =\int_0^1f(x)g(x)\,dx$$ Let $S$ be the subspace spanned by $1$ and $2x-1$
Find the best least squares approximation of $\sqrt x$ by a function from the subspace $S$.
What exactly is the relationship between least squares approximation and orthonormal vectors? In other words, I would appreciate if this question were explained to me in the most basic way.
I know that we use $$p=\sum_ic_iu_i \quad c_i=\langle x,u_i\rangle$$ where $u_i$ are the vectors in the orthonormal basis.
But $p = A\hat{x} $ and I thought we were supposed to be looking for $\hat{x} $
The least square approximation of a function $f$ by a function from a subspace $S$ is the projection $f$ of onto $S$. You have denoted it by it $p$ in the question.
An orthonormal basis just simplifies our working using the formula that you have pointed out.
$$p=\sum_ic_iu_i \quad c_i=\langle f,u_i\rangle \tag{1}$$
where $u_i$ forms an orthornomal basis.
Guide:
We should first check if $\{ 1, 2x-1\}$ is an orthonormal basis and if it is not, we can construct an orthonomal basis. Verify that
$$\int_0^1 1^2 \, dx = 1$$
$$\int_0^1 1 \cdot (2x-1) \, dx = 0$$ $$\int_0^1 (2x-1)^2 \, dx= \frac13$$
Hence it is not orthonormal. It is just orthogonal.
Note that we have $$\int_0^1 ((\sqrt3)\cdot (2x-1) )^2 \, dx= 1$$
Try to construct an orthornormal basis and then use formula $(1)$ to compute its projection.