Consider the inner product space $P([0,1])$ of all real polynomials on $[0,1]$ with inner product $\langle f,g\rangle=\int_0^1f(x)g(x)dx$ and $V=span\{t^2\}$. Let $h(t) \in V$ be such that $\|(2t-1)-h(t)\| \le \|(2t-1)-x(t)\|$ for $x(t) \in V$, then $h(t)$ is
$$a ) \frac{5t^2}{6}$$ $$b ) \frac{5t^2}{3}$$
This is my thought. Since $2t-1-h(t)$ is orthogonal to $V$ therefore $a$ must be the correct answer but in the book $b$ is given as correct answer.
Assuming that I did the integral correctly, I agree that the dot product of $q(t) = 2t -1 - h(t) $ (for the function $h$ given in answer "a") and $r(t) = t^2$ is indeed $0$, so the answer should be "a" rather than "b".