We can approximate a function $g$ in the interval $[0,1]$ by a polynomial of degree $\le n$ by minimizing the criterion function:
$$f(a) = \int_0^1 [g(x) - p(x)]^2\ dx$$
where $p(x) = a_0 + a_1x + \cdots + a_n x^n$. Find the equations that the optimal coefficients have to satisfy.
I normally would try to take the partials and equate to $0$:
$$\frac{\partial f}{\partial a_j} = \int_0^1\frac{\partial f}{\partial a_j}(g(x)-p(x))^2\ dx = \int_0^12(g(x)-p(x))x^j\ dx$$
So I know that at least it should satisfy $\int_0^1(g(x)-p(x))x^j\ dx = 0$. I think that by optimal it means that it's either a maximum or a minimum, not a saddle, right? So the hessian should satisfy something to: either positive definite or negative definite. Am I right? Then
$$\frac{\partial^2 f}{\partial a_i\partial a_j} = 2\int_0^1-x^jx^i\ dx = -2\int_0^1 x^{i+j}\ dx = \frac{-2x^{i+j+1}}{i+j+1}|_{x=1} = \frac{-2}{i+j+1}$$
Let $M$ be the matrix with these elements, then I'd have $x^tMx>0$ or $x^tMx<0$ depending on the case (minimum or maximum).
I know these $2$ conditions, when met, should let to $x$ be a minimum or maximum. Is this it? The exercise says 'find the equations that the optimal coefficients have to satisfy'. What are optimal coefficients exactly?
Since the goal is to approximate $g$ by the polynomial $p$, the optimal coefficients would be those that minimize the given integral (so that $g\approx p$). The determinant of the Hessian matrix must then be $>0$.
When the first derivatives are set to zero, the linearity of integration allows to write it as a system of $n$ equations with the coefficient of $a_k$ being $-\int_0^1 x^{k+j}\ dx$. This can be solved with Cramer's rule if desired.