For a homework problem I am asked to find the orthonormal basis of a set of polynomials $ V = span \{ 1, x ,x^2 + 1 \}$ over the defined inner product " $ \langle f, g \rangle = \int_0^1 fgx^2 dx $ " (exact type set of the homework problem).
Does the notiation of the inner product specified mean composition of $f, g = f(g(x))$ or the multiplication of $f, g = f(x) * g(x)$?
On
It is the multiplication.
Only the multiplication here conforms to the (conjugate) symmetry property of an inner product, the composition does not. If $f=1$ and $g=x^2+1$, then you would have that $\int\limits_0^1 f(g(x))x^2 dx=\int\limits_0^1 x^2 dx\neq 2\int\limits_0^1 x^2 dx=\int\limits_0^1 g(f(x))x^2 dx$
Ask yourself, if it meant composition of functions would $\langle f, g \rangle $ even be an inner product? I think the only thing this question can mean is that $$\langle f, g \rangle = \int_o^1 f(x)g(x) x^2 dx.$$
The notation the question uses is a bit unhelpful though...