$p(x)\:=\:a+ax+ax^2$ and $q(x)\:=\:b+bx+bx^2$ are vectors in $P_2$
The inner product is the dot product: $$\langle p,q\rangle=a_0b_0+a_1b_1+a_2b_2$$
the set is:
$$\left\{\left(\frac{3x^2+4x}{5}\right),\:\left(\frac{-4x^2+3x}{5}\right),\:1\right\}$$
So I have to determine if this thing an orthonormal set.
after computing the magnitude to see if it equals 1, I ended up with the magnitude equaling $\frac{25}{25}x^4+\frac{25}{25}x^2+1$
I compared this question's answer to a similar one in the back of the textbook and it was in the same format as this one.
Does this answer makes the set orthonormal or not?
you have three vectors $$(0,4/5,3/5)^T, (0, 3/5, -4/5)^T, (1, 0, 0)^T $$ with the usual inner product. each of them have length $1$ and they are mutually orthogonal, therefore they form an orthonormal basis for $P_2.$