For the inner product space $V$ and linear transformation $g: V \to F$, find vector $y$ such that $g(x) = \langle x, y \rangle$ for $x \in V$.
Specifically $V = P_{2}(\mathbb{R})$ with $\langle f,h \rangle = \int_{0}^{1}f(t)h(t)dt$ and $g(f) = f(0) + f'(1)$.
So in this scenario my $y$ vector is actually going to be $h$.
To solve the question one needs to use an orthonormal basis which I obtained as $\beta = \{\frac{1}{\sqrt{2}}, \sqrt{\frac{3}{2}}x, \sqrt{\frac{5}{8}}(3x^{2}-1) \}$.
Using this I proceeded in the following manner:
$$h = \langle h, \frac{1}{\sqrt{2}} \rangle\frac{1}{\sqrt{2}} + \langle h, \sqrt{\frac{3}{2}}x \rangle \sqrt{\frac{3}{2}}x + \langle h, \sqrt{\frac{5}{8}}(3x^{2}-1) \rangle \sqrt{\frac{5}{8}}(3x^{2}-1)$$
Which can be done due to the orthonormal basis.
Since I'm dealing with polynomials over $\mathbb{R}$ the conjugate will be the same so each inner product that was in the form $\langle h, v_{i} \rangle$ for basis vector $v_{i}$ can be rewritten as $\langle v_{i}, h \rangle$.
Switching components around due to the conjugate then allows me to express $h$ in the form:
$$h = g(\frac{1}{\sqrt{2}})\frac{1}{\sqrt{2}} + g(\sqrt{\frac{3}{2}}x)\sqrt{\frac{3}{2}}x + g(\sqrt{\frac{5}{8}}(3x^{2}-1))\sqrt{\frac{5}{8}}(3x^{2}-1)$$
which after the necessary computations results in:
$$h = \frac{-21}{8} + \frac{3}{2}x + \frac{75}{8}x^{2}$$
I wanted to check that I obtained the correct polynomial from my process so I proceeded to take a general polynomial $f(x) = ax^{2} + bx + c$. Applying $g(f)$ to it using the specified formula I get:
$$g(f) = 2a + b + c$$
So I expect to get this same expression when I use the inner product form of the expression. i.e I expect:
$$\langle f, h \rangle = 2a + b + c$$
Thus $$\langle f, h \rangle = \int_{0}^{1}(at^{2}+bt+c)(\frac{-21}{8} + \frac{3}{2}t + \frac{75}{8}t^{2})dt$$
After a bunch of algebra I ended up with:
$$\frac{11a}{8} + \frac{49b}{32} + \frac{11c}{16}$$
Which means trouble to me. My question is did I proceed correctly? Should I be expecting to end up with the result I anticipated or because of my unorthodox orthonormal basis the result won't be exact?
I should also mention there was a question asked about same problem in the past: Find a vector $y$ such that $g(x)=\langle x,y \rangle$ for all $x \in V$ , but I was curious to know if my approach to the solution should be viable as well?
The problem is that your basis $\beta$ is not orthonormal. It's not even orthogonal. It would be orthonormal if the inner product were $\int_{-1}^1 f(x)g(x)\, \mathrm{d}x$. That will prevent your computations from being correct from the start.
After that, things look fine. The fact that $h = g(e_1)e_1 +g(e_2) e_2 + g(e_3)e_3$, where $e_1, e_2, e_3$ is orthonormal tracks (and indeed it's well-noticed that there would have to be conjugates over the coefficients in the case where $V$ were complex).
So, yeah, just try again with an orthonormal basis.