Find an ortonormal base for the subspace $S=\left\{p(t)\in P_3(\mathbb{R}): tp'(t)=p(t)\right\}$
where $$\langle p,q\rangle=\int_{0}^{1}p(t)q(t)dt$$
And find also a base for the space $S^⊥$.
My attempt:
The only polynomials that satisfy $tp'(t)=p(t)$ are $p(t)=ct$ where $c\in\mathbb{R}$ is a constant.
Then $\{ct\}$ is an orthogonal base for $S$ and $\{\frac{\sqrt{3}}{|c|}ct\}$ is an orthonormal base for $S$.
Now, if $q(t)=a_0+a_1t+a_2t^2+a_3t^3$ then one condition I founded to satisfy $\langle ct, q(t)\rangle=0$ is that $c(\frac{a_0}{2}+\frac{a_1}{3}+\frac{a_2}{4}+\frac{a_3}{5})=0$, but I don't know how to write the orthogonal space knowing this, can you help me? Thanks!
You found that orthogonal basis of $S$ is $\{t \sqrt{3}\}$.
And $a_0+a_1t+a_2t^2+a_3t^3 = q(t) \in S^⊥$ if and only if $\frac{a_0}{2}+\frac{a_1}{3}+\frac{a_2}{4}+\frac{a_3}{5}=0$. We can calculate $a_0 = -\frac{2a_1}{3}-\frac{a_2}{2}-\frac{2 a_3}{5}$. Let us find a basis of the null space.
This produces a basis of the space $S^⊥$ in the form $\{t - \frac{2}{3}, t^2 - \frac{1}{2}, t^3 - \frac{2}{5}\}$.
Now we can build an orthogonal basis of the space $S^⊥$ using the Gram–Schmidt process.
From this build orthogonal basis we can easily build an orthonormal basis.
P.S. If we run Gram–Schmidt process (and orthonormalization), we obtain the answer: $\left\{3 t-2,6 \sqrt{5} t^2-6 \sqrt{5} t+\sqrt{5}, 20 \sqrt{7} t^3-30 \sqrt{7} t^2+12 \sqrt{7} t-\sqrt{7}\right\}$.