Given $f$ and $g$ polynomial functions $\mathbb{R}\rightarrow \mathbb{R}$ with the height degree $1$, and $T(bx+a)=ax+b$ and the inner product integral$\int_0^1fg\enspace dx$.
find $T^*$.
I thought that I should just take the transpose matrix of $T$ but it seems an incorrect answer. Why? I thought that on $\mathbb{R}$, $T^*$ must be the transpose
Here's how we would solve the problem using the definition. For a linear map $T$ on the inner product space $\mathcal P(\mathbb R)$, $T^*$ is defined so that for all $f,g \in \mathcal P(\mathbb R)$ we have $$ \langle Tf,g\rangle = \langle f, T^*g \rangle. $$ Now, suppose $f(x) = a_1 x + b_1$ and $g(x) = a_2 x + b_2$. We then have $$ \langle Tf, g \rangle = \int_0^1 (b_1 x + a_1)(a_2 x + b_2) = \int_0^1 [a_2 b_1 x^2 + (a_1 a_2 + b_1 b_2)x + a_1 b_2]\,dx \\ = \frac 13 a_{2}b_1 + \frac 12(a_1 a_2 + b_1 b_2) + a_1 b_2. $$ Now, in terms of $a_2$ and $b_2$, we want to find $\alpha,\beta$ such that we'll have $T^*g = \alpha x + \beta$, which is to say that for all pairs $a_1,b_1$ we have $$ \langle f, T^*g \rangle = \frac 13 a_{2}b_1 + \frac 12(a_1 a_2 + b_1 b_2) + a_1 b_2 \\ = \left(\frac 12 a_2 + b_2 \right)a_1 + \left(\frac 13 a_2 + \frac 12 b_2\right)b_1 \\ = \int_{0}^1 (a_1x + b_1)(\alpha x + \beta)\,dx\\ = \int_0^1 [\alpha a_1 x^2 + (b_1 \alpha + a_1 \beta)x + b_1 \beta]dx\\ = \frac 13 \alpha a_1 + \frac 12 (b_1 \alpha + a_1 \beta)+ b_1 \beta \\ = \left(\frac 13 \alpha + \frac 12 \beta \right)a_1 + \left(\frac 12 \alpha + \beta\right) b_1 $$ In other words, we are solving system of equations: $$ \frac 13 \alpha + \frac 12 \beta = \frac 12 a_2 + b_2, \quad \frac 12 \alpha + \beta = \frac 13 a_2 + \frac 12 b_2. $$ Note: We can get the above equations in a neater way if we separately note that $\langle Tf,g\rangle = \langle f, T^*g \rangle$ for $f(x) = 1$ and $f(x) = x$.
Assuming my calculations are correct, this yields $$ \alpha = 4a_2 + 9b_2, \quad \beta = - \frac 53 a_2 - 4 b_2. $$ Correspondingly, the adjoint to $T$ is given by $$ T^*(ax + b) = (4a + 9b)x - 5a/3 - 4b. $$
Another approach, using Gram Schmidt. We build an orthonormal basis from $\{1,x\}$. Since $1$ already has norm 1, we compute $$ \langle 1,x \rangle = \int_0^1 x\,dx = \frac 12 $$ which means that $$ x^{\perp} = x - \langle 1,x \rangle \cdot 1 = x - \frac 12. $$ To normalize this vector, we compute $$ \langle x^\perp,x^{\perp} \rangle = \int_0^1 (x - \frac 12)^2\,dx = \frac 1{12} $$ which means that our orthonormal basis is $$ \{f_1,f_2\} = \{1, \frac{x^\perp}{\|x^\perp\|}\} = \{1, \frac 1{\sqrt{12}} (x - \frac 12)\}. $$ We find that $$ T(f_1) = x = \frac 12 f_1 + \sqrt{12} f_2 \\ T(f_2) = \frac 1{\sqrt{12}} (1 - \frac 12 x) = \frac 1{\sqrt{12}} - \frac 12 f_2 - \frac{1}{4\sqrt{12}} = \frac{3}{4\sqrt{12}} f_1 - \frac 12 f_2. $$ So, the matrix of $T$ relative to this orthonormal basis is $$ [T]_{\mathcal B} = \pmatrix{1/2 & \frac{3}{4\sqrt{12}} \\ \sqrt{12} & - \frac 12}. $$ The matrix of the adjoint is simply the transpose of the above.
A more elegant approach, using a judicious choice of orthonormal basis. We know that $x + 1$ is an eigenvector of $T$. Apply the Gram Schmidt process to the basis $\{v_1,v_2\} = \{x+1,1\}$. First, normalizing $x + 1$ yields $$ \langle v_1,v_1 \rangle = \int_0^1 (x+1)^2\,dx = \frac 73 $$ so our first vector is $u_1 = \sqrt{\frac 37} (x+1)$. We compute $$ v_2^\perp = v_2 - \langle v_2,u_1 \rangle u_1 = v_2 - \frac 37 \left(\int_0^1 x(x+1)\,dx\right) (x+1)\\ = x - \frac 37 \cdot \frac 56 (x+1) = x - \frac 5{14} (x+1) = \frac{1}{14}(9x - 5) $$ Normalizing this yields $u_2 = \frac{9x - 5}{\sqrt{7}}$. We then compute $$ T(u_1) = 1 \cdot u_1 + 0 \cdot u_2\\ T(u_2) = \frac{-5x + 9}{\sqrt7} = \cdots = \alpha u_1 + \beta u_2 $$ From there, we know that the matrix of $T$ relative to this basis is $$ \pmatrix{1&\alpha\\0 & \beta} $$ and that the matrix of its adjoint is the (conjugate) transpose, $$ \pmatrix{1&0\\\alpha & \beta}. $$