f(x) = ax + b
g(x) = c$x^2$ + dx
So here's what I have:
$f ◦ g = a(cx^2 + dx) + b = acx^2 + adx + b$
$g ◦ f = c(ax + b)^2 + d(ax + b) = ca^2x^2 + 2acxb + b^2c + dax + db$
$g ◦ f = f ◦ g$
$ca^2x^2 + 2acxb + b^2c + dax + db = acx^2 + adx + b$
$ca^2x^2 + 2acxb + b^2c + db = acx^2 + b$
I'm stuck here and not sure what to do next. Did I go down the wrong path? Is there something I could simplify further?
On
First I would suggest that you move the $acx^2 + b$ to the LHS. As you want $f ◦ g = g ◦ f$, it must be true that $f(g(x)) - g(f(x)) = 0$ $\forall x\in \mathbb{R}$ and $f(g(-x)) - g(f(-x)) = 0$.
This should help you narrow down the "search space". And then you can substitute in any value for $x$ and get a equation. Using all these equations solve for $a, b, c $ and $d$.
I am also confident that it would be tedious to solve them.
On
You are almost there.
Note: This may be incorrect because your problem may be stated incorrectly. See my comments.
From $ca^2x^2 + 2axb + b^2 + db = acx^2 + b $, we get $(ca^2-ac)x^2+2abx+b(b+d-1) = 0 $.
If this is true for all $x$, then all the coefficients must be zero, so $0 =ca(c-1) =2ab =b(b+d-1) $.
If $b=0$, then $a=0$ or $c=0$ or $c-1$.
If $b \ne 0$ then $b+d-1 = 0$ and $a=0$ ($c$ can be anything).
The algebra in your substitution was not correct. $f\circ g$ is right, but $g\circ f(x)=a^2cx^2+(2abc+ad)x+b^2c+bd$. When you set the two equal to each other and compare the coefficients of $x^2,x,1$, you get the equations $a^2c=ac$, $2abc=0$, $b^2c+bd=b$.
This gives you three cases, depending on whether $a=0$ or $b=0$ or $c=0$, and you should be able to find the proper restrictions on $a,b,c,d$ in these cases.
When $a=1,b=0$, $f$ is the identity function, so equality holds whatever $c,d$ are. This shows that the condition $b+d=1$ doesn't always hold. Also note there are combined cases like $a=b=0$, where $c,d$ can be anything, or $b=c=0$, where $a,d$ can be anything.