suppose that the conditional expectation of Y, given $X=x$, is linear in x, i.e. $E(Y|X=x)=a+bx$ for some constants a and b.
Compute $E(XY)$ using the law of total expectation and deduce that a and b satisfy a $\mu_X+bE(X^2)=E(XY)$
Question: From the problem, can write $E(XY)=E[E[XY|X=x]]$, I am not sure how to proceed.
From $E[E[XY|X= x]]$, realize that you are conditioning on $X = x$. Therefore, you may write : $$ E[E[XY | X= x]] = E[XE[Y | X = x]] $$
(If this is contentious, kindly see below) and now simply put $E[Y |X] = a+bX$ in to get $$E[XY] = E[aX + bX^2] = aE[X] + bE[X^2]$$
See, $E[Y|X]$ is a random variable, which is defined by $E[Y|X] (x) = E[Y|X = x]$.
We will show that $E[XY|X] = X E[Y|X]$. As these are functions, it is enough to show that they are the same at each point.
$$ E[XY|X] (x) = E[XY | X = x] = E[xY | X = x] $$
The second equality is true because whenever $X = x$, we have $XY = xY$, so they are the same random variable under the conditioning, hence have the same conditional expectation.
Now, taking the $x$ out gives $x E[Y | X = x] = (XE[Y|X]) (x)$, and we conclude.
If even that was not convincing, then on your request I will edit and provide a better proof from basics. But if you are convinced, it is great!