A question defines $$f(x)= (ax+b)e^x$$ and states that it satisfies the equation $$f(x)=\int _0 ^x e^{x-y}f'(y)dy - (x^2-x+1)e^x$$ You're required to find a and b.
Here, they've solved it by taking the $e^x$ in the second equation to the left side and differentiating using the Leibniz rule. You end up with $(a-2)x+(b-1)=0$, which gives you a and b as 2 and 1, since this equation has to hold for all x, I assume. This makes sense.
But I did it by differentiating f to get $f'(x)$, plugged that into the integral and integrated it, substituted f(x) for the expression in the first equation, and simplified it to get $$x^2(\frac{a}{2}-1)+x(b+1)-(1+b)=0$$ Using the same thing about this having to satisfy all x, I tried the values 0 and 1 for x in the equation to get $$b=-1, a=0$$
What did I get wrong here?
Going by your method, we get, \begin{align} &f'(x) = e^x(ax+b+a)\\ \implies &f(x)=\int_0^xe^{x-y}e^y(ay+b+a)dy-(x^2-x+1)e^x\\ \implies &f(x) = e^x\int_0^x(ay+b+a)dy-(x^2-x+1)e^x\\ \implies &f(x) = e^x\left(\dfrac{ax^2}{2}+(b+a)x\right)-(x^2-x+1)e^x\\ \implies &(ax+b)e^x = e^x\left(\left(\dfrac{a}{2}-1\right)x^2+(b+a+1)x-1\right) \end{align} Cancelling $e^x$ and equating coefficient both sides, you get $a=2$ and $b=-1$.
Going by their method, we get, \begin{align} &e^{-x}f(x) = \int_0^xe^{-y}f'(y)dy-(x^2-x+1)\\ \implies &\dfrac{d}{dx}(ax+b) = e^{-x}f'(x)-(2x-1)\\ \implies &a = (ax+b+a)-(2x-1)\\ \implies &0=(a-2)x+(b+1)=0 \end{align} which leads to the same solution. Note that the given answer that $b=1$ is wrong.