Recently, while studying calculus, I have come across multiples problems which asked the following: If $f(x)$ is a polynomial, find all $f(x)$ that $f(f(x))=f'(x)f(x)+c$, where $c$ is a constant.
This problem can be solved as so:
If $deg(f(x))=n$, the upper equation implies that $n^2=2n-1$, or that $n=1$.
This implies that $f(x)=ax+b$, and it is just a matter of calculation from here.
But how does one find all $f:\mathbb {R} \rightarrow \mathbb {R}$ where $f(f(x))=f'(x)f(x)+c$? I am asking for solutions whe $f(x)$ is not necessarily a polynomial.
Because of my above method, I thought that $f'(x)$ would be a constant.
However, I was not able to prove this. Differentiating both sides gave me that $(f'(f(x))-f'(x))(f'(x))=f''(x)f(x)$. This proved no help at all.
Since I am young, please use methods that are comprehensible to a high-school student.
Any help would be appreciated.
There is a solution with $f(x)=ax+bx^2+cx^3+dx^4+...$. Here, I am taking your $c=0$, but introducing my $c$ as one of the coefficients.
$$f(f(x))=af(x)+bf(x)^2+cf(x)^3+...\\ =a^2x+abx^2+acx^3+...+ba^2x^2+2ab^2x^3+...+ca^3x^3+...\\ f'(x)f(x)=(a+2bx+3cx^2+...)(ax+bx^2+cx^3+...)\\ =a^2x+3abx^2+(4ac+2b^2)x^3+... $$ Equate coefficients $$ a^2=a^2\\ab+ba^2=3ab\to a=2\\ac+2ab^2+ca^3=4ac+2b^2\to c=-b^2$$ so it seems the first few terms must be $f(x)=2x+bx^2-b^2x^3+...$
EDIT: I made Maple find this solution, when your $c=0$: $$\color{red}{f(x)=\frac2b(1+2bx-\sqrt{1+2bx})}$$ When $x=2y$, the coefficients of the series above were $1,1,-2,5,-14,42$. This sequence is fairly famous, or I could have looked it up in the OEIS (Online Encyclopedia of Integer Sequences). I then used the formula, which involves $\frac1{n+1}{2n\choose n}$, in the sequence, and Maple worked out the infinite sum.
To check that, let $y=\sqrt{1+2bx}$, and suppose $y>1/2$. Then $$f(x)=\frac2b(y^2-y)\\ 1+2bf(x)=4y^2-4y+1=(2y-1)^2\\f(f(x))=\frac2b(4y^2-4y+1-(2y-1))$$
On the other hand, $f'(x)=4-\frac{2}{\sqrt{1+2bx}}=4-\frac2y$, so $$f(x)f'(x)=\frac2b(y^2-y)(4-\frac2y)=\frac2b(y-1)(4y-2)=\frac2b(4y^2-6y+2)=f(f(x))$$