I am trying to find all solutions for the following functional equation: $$f:\mathbb{N}\to\mathbb{R}, \ \ s.t. \ \ (f(x)+1)f(f(x))=x(x+1)$$ It is clear that $f(x)=x$ is a solution for this equation. However, I'd like to prove that this is the only solution (or find a general formula for all the solutions).
My attempt was to first find $f(0)$: For $x=0$, $(1+f(0))f(f(0)) = 0$ so either $f(0)=-1$ or $f(f(0))=0$. For the second case, let $x=f(0)$, so $(1+f(f(0)))f(f(f(0))) = f(0)(f(0)+1)$, therefore $f(0)=f(0)(f(0)+1)$, where we find that $f(0)=0$.
As such, I have proven that either $f(0)=-1$ or $f(0)=0$.
Being inspired by the technique for finding $f(0)$, I wrote the functional equations for $x=n$ and $x=f(n)$.
$$(1+f(n))f(f(n)) = n(n+1)$$ $$(1+f(f(n)))f(f(f(n))) = f(n)(f(n)+1)$$ and by dividing we obtain: $$f(n) \cdot f(f(n)) = n(n+1) \cdot (1+f(f(n)))f(f(f(n)))$$
Or, by plugging in $x=n$ and $x=n-1$, we get the two equations $$(1+f(n))f(f(n)) = n(n+1)$$ $$(1+f(n-1))f(f(n-1))=n(n-1)$$ an idea is to find a formula for $f(n)$ in terms of $f(n-1)$, but I am stuck how I could find such an equation without including $f(f(n))$.
Is there any way we can find all solutions, or find a relationship involving only $f(n)$ and $f(n-1)$?
Firstly, I think the question you were given would have said "$f:\mathbb{N}\to\mathbb{N}$". If $f(x)$ is some real number that is not in $\mathbb{N}$, then $f(f(x))$ is undefined.
Now that we've corrected that, as you would know, $f(0)=0$, because $f(0)\neq-1$.
How about $f(1)$? You have $(f(1)+1)f(f(1))=2$. The only factorisations are $1\times2$ and $2\times 1$.
Clearly we can't have $f(1)+1=1$, otherwise $f(f(1))=f(0)=0$.
So $f(1)+1=2$, or equivalently $f(1)=1$.
By now you might guess that an argument by induction will work.
Suppose we have proven, for $0\leq k\leq n$, that $f(k)=k$.
Let's now plug in $x=n+1$, to get
$(f(n+1)+1)\times f(f(n+1))=(n+1)\times (n+2)$.
You would guess that $f(n+1)=n+1$. Indeed we can prove this, by finding out about the size of the two factors on the left hand side.
For instance, $f(n+1)+1<n+2$ would imply that $f(n+1)\leq n$, which would in turn imply that $f(f(n+1))\leq n$. But there would be a contradiction as $n^2<(n+1)(n+2)$.
So we can actually conclude that $f(n+1)\geq n+1$.
Similarly, $f(f(n+1))<n+1$ would imply that $f(f(n+1))\leq n$. Plugging in $x=f(n+1)$ would yield a contradiction. The statement would be the following:
$(f(f(n+1))+1)\times f(f(f(n+1)))=f(n+1)\times(f(n+1)+1)$
The right hand side of this is at least $(n+1)(n+2)$, but the left hand side is less than or equal to $(n+1)n$. Immediate contradiction.
So we are certain that $f(n+1)\geq n+1$ and $f(f(n+1))\geq n+1$.
At the same time, you can't have $f(n+1)>n+1$, right? Otherwise the equal sign in $(f(n+1)+1)\times f(f(n+1))=(n+2)\times (n+1)$ would have to be replaced by "$>$".
Therefore, $f(n+1)=n+1$.