This means that
$ay+1
=(1-\dfrac{b}{a}+\dfrac{b}{a}(1+by)^r)^r
=\dfrac1{a^r}(a-b+b(1+by)^r)^r
$
or
$a^r(ay+1)
=(a-b+b(1+by)^r)^r
$.
If $y=0$,
both sides are $a^r$.
Otherwise
the derivative of
the left side
with respect to $y$
is
$a^{r+1}$
and that of the right side is
$(b(1+by)^r)'(a-b+b(1+by)^r)^{r-1}
=b^2(1+by)^{r-1}(a-b+b(1+by)^r)^{r-1}
$
which is definitely not the same.
In particular,
that at $0$ is
$b^2a^{r-1}
$
which is not the same
unless $a=b$.
If $a=b$,
then $r=1$
so that
$(a-b+b(1+by)^r)^r
=(b(1+by)^r)^r
=b(1+by)
$
so the two expressions
are equal.
(I'm the editor.Thanks for your answer.By the way,can you suggest some books about special-functions?)
I get that the result is true only if $a=b$ in which case $W(a, b, x) = x$.
Note that if $a=2b$ then $W(x) = x+bx^2/2$ so
$\begin{array}\\ W(W(x)) &=W(x)+bW(x)^2/2\\ &=x+bx^2/2+b(x+bx^2/2)^2/2\\ &=x+bx^2/2+b(x^2+bx^3+b^2x^4/4)/2\\ &=x+bx^2+b^2x^3/2+b^3x^4/8\\ &\ne x\\ \end{array} $
Let's see what $W$ is.
$\begin{array}\\ W(a, b, x) &=W(x)\\ &=\sum_{n=1}^{\infty} \dfrac{x^n}{n!}\prod_{k=1}^{n-1}(a-kb)\\ &=\sum_{n=1}^{\infty} \dfrac{x^nb^{n-1}}{n!}\prod_{k=1}^{n-1}(\dfrac{a}{b}-k)\\ &=\sum_{n=1}^{\infty} \dfrac{x^nb^{n-1}}{n!}\prod_{k=1}^{n-1}(r-k) \qquad r = \dfrac{a}{b}\\ &=\sum_{n=1}^{\infty} \dfrac{x^nb^{n-1}}{rn!}\prod_{k=0}^{n-1}(r-k)\\ &=\dfrac1{a}\sum_{n=1}^{\infty} x^nb^{n}\binom{r}{n}\\ &=\dfrac1{a}(-1+\sum_{n=0}^{\infty} x^nb^{n}\binom{r}{n})\\ &=\dfrac1{a}(-1+(1+bx)^r)\\ \end{array} $
Therefore
$\begin{array}\\ W(W(y)) &=\dfrac1{a}(-1+(1+bW(y))^r)\\ &=\dfrac1{a}(-1+(1+\dfrac{b}{a}(-1+(1+by)^r))^r)\\ &=\dfrac1{a}(-1+(1-\dfrac{b}{a}+\dfrac{b}{a}(1+by)^r)^r)\\ \end{array} $
You claim that this is $y$.
This means that $ay+1 =(1-\dfrac{b}{a}+\dfrac{b}{a}(1+by)^r)^r =\dfrac1{a^r}(a-b+b(1+by)^r)^r $ or $a^r(ay+1) =(a-b+b(1+by)^r)^r $.
If $y=0$, both sides are $a^r$.
Otherwise the derivative of the left side with respect to $y$ is $a^{r+1}$ and that of the right side is $(b(1+by)^r)'(a-b+b(1+by)^r)^{r-1} =b^2(1+by)^{r-1}(a-b+b(1+by)^r)^{r-1} $ which is definitely not the same.
In particular, that at $0$ is $b^2a^{r-1} $ which is not the same unless $a=b$.
If $a=b$, then $r=1$ so that $(a-b+b(1+by)^r)^r =(b(1+by)^r)^r =b(1+by) $ so the two expressions are equal.
(I'm the editor.Thanks for your answer.By the way,can you suggest some books about special-functions?)