Find all functions $f : \mathbb{Z} \to \mathbb{Z}$ such that $f(x-f(y)) = f(f(x)) - f(y) - 1$.
So far, I've managed to prove that if $f$ is linear, then either $f(x) = x + 1$ or $f(x) = -1$ must be true. I did this by plugging in $x=0$ to the above equation, which yields $$ f(-f(y)) = f(f(0)) - f(y) - 1$$ and plugging in $x$ instead of $y$ and subtracting, this becomes $$ f(-f(y)) - f(-f(x)) = f(x) - f(y)$$ Assuming $f(x) = ax + b$ then gives \begin{align*} f(-ay-b) - f(-ax-b) = ax - ay &\Rightarrow -a^2y - ab + a^2x + ab = a(x-y) \\&\Rightarrow a^2(x-y) = a(x-y) . \end{align*} Thus $a=1$ or $a=0$. If $a=0$, then the original equation becomes $b = b - b - 1$, thus $b=-1$. If $a=1$, the original equation becomes $$ x-y-b+b = x+2b-y-b-1 \Longrightarrow b=1.$$
I briefly tried finding a quadratic function that works but didn't find anything. So my question is: how can I either show that $f$ must be linear or find all other representations?
I think $f$ has to be linear. You derived the expression
$$ f(-f(y)) = f(f(0)) - f(y) -1. $$ Now let $a=f(0)$ and $t=-f(y)$. Then
$$ f(t) = f(a)+t-1 $$
Edit: plugging in $t=0$ we obtain $f(a)=a+1$, while plugging in $t=a$ we find $a=1$, which gives $f(t)=t+1$, in agreement with what you found.
Edit: As pointed out by Najib Idrissi I did not check for constant solutions. The only constant solution is clearly $f(t)=-1$ as it is promptly inferred from the original equation. However, I believe the expression $f(t) = f(0)+t$ is still the general expression of any non constant solution, regardless of surjectivity. As a matter of fact, we are only interested to see what's the action of $f$ on its range (cause we need to be able to apply $f$ twice). If $f(f(x)) = g(f(x))$ for every $x$, then $f=g$ on $Range(f)$. If after finding $g(x)$ we see that $Range(g)=\mathbb{Z}$, then $f(x)=g(x)$ on $\mathbb{Z}$.
Edit: I finally get completely the point of Najib and Mattia. The expression I found works only if $t$ is in the Range of $f$. Unfortunately, there is no way from there to infer how big $Range(f)$ is. As a matter of fact, it could consist of simply one point. Please, disregard this answer (but keep it as an example of bad reasoning!).