Complicated real to real functional equation

Question

Find all $f:\mathbb{R} \rightarrow \mathbb{R}$ satisfying $$f(f(y)+x^2+1)+2x=y+(f(x)+1)^2$$ for all $x,y \in \mathbb{R}.$

So far I have proved that $f$ is bijective. How should I continue?

Answer 1

Let $$ I=\{\,x\in\mathbb R\mid \forall y\in\mathbb R\colon f(y+x)=f(y)+x\}$$ (so that specifically $f(x)=f(0)+x$ for all $x\in I$) and $$ S=\{\,x\in\mathbb R\mid f(x)=x\,\}.$$ Clearly, $I$ is an additive subgroup of $\mathbb R$.

Let $a=x^2+1$ and $b=(f(x)+1)^2-2x$. Then for all $y$ we have $ f(f(y)+a)=y+b$, hence $$f(y+a+b)= f(f(f(y)+a)+a)=f(y)+b+a$$ so that $$\tag1(f(x)+1)^2+(x-1)^2\in I\qquad\text{for all }x\in\mathbb R.$$ Let $x,y\in I$. Then also $-x\in I$ and the functional equation reads $$ f(y+f(0)+(\pm x)^2+1)\pm 2x=y+ (\pm x+f(0)+1)^2$$ i.e., $$f(y+f(0)+x^2+1) = y+f(0)+x^2+1+ f(0)(1\pm 2x+f(0))$$ and by subtracting both variants $f(0)x=0$. From $(1)$ we see that $I$ is not trivial, hence we can pick $x\ne0$ and obtain $f(0)=0$, hence $$\tag2 I\subseteq S.$$

Remark. If we add the condition that $f$ be continuous, we get immediately from $(1)$ that $I$ contains an unbounded connected set, hence $I=\mathbb R$. We try to continue with not necessarily continuous $f$.

Immediately form the functional equation we get $$\tag3 x,y\in S\implies y+x^2+1\in S.$$ and also from $(1)$ $$\tag4 x\in S\implies 2x^2+2\in I.$$ Specifically, $2\in I$ because $0\in S$, so $2\mathbb Z\subseteq I$. Then from $(2)$ and $(3)$ we have $\mathbb Z\subseteq S$. Thus $1\in S, 2\in I$ allows us to conclude using $x=1$ that $$\tag5 f(f(y))=f(f(y)+1^2+1)-2=y$$ so that $f$ is an involution.

Plugging $y=-1\in S$ into the functional equation, we get $$\tag6f(x^2)=(f(x)+1)^2-2x-1.$$ For $z\in\mathbb R$ let $x=-\frac{z+1}2$. Using the functional equation for both sides of $$ f(y+x^2+1)=f((y+z)+(x+1)^2+1)$$ we see that $f(y)-f(y+z)$ depends only on $x$, not on $y$. With $f(0)=0$ we obtain $$ \tag7f(y+z)=f(y)+f(z),$$ i.e., $f$ is additive.

Finally, assume for some $c\in\mathbb R$ that $f(c)\ne c$. Then from $(5)$ and $(7)$ we obtain $f(f(c)-c)=c-f(c)$, i.e., there exist $v=f(c)-c\ne 0$ with $f(v)=-v$. Per additivity we may replace $v$ with an integer multiple of itself and so can assume wlog. thet $v>9$. Then we can write $v=x^2$ with $x>3$ and obtain from $(6)$ $$ -x^2=f(x^2)=-2x-1+(f(x)+1)^2$$ and so $$ 2=(f(x)+1)^2+(x-1)^2\ge (x-1)^2>4.$$ To resolve this contradiction, we have to drop the assumption that $f(a)\ne a$. We conclude

Proposition. $f(x)=x$ for all $x\in\mathbb R$.