Functional Equation: When $f(x+y)=f(x)+f(y)-(xy-1)^2$

Question

How does one solve the following functional equation when $f:\mathbb{R}\rightarrow\mathbb{R}$

$$f(x+y)=f(x)+f(y)-(xy-1)^2$$

When I assumed it was a polynomial equation, it can be seen through induction that $$f(nx)=nf(x)-\sum _{ i=1 }^{ n-1 }{ (ix^2-1)^2 } $$

for some natural number $n$.

This implies that $$f(n)-nf(1)+\sum _{ i=1 }^{ n-1 }{ (i-1)^2 }=0 $$ is true for all natural $n$, or that for all $n$ $$f(n)-nf(1)+\frac{(n-2)(n-1)(2n-3)}{6}=0$$

Since there are infinite number of solutions to $f(n)-nf(1)+\frac{(n-2)(n-1)(2n-3)}{6}=0$, it can be said that $$f(n)=nf(1)-\frac{(n-2)(n-1)(2n-3)}{6}$$ for all real number $n$.

This implies that $f(n)$ is of the degree $3$, but this is a contradiction since if $f(n)$ had a degree of $3$, the coefficient for $x^2y^2$ would be $0$.

So there appeared to be no polynomial solutions to this function. I further thought that there would be no functions that satisfied this equation.

How does one solve this equation? Any help would be appreciated.

Answer 1

$$f(x+y)=f(x)+f(y)-(xy-1)^2\\ f(2)=2f(1)\\ f(3)=f(2)+f(1)-1=3f(1)-1\\ f(2)+f(2)-9=f(4)=f(1)+f(3)-4\\ 4f(1)-9=4f(1)-5$$

Answer 2

First put $x=y=0$ into the functional equation to obtain that

$$f(0)=1 \tag{1}$$

Then suppose that $y \ne 0$ so dividing the functional equation by $y$ we can write

$$\frac{f(x+y)-f(x)}{y}=\frac{f(y)-1}{y}-\frac{(xy-1)^2-1}{y} \tag{2}$$

$$\frac{f(x+y)-f(x)}{y}=\frac{f(y)-f(0)}{y-0}-\frac{x^2y^2-2xy}{y} \tag{3}$$

Next, take the limit when $y \to 0$ to obtain

$$f'(x)=f'(0)+2x \tag{4}$$

Where we have assumed that $f(x)$ is differentiable over $\mathbb{R}$. Next, integrating $(4)$ we get

$$f(x)-f(0)=f'(0)x+x^2 \tag{5}$$

Or equivalently

$$f(x)=x^2+f'(0)x+1 \tag{6}$$

But we can observe that $(6)$ will never satisfy the functional equation since $f(x)$ is a second degree polynomial and there are fourth degree terms like $x^2y^2$ in the functional equation . So the only thing that we are left with is that

There is no differentiable $f(x)$ which satisfies the functional equation!