I ran into this problem a few days ago, and have been trying to prove it, but all my attempts have been unsuccessful.So I am asking for your help. Here's the problem:
If $xy=(x+y)^2-5(x+y)+8$ for some $x,y\in\mathbb{R}$ then prove the following inequality holds $x^2+y^2\leq8$.
Let $p=x+y$, $q=x-y$, then $$\frac{p+q}{2}\frac{p-q}{2}=p^2-5p+8$$ $$3p^2-20p+q^2+32=0$$ $$q^2=-3p^2+20p-32 \ge 0$$ So $$4\ge p \ge \frac{8}{3}$$ $$p^2+q^2=-2p^2+20p-32=18-2(p-5)^2 \le 18-2(4-5)^2=16$$ $$x^2+y^2=\frac{1}{2}(p^2+q^2)\le 8$$