I know that this question has already been asked, but date of sending solutions passed (30th Sep), so this part of contest is inactive.
$$|(ax+by)(ay+bx)|\le x^2+y^2$$ $$|ab(x^2+y^2)+xy(a^2+b^2)| \le x^2+y^2$$ $$|k(a^2+b^2)+ab| \le 1$$ where $k=\frac{xy}{x^2+y^2}$
$-\frac12\le k \le \frac12$ since $x^2+y^2\ge \pm 2xy$
Then I tried to treat left hand side as quadratic, but it did not lead to anything useful. On the other hand from https://www.desmos.com/calculator/ljmcqbpobn it follows that there are $a,b$ s.t $a^2+b^2>2$, so I guess there are some mistakes in my approach.
For $x=y=1$: $(a+b)^2\le 2$, so $a^2+b^2+2ab\le 2$.
For $x=1$ and $y=-1$: $(a-b)^2\le 2$, so $a^2+b^2-2ab\le 2$.
$$a^2+b^2=\frac{(a^2+b^2+2ab)+(a^2+b^2-2ab)}{2}\le\frac{2+2}{2}=2.$$