Problem 6 of the 1988 International Mathematical Olympiad notoriously asked:
Let $a$ and $b$ be positive integers and $k=\frac{a^2+b^2}{1+ab}$. Show that if $k$ is an integer then $k$ is a perfect square.
The usual way to show this involves a technique called Vieta jumping. See Wikipedia or this MSE post.
I can follow the Vieta jumping proof, but it seems a bit strained to me. You play around with equations that magically work out at the end. I don't see how anyone could have come up with that problem using that proof.
Is there a natural or canonical way to see the answer to the problem, maybe using (abstract) algebra or more powerful tools? In addition, how can someone come up with a problem like this?
The same Wikipedia page quotes Arthur Engel about the history of the problem.
It is implied that the Australians had tried to solve the problem with more general tools and failed, and that the idea of Viète jumping was created specifically for this problem.
Another reason why Viète jumping is the canonical solution to the problem is that the problems of the IMO usually have only a few solutions, with special prizes given out for particularly ingenious ones, such as to Boreico Iurie for IMO 2005 Q3:
(I got this from a book about the mathematical olympiads in China from 2003 to 2006, which had the IMO problems for completeness.) In particular, the Viète jumping proof is the only one with mathematics simple enough for a competitor to understand.
If you insist on a more "natural" solution, a geometric interpretation of the technique is also on the Wikipedia page, involving lattice points on a hyperbola. Dubuque's answer has more on this, including the possible sources for the problem.