My friend's challenge: prove that $\sqrt {\frac {a^2+b^2}{ab+1}} \in \mathbb N$

Question

A few days ago, I told my friend that I wanted to learn trigonometry by the end of $2018$. His immediate reply was that I should go on if I could solve this mathematics problem:

a and b, given that $a\text{ and } b \in \mathbb N$ and $\left( \left( a^2+b^2 \right) \div \left( ab+1 \right)\right) \in \mathbb N$, then prove that $\sqrt{\frac {\left( a^2+b^2 \right) }{ \left( ab+1 \right)}} \in \mathbb N$

Over the last few days, I went about trying to see if I could prove it. (My friend has a tendency to occasionally trick me with unsolvable math problems) this is what I proved:

1. If $a \text{ xor } b=0$, it works.
2. If $b=a^3 \text{ and } a \neq 1$, it works.
3. Besides $n=0$, $x^n=a \text{ and } x^{n+2}=b $will work

To recap, one number must be $2$ powers greater than the other (i.e. $32$ and $128$. [$2^5$ and $2^7$]).

Or, at least what I thought... When I sent an email to him, he said that I missed 2 very seldom noticed things and that everyone he had ever asked the problem missed it. Which either means he intentionally made the problem too hard for my intelligence ( which is -spoiler alert- very low, compared to you people), or he is just making fun of me for not noticing his problem is impossible. Either way, a plethora of mathematicial geniuses whom are much more experienced than me at algebra could solve this easily. Could someone tell me what I'm missing?? Thanks in advance. (And DON'T tell my friend about this!!)

Answer 1

Your friend is evil$^*$: this was problem $6$ in the $1988$ IMO, which was ... difficult. The wiki page on the technique Vieta jumping has some details.

Quoth said page (actually, quoth Engel):

Nobody of the six members of the Australian problem committee could solve it. Two of the members were husband and wife George and Esther Szekeres, both famous problem solvers and problem creators. Since it was a number theoretic problem it was sent to the four most renowned Australian number theorists. They were asked to work on it for six hours. None of them could solve it in this time. The problem committee submitted it to the jury of the XXIX IMO marked with a double asterisk, which meant a superhard problem, possibly too hard to pose. After a long discussion, the jury finally had the courage to choose it as the last problem of the competition. Eleven students gave perfect solutions.

---

$^*$And I'm joking less than it may sound. Assuming things are as you've described them, I find it extremely rude that your friend brought this problem up in the context described. It's ridiculous to claim even that success as a professional mathematician relies on one's ability to solve problems like this.

Don't let your friend discourage you - if what you want to do is learn trigonometry, go for it!