What is the rule to "guess" how to multiply both equations of a system so that their sum solves in "good" (perfect squares) numbers?

Question

How did the author guess from the beginning that 1st equation must be multiplied by $3$ and the 2nd - by $17$?

Answer 1

This guess is the only way to remove the constant values on the right hand side of the two equations. $17(3)-3(17)=0$. In general for $$f(x,y)=a$$ $$g(x,y)=b$$ We have $$b\cdot f(x,y)-a\cdot g(x,y)=0$$

Answer 2

You have: $$\begin{cases}x^2+2y^2=17 &(1) \\ x^2-2xy=-3 &(2) \end{cases}$$ It is impossible here to eliminate anything but the $x^2$ terms, and that won't help.

Multiplying $1$ by the constant of $2$, and multiplying $2$ by the constant of $1$ gives us: $$\begin{cases}-3x^2-6y^2=-51 &(3) \\ 17x^2-34xy=-51 &(4)\end{cases}$$

The constants match, so performing $(4)-(3)$, we get $20x^2-34xy+6y^2=0$ Which we may solve by the quadratic formula, as the author did.

We multiply by those constants to get rid of the constant terms, as something set equal to $0$ is much easier to solve.