The Art of Problem Solving: Volume 1 by Sandor Lehoczky and Richard Rusczyk - Example 6-14
We are trying to solve the system $xy = -12$ and $x^2 + 2y^2 = 34$, which will eventually help us solve for the square root of an irrational expression. Solving for $y$ in terms of $x$ in the first equation gets us $y = -12/x$, and substituting that into the second equation gets us $x^4 - 34x^2 + 288$ Factoring the equation $x^4 - 34x^2 + 288=0$ will get $(x^2-16)(x^2-18) = 0$. This leads to two integer solutions, $4$ and $-4$, and two irrational solutions $3\sqrt 2$ and $-3\sqrt 2$.
The authors of the book only use the integer solutions. Why can't we use the irrational solutions?
Edit to clarify: They are using the integer solutions to rewrite the expression $\sqrt{34-24\sqrt 2}$ in the form $x + y\sqrt 2$.
As you note, the author wants to find integers $x$ and $y$ such that $$x+y\sqrt{2}=\sqrt{34-24\sqrt{2}},$$ so you are looking for integer solutions only. Do note that for $x=\pm3\sqrt{2}$ you get $y=\mp2\sqrt{2}$ and so $$x+y\sqrt{2}=\pm(3\sqrt{2}-2\sqrt{2}^2)=\mp(4-3\sqrt{2}),$$ which is again the same solution as for $x=\pm4$ and $y=\mp3$.