I think I am close to the answer, but still not sure how to finish it. $$a^2b^2(a^2b^2+4)=2(a^6+b^6)$$ I know I can rewrite that equation as $$(a^2-b\sqrt2)(a^2+b\sqrt2)(a\sqrt2-b^2)(a\sqrt2+b^2)=0$$ Thus $$a^2=\pm b\sqrt2 \space\space\space\space\space\space b^2=\pm a\sqrt2$$ Now, am I allowed to substitute one equation into the other? This would mean that $b=0$ or $b=\pm\frac{\sqrt[3]4}{\sqrt[3]{\sqrt2}}$. But then when I try $b=0$ I get $a=0$ as well.
How can I prove at least one of them is irrational?
As Wojowu pointed out, you can't plug one equation into another since you only know at least one of them holds. However, you can deduce from both equations individually that $a$ and $b$ cannot both be rational, unless they are both zero.
Suppose the first equation holds: $a^2 = \pm b\sqrt{2}$. Then $a^4=2b^2$, so $2=\frac{a^4}{b^2}$ (assuming $b \neq 0$). Why is this impossible if $a$ and $b$ are both rational? Do a similar argument for the other equation.