Compare two expressions in term of two variables

Question

Suppose that$a>0$ and $b>0$. Compare two numbers: $$ S_1=a^2b^2(a^2+b^2-2), \quad S_2=(a+b)(ab-1). $$ My attempt. If $a=b$, then we have $$ S_1-S_2=2a^4(a^2-1)-2a(a^2-1)=2a(a-1)^2(a+1)(a^2+a+1)\geq 0. $$ Therefore, we think that $S_1\geq S_2$.

Answer 1

We'll prove that $S_1\geq S_2$.

Indeed, let $a+b=2u$ and $ab=v^2$, where $v>0$.

Thus, by AM-GM $u\geq v$ and we need to prove that $$v^4(4u^2-2v^2-2)\geq2u(v^2-1)$$ or $f(u)\geq0,$ where $$f(u)=2v^4u^2-(v^2-1)u-v^6-v^4.$$ But by AM-GM $$f'(u)=4v^4u-v^2+1\geq4v^5-v^2+1=2u^5+2u^5+3\cdot\frac{1}{3}-v^2\geq$$ $$\geq5\sqrt[5]{(2u^5)^2\cdot\left(\frac{1}{3}\right)^3}-v^2=\left(5\sqrt[5]{\frac{4}{27}}-1\right)v^2>0,$$ which says that $f$ increases.

Id est, $$f(u)\geq f(v)=2v^6-(v^2-1)v-v^6-v^4=v^6-v^4-v^3+v=v(v^2-1)(v^3-1)\geq0.$$ Done!