I am trying to factor $a^3b-ab^3+a^2+b^2+1$.
I have tried factoring out an $a$ in the first two terms and a $b$ in the third and fourth terms, but get $a^2(a+b)-b^2(b-a)+1=a^2(a+b)+b^2(a-b)+1$. I see no obvious way to factor it. Can you give me a hint?
I started with: $a^3b - b^3a = ab(a-b)(a+b) = (a^2 - ab)(b^2 + ab)$
Then to work in the remaining terms:
$((a^2 - ab)+1)((b^2 + ab) + 1) = a^3b - b^3a + (a^2-ab) + (b^2 + ab) + 1$
It was more about noodling around than any algorithmic approach.