After multiplying it out I get: $\ f(x,y)=x^2y+xy^2-6xy-2x^2-2y^2+8x+8y-8$
Where I would take the partial derivative
with respect to x and set to zero: $\ y^2+2xy-6y-4x+8=0$,
with respect to y and set to zero: $\ x^2+2xy-6x-4y+8=0$
This is where I am stuck and unable to proceed. Any tips would be helpful.
Since both $x^2-6x-4y$ and $y^2-6y-4x$ are equal to $-2xy-8$, you have $x^2-6x-4y=y^2-6y-4x$, or $x^2-y^2=2(x-y)$. Therefore, $x=y$ or $x+y=2$.
If $x=y$, your first equation becomes $3y^2-10y+8=0$, which has two solutions: $\frac43$ and $2$. So, this provides two critical points: $\left(\frac43,\frac43\right)$ and $(2,2)$.
And, if $x=2-y$, your first equation becomes $2y-y^2=0$, which has two solutions: $0$ and $2$. So, this provides another two solutions: $(2,0)$, and $(0,2)$.