The problem is asking to find the critical points of this function and determine if they are local extrema or saddle points
I found that $$\frac{df}{dx} = \frac{y\cos(xy)}{2+\sin(xy)}$$ and by symmetry $df/dy$ is the same with an x at the top instead of $y$
This gives me that the critical points occur when $(x,y) = (0,0)$ or when $xy$ is an odd multiple of $\pi/2.$
For the second case, I have no idea how to approach this problem. I started by breaking it up into the cases where $\sin(xy)< 0$ and the cases where $\sin(xy) < 0$ and see how that affects the sign of the determinant, but the problem is there are too many working pieces here for me to tell with certainty what the value of the determinant will be.
Any ideas on how to approach this?
Hessian
Taking it from where you stopped i computed the Hessian, and i got \begin{equation} H(x,y) = \begin{bmatrix} \frac{\partial^2}{\partial^2 x} f(x,y) & \frac{\partial^2}{\partial x\partial y } f(x,y) \\ \frac{\partial^2}{\partial y\partial x } f(x,y) & \frac{\partial^2}{\partial^2 y} f(x,y) \end{bmatrix} \end{equation} where after some calculations, i get \begin{align} \frac{\partial^2}{\partial^2 x} f(x,y) &= -\dfrac{y^2\left(1+2\sin\left(yx\right)\right)}{\left(\sin\left(yx\right)+2\right)^2}\\ \frac{\partial^2}{\partial x\partial y }f(x,y) &=-\dfrac{xy\sin\left(xy\right)}{\sin\left(xy\right)+2}+\dfrac{\cos\left(xy\right)}{\sin\left(xy\right)+2}-\dfrac{xy\cos^2\left(xy\right)}{\left(\sin\left(xy\right)+2\right)^2} \\ \frac{\partial^2}{\partial^2 y} f(x,y) &= -\dfrac{x^2\left(1+2\sin\left(yx\right)\right)}{\left(\sin\left(yx\right)+2\right)^2} \end{align}
Hessian at $x=y=0$
The Hessian at $(0,0)$ is \begin{equation} H(0,0) = \begin{bmatrix} 0 & \frac{1}{2} \\ \frac{1}{2} & 0 \end{bmatrix} \end{equation} The determinant is negative, clearly \begin{equation} \det H(0,0) < 0 \end{equation} So, we have a saddle point at $(0,0)$
Hessian at $xy = (2k+1)\frac{\pi}{2}$
Now, on the other points where \begin{equation} xy = (2k+1) \frac{\pi}{2} \end{equation} we get \begin{equation} H(x,y) \Bigg\vert_{ xy = (2k+1) \frac{\pi}{2}}= \begin{bmatrix} - y^2 (\frac{1+2(-1)^k}{((-1)^k + 2)^2}) & - \frac{(2k+1) \frac{\pi}{2} (-1)^k}{(-1)^k + 2} \\ - \frac{(2k+1) \frac{\pi}{2} (-1)^k}{(-1)^k + 2} & - x^2 (\frac{1+2(-1)^k}{((-1)^k + 2)^2}) \end{bmatrix} \end{equation}
The determinant is \begin{equation} \det H(x,y) \Big\vert_{ xy = (2k+1) \frac{\pi}{2}} = x^2y^2 (\frac{1+2(-1)^k}{((-1)^k + 2)^2})^2 - \big( \frac{(2k+1) \frac{\pi}{2} (-1)^k}{(-1)^k + 2} \big)^2 \end{equation} And $$f_{xx}=\frac{\partial^2}{\partial^2 x} f(x,y)\Big\vert_{ xy = (2k+1) \frac{\pi}{2}} = - y^2 \frac{1+2(-1)^k}{((-1)^k + 2)^2}$$ But $xy = (2k+1) \frac{\pi}{2}$ so \begin{equation} \det H(x,y) \Big\vert_{ xy = (2k+1) \frac{\pi}{2}} = \big((2k+1) \frac{\pi}{2} \frac{1+2(-1)^k}{((-1)^k + 2)^2} \big)^2 - \big( \frac{(2k+1) \frac{\pi}{2} (-1)^k}{(-1)^k + 2} \big)^2 \end{equation} When $k$ is odd, the determinant is zero. (Higher order tests may be used).
When $k$ is even, the determinant is positive and $f_{xx} < 0$ (Local Maximum)