$ f\left(x,y\right)=xy\sqrt{1-\frac{x^2}{a^2}-\frac{y^2}{b^2}} $ ; where a,b are constants.
I have tried to differentiate the equation, but I get stuck in a very hard system after this.
The answer of the problem is:
$ f_{max}=\frac{ab}{3\sqrt{3}};f_{min}=-\frac{ab}{3\sqrt{3}} $
$\frac{f(x,y)}{ab}=\pm\sqrt{g(x^2/a^2,y^2/b^2)}$ where $$g(u,v)=uv(1-u-v)\quad(u,v\ge0,u+v\le1).$$ $g_\min=0$ is attained all along the boundary of the triangle, so that $g_\max$ must be attained at some point of the interior ($u,v>0,u+v<1$).
$$\frac{\partial g}{\partial u}=v(1-2u-v),$$ $$\frac{\partial g}{\partial v}=u(1-u-2v),$$ and solving $$1-2u-v=1-u-2v=0$$ you find that $$g_\max=g(1/3,1/3)=1/27$$ hence $$f_\max=\frac{ab}{\sqrt{27}}$$ (and $f_\min=-f_\max$).