I am working on the following exercise:
Let $f(x,y) = x^3y^2(1-x-y)$, where $(x,y) \in \mathbb{R}^2$. Where in $\mathbb{R}^2$ is $f$ strictly concave/convex?
I calculated:
\begin{align*} f_x &= -x^2y^2\left(4x+3y-3\right)\\ f_y &= -x^3y\left(2x+3y-2\right)\\ f_{xx} &= -6y^2x\left(2x+y-1\right)\\ f_{xy} &= -x^2y\left(8x+9y-6\right)\\ f_{yx} &= -x^2y\left(8x+9y-6\right)\\ f_{yy} &= -2x^3\left(3y+x-1\right)\\ \end{align*}
So we have the gradient and the Hessian of $f$ as:
$$ \nabla f(x,y) = (-x^2y^2\left(4x+3y-3\right), -x^3y\left(2x+3y-2\right))^T$$
$$H_f(x,y) = \begin{bmatrix} -6y^2x\left(2x+y-1\right) & -x^2y\left(8x+9y-6\right)\\ -x^2y\left(8x+9y-6\right) & -2x^3\left(3y+x-1\right) \end{bmatrix}$$
I do not know how to examine strict convexity/concavity here as the Hessian seems too complicated to calculate it's definiteness via eigenvalues. Is there some kind of trick?