I am stuck in solving the cubic equation which does arise from the calculation of the characteristic polynomial of a Hessian matrix.
Starting from the beginning, the problem is a max-min problem:
$$f(x, y, z) = yz + x^2y + x^2 + y^2 - z^2$$
$$\nabla f(x, y, z) = (2xy + 2x, z + x^2 + 2y, y - 2z)$$
Solutions of $\nabla f = 0$ are the points
$$P = (0, 0, 0)$$ $$Q = \left(\sqrt{\dfrac{5}{2}}, -1, -\dfrac{1}{2}\right)$$ $$K = \left(-\sqrt{\dfrac{5}{2}}, -1, -\dfrac{1}{2}\right)$$
The general Hessian reads
$$ H = \begin{pmatrix} 2y + 2 & 2x & 0 \\ 2x & 2 & 1 \\ 0 & 1 & -2 \end{pmatrix} $$
The point P is a saddle point, since the eigenvalues are both positive and negative.
But when I am going to calculate the characteristic polynomial in $P$ and or $K$ I obtain, for example in $P$, the equation
$$-\lambda^3 + 15\lambda + 20 = 0$$
How can I solve this?
You just face a depressed cubic equation.
Following the steps given here, there three real roots since $\Delta=2700$. So, use the trigonometric method described in the linked page $$\lambda_k=2 \sqrt{5} \cos \left(\frac{2 \pi k}{3}-\frac{1}{3} \cos ^{-1}\left(\frac{2}{\sqrt{5}}\right)\right) \qquad \text{with} \qquad k=0,1,2$$