I want to determine the nature of the critical point $(x_0, y_0, z_0)=(0,0,0)$ for the function
$$ f(x,y,z)=\alpha x^2+\beta(y^2+z^2)+\gamma x y+\delta xz $$ where $\alpha,\beta,\gamma,\delta\in\mathbb{R}$ and $\alpha,\beta$ have the same sign.
I found the Hessian $$ \begin{pmatrix} 2\alpha & \gamma & \delta\\ \gamma & 2\beta & 0\\ \delta & 0 & 2\beta \end{pmatrix} $$
and that it has eigenvalues $$ \lambda = 2\beta, \alpha +\beta \pm \sqrt{\alpha^2+\beta^2+\gamma^2+\delta^2-2\alpha\beta}. $$
I am stuck on what to do if $4\alpha\beta = \gamma^2+\delta^2$. In this case, the eigenvalues are $$ \lambda = 0, 2\beta, 2\alpha+2\beta $$ and since $\alpha$ and $\beta$ have the same sign, this means we have all non-zero eigenvalues being the same sign and at least one eigenvalue being zero so the test is inconclusive??
How can I determine the nature of the critical point in this case?
Yes. The test is inclusive and further deductions through 1st-order and 2nd-order partial differentiations are not possible. An example is the following function $$ f(x,y,z)=x^2+y^2+\alpha z^4. $$ This function has only one minimizer candidate point $(x,y,z)=(0,0,0)$. Note the word candidate here. By this word, we mean that if $f(x,y,z)$ has any minimizer, it has to be $(0,0,0)$, not that it necessarily is. The Hessian is rather interesting: $$ H=\begin{bmatrix} 2&0&0\\ 0&2&0\\ 0&0&12\alpha z^2 \end{bmatrix} $$ which is equal to $ \begin{bmatrix} 2&0&0\\ 0&2&0\\ 0&0&0 \end{bmatrix} $ in $(x,y,z)=(0,0,0)$. However, the point $(0,0,0)$ is a minimizer only for $\alpha\ge 0$ and is a saddle point for $\alpha<0$. Such a case happens for your function (nevertheless, you may still be able to determine the nature in other ways).
Further Reading
You are also encouraged to read about First-Order Necessary Condition (FONC), Second-Order Necessary Condition (SONC) and Second-Order Sufficient Condition (SOSC).