Given the function $f(x,y) = 4y^2 +6yx^2 + 17$, I found the single stationary point $(0,0)$.
I then tried to use the Hessian matrix to classify it as a local minimum/maximum/saddle point, but the determinant is zero. I'm not sure how else I can classify this point. Any help would be much appreciated!
In this kind of situation the Hessian test is inconclusive so you can try to prove/disprove the existence of a local maxima/minima by definition.
For simplicity's sake I'll work instead with the function $f(x,y) = 4y^2 +6yx^2$ since the $17$ doesn't change the nature of the stationnary point.
With this new function we have $f(0,0) = 0$.
Notice that for $y \geq 0$ we have $f(x,y) \geq 0$. Indeed if $f(x,y)$ was negative then we would have \begin{align*}4y^2+6yx^2 < 0 &\iff 4y^2 < -6yx^2 \\ & \iff y < -\frac{3}{2}x^2 < 0 \end{align*} which is a contradiction.
Therefore $f$ cant have a local maximum at $(0,0)$ since the function takes larger value whenever $y>0$.
To show that $(0,0)$ is a local minimum we must show that $f$ is positive on a small open neighbourhood around $0$. That is
$$ \exists \epsilon >0 : \vert (x,y)\vert < \epsilon \implies f(x,y) \geq 0. $$
We've already shown that $f$ is positive when $y$ is positive. If $y\leq 0$ then we have
\begin{align*} 4y^2+6yx^2 \geq 0 &\iff 4y^2 \geq -6yx^2 \\ &\iff y \leq -\frac{3}{2}x^2 \end{align*}
Let $1 > \epsilon >0$ then consider
$$\tilde x = \epsilon/2$$ $$\tilde y = \frac{1}{2} \frac{-3}{2}\tilde x^2 > \frac{-3}{4}\tilde x^2$$
By our choice for $\tilde x$ and $\tilde y$ we have $f(\tilde x,\tilde y) < 0$ but also $\tilde x^2 + \tilde y^2 < \epsilon^2$ which means that $f$ cant be a local minimum.
Since $(0,0)$ isn't a local maxima or local minima it is a saddle point.