Let the implicit equation $$F(x,y)=0, \quad (x,y)\in\mathbb{R}^2$$ defines a curve $\gamma$.
The question is what properties must have the function $F$, s.t. the curve $\gamma$ be topologically equivalent to circle?
It seems if $\operatorname{Hess}F$ is positive defined on some simply-connected domain containing the curve, then the curve must be closed. Is it correct? Is there a theorem about this topic?
That's not quite correct. For example, your curve could be empty, or a single point, or a line segment. If the Hessian matrix is positive definite on some convex region $R$, then $F$ is a convex function there. If $R$ is bounded, contains $\gamma$ and $\inf_R F < 0 \inf_{\partial R} F$, then $\gamma$ is the boundary of the convex open set $\{(x,y) \in R: F(x,y) < 0\}$, and this is homeomorphic to a circle.