I want to use the Poincaré-Bendixson theorem to show that there exists a nontrivial (and periodic) solution to $$z'' + [\log (z^2 +4(z')^2)]z' + z = 0.$$
Therefore I substituted $u = z'$ to get $$u' = - \log(z^2 + 4u^2) u - z, \quad z' = u.$$
We haven't covered the theorem in our classes yet, so I just try to follow the page 10 from these dynamical systems notes. I think that we can differentiate $f$ on every open set disjoint with $(0,0)$, but what would the $\mathcal S$ be here? How to proceed?
Here is the plot of dynamical system mentioned above and a sample trajectory starting from $(-5,5)$.
@Did's hint is correct. Here is a complete answer: Consider a Lyapunov function of the form $$V(u,z)=u^2+z^2.$$ We have $$ V'(u,z)=2(uu'+zz')=-2u^2\log(z^2+4u^2),\\ $$ where the derivative is taken respect to $t$. Now consider the annulus $A$ defined by: $$\frac{1}{4}\leq u^2+z^2\leq1.$$ First, there is no stationary point in $A$ since the only stationary point of the system is $(0,0)$. Second, we verify that $A$ is indeed a trapping region so that we can apply Poincaré-Bendixson theorem:
$$u^2+z^2\leq\frac{1}{4}\implies\log(z^2+4u^2)\leq0\implies V'(u,z)\geq0,$$ $$u^2+z^2\geq1 \implies\log(z^2+4u^2)\geq0\implies V'(u,z)\leq0.$$ This implies that the flows are going outward on $u^2+z^2=\frac14$ the inner boundary of $A$ and going inward on $u^2+z^2=1$ the outer boundary of $A$. Thus, every semi-orbit that starts within $A$ remains in $A$ forever. Thus, $A$ is indeed a trapping region. Now Poincaré-Bendixson theorem implies that there exists a periodic orbit within $A$.