I'm having trouble solving the first part of this problem. I would like your help.
Question:
Show that all maximal solution of the non-linear system
$x_1'=2x_1x_3+x_2$
$x_2'=-x_1+2{x^2}_3$
$x_3'=-{x_1}^2-x_2x_3$
in $\mathbb{R}^3$ is defined in all real line. Verify if the solution in $(0,0,0)$ is a stable equilibrium or assintotically stable.
proof of second part: We will try to obtain a Lyapunov function in the quadratic format $V(x_1, x_2, x_3)=a{x_1}^2+b{x_2}^2+c{x_3}^2$, with $a,b,c>0$.
if $f(x_1,x_2,x_3)=(2x_1x_3+x_2, -x_1+2{x^2}_3, -{x_1}^2-x_2x_3)$, then
$$\langle \nabla V(x_1, x_2, x_3), f(x_1,x_2,x_3) \rangle$$
$$={x_1}^2x_3(4a-2c)+x_1x_2(2a-2b)+x_2{x_3}^2(4b-2c)$$
taking $a=b=1$ and $c=2$, then
$$\langle \nabla V(x_1, x_2, x_3), f(x_1,x_2,x_3) \rangle=0\leq 0$$
By Lyapunov theorem, we have that $(0,0,0)$ is a stable equilibrium.
You have found a function $V$ that is not only a Lyapunov function but a first integral. Consequently, for any solution its values belong to a level set of $V$. Such a level set is an ellipsoid, therefore a compact set.
Now, there is a theorem stating that if the values of a maximally defined solution are contained in a compact set then the solution is defined on $(-\infty,\infty)$; see, e.g., Corollaries 2.15 and 2.16 on pp. 52-53 of G. Teschl's book Ordinary Differential Equations and Dynamical Systems.