I have the dynamical system
$\dot{x_1}=x_2+x_1^2-\frac{1}{4} x_1(x_2-1+2x_1^2)$
$\dot{x_2}=-2(1+x_2)x_1$
and I have shown that the curves $x_2=1-2x_1^2$ and $x_2=-1$ are invariant.
Let $D$ be the region enclosed by the two curves just described. I am trying to make use of the function $H=x_1^2(1+x_2)+\frac{x_2^2}{2}$ to show that trajectories inside this region $D$ converge to the origin as $t\rightarrow-\infty$.
I am not sure what to do here because the only problems of this type that I have seen have to do with convergence as $t\rightarrow\infty$, and it is suitable to use Lyapunov's stability theorem in such a case. So how can I approach this problem?
Convergence with $t\to -\infty$ is essentially the same as with $t\to\infty$, because you can reverse time. Let $$\dot{x} = f(x).$$ Then the function $y(t)=x(-t)$ would satisfy $$\dot{y}=-\dot{x}(-t)=-f(x(-t))=-f(y)$$ and $\lim_{t\to-\infty}x(t) = \lim_{t\to\infty}y(t)$. You can now examine the stability of $y$ with $t\to\infty$ in the usual way.
In consequence, you can see that to examine the convergence as $t\to-\infty$ you can apply the Lyapunov theorem with the opposite signs. In particular if $\frac{d}{dt}H(x(t))>0$ for $x\neq 0$, then from Lyapunov theorem we have instability as $t\to\infty$ and asymptotic stability as $t\to-\infty$.