Consider the system:
$ \dot x_1 = x_2 $
$ \dot x_2 = {x_1}^2 $
It is obviously unstable, I say that because by inspection the derivative of $x_2$ can never be negative.
How can we concretely show that, in the general case? If we were to prove stability we can use Lyapunov, but we cannot use that in the opposite way.
There is an easy way to find out what the trajectories of your system look like: you just need to show that it is a Hamiltonian system.
Sketching the phase portrait
A system
\begin{cases} \dot{x}_1 = f(x_1,x_2) \\ \dot{x}_2 = g(x_1,x_2) \\ \end{cases}
is said to be Hamiltonian if there exists a smooth, real-valued function of two variables $H(x_1,x_2)$ such that $\partial H/\partial x_1 = g(x_1,x_2)$ and $\partial H/\partial x_2 = -f(x_1,x_2)$. For your system, we need $H$ to satisfy $\partial H/\partial x_1 = x_1^2$ and $\partial H/\partial x_2 = -x_2$. A quick integration tells us that any function of the form
$$ H(x_1,x_2) = \frac{x_1^3}{3} - \frac{x_2^2}{2} + C\qquad\qquad (C\in\mathbb{R}), $$
does the job (we'll just choose $C = 0$ for simplicity). Now, Hamiltonian systems are interesting because their trajectories actually lie on the contour curves $H(x_1,x_2) = k$. If we plot all curves whose equation is $x_1^3/3 - x_2^2/2 = k$, for all real values of $k$, we have actually sketched the phase portrait!
Doing it for a few values of $k$ will produce something like this ($x_1$ is on the $x$-axis, $x_2$ on the $y$-axis):
Presence of a stable manifold
Now that we know how the flow looks like, we just need to find the direction of the arrows. For $k\neq 0$, every trajectory is a smooth curve of $\mathbb{R}^2$ that contains points arbitrarily far away from the origin, a fact that becomes obvious once we rewrite $x_1^3/3 - x_2^2/2 = k$ as:
$$ x_1 = \Bigl(\frac{3}{2}x_2^2 + 3k\Bigr)^{\frac{1}{3}}; $$
here it can be noted that $x_1\to+\infty$ when $x_2\to\pm\infty$. You also know that phase points are forced to move monotonically on their trajectories: they cannot just "stop" midway or, worse, reverse their path and start moving backwards. This means that almost every initial condition will eventually go to infinity, thus proving the instability of your fixed point.
But pay attention to my "almost": what I said fails for $k=0$, when the contour line is
$$ x_1 = \sqrt[3\,]{\frac{3}{2}}x_2^{2/3}. $$
In my picture, it's the one with a cusp at $(0,0)$. This equation does not represent a trajectory! It actually represents the union of three different orbits: the fixed point at the origin, and the two remaining branches of the curve. Now, $\dot{x_2} > 0$ in the fourth quadrant: this means that if you choose an initial condition on the lower portion of that contour line, the phase point will flow upwards until it gets close to the origin, which cannot be crossed because it is an equilibrium point; so the phase point will just slow down, always approaching the fixed point but never quite making it there.
Saying it rigorously, that line is the stable manifold of $(0,0)$.