I'm trying to prove that a system on nonlinear differential equations is not asymptotically stable to the origin.
The overall problem is an inverted pendulum on a cart that can be controlled with an input u, and uses the variables $ \theta, \dot{\theta}, \ddot{\theta}, y, \dot{y}, \ddot{y} $.
I converted the problem into State Space via
$X = \begin{bmatrix}x_1\\x_2\\x_3\\x_4\end{bmatrix} = \begin{bmatrix}\theta\\ \dot{\theta} \\ y \\ \dot{y} \end{bmatrix}$
The State Space becomes
$\dot{X} = f(X) + g(X)*u$
The output is $y = x_3$
I am trying to prove that the system either is or is not minimum phase.
I am stuck trying to show that
$\begin{bmatrix}\dot{x_1}\\\dot{x_2}\end{bmatrix} = \begin{bmatrix}\ \dot{\theta}\\ \ddot{\theta} \end{bmatrix} = \begin{bmatrix}\ x_2 \\ \frac{mgL\sin(x_1)}{(J+mL^2)} \end{bmatrix}$
is not asymptotically stable to the origin. Generally, if the system is linear, I can look at the real part of the eigenvalues of the A matrix to determine stability. For proving a nonlinear system asymptotically stable to the origin, I can use a Lyapunov function to show asymptotic stability.
I am unsure how to prove this reduced system is not asymptotically stable. By inspection, the system is not asymptotically stable to the origin (the origin being $\theta = 0 =>$ pendulum inverted straight up. The system contains no damping, and would oscillate forever if the initial conditions are not the origin.
Would it be enough to state that because $ (\theta,\dot{\theta}) = (\pi,0) $ is a solution $$\begin{bmatrix}\dot{x_1}\\\dot{x_2}\end{bmatrix} = \begin{bmatrix}\ 0 \\ 0 \end{bmatrix} $$
the system is not asymptotically stable to the origin?
As other user already proposed you can linearize the system at the origin to obtain:
$$\Delta \dot{\boldsymbol{x}}=\begin{bmatrix}0 & 1\\ \dfrac{mgL}{J+mL^2} & 0\end{bmatrix}\Delta \boldsymbol{x}.$$
The characteristic polynomial of this equation is given by
$$\chi(s) =s^2-\dfrac{mgL}{J+mL^2}.$$
The roots of this characteristic polynomial are $$s_{1,2}=\pm\sqrt{\dfrac{mgL}{J+mL^2}}.$$
Hence, by Lyapunov's indirect method via linearization, we see that the origin of the nonlinear system is unstable because one of the eigenvalues has a strictly positive real part.
An alternative way is to use a Lyapunov function for instability.
A possible choice could be the indefinite function
$$V(\boldsymbol{x}) = x_1x_2$$ $$\implies \dot{V}=\dot{x}_1x_2+x_1\dot{x}_2=x_2^2+x_1\dfrac{mgL}{J+mL^2}\sin x_1$$
For $D=\{\boldsymbol{x}\in \mathbb{R}^2\, | -\pi<x_1<\pi\, \}$ it is possible to find an arbitrarly small $\boldsymbol {x}_0 \in D$ such that $V(\boldsymbol{x}_0)>0$. Additionally we have $\dot{V}>0$ for $\boldsymbol{x}\in D$. Hence, by Lyapunov's Theorem for instability the origin is unstable (See: Hassan Khalil's book Nonlinear Systems Theorem 4.3).