Let's consider the following nonlinear system,
$$\dot{x}_1=x_2^3+u$$ $$\dot{x}_2=-u$$ $$y=x_1$$
Taking the derivative of $y$,
$$\dot{y}=\dot{x}_1=x_2^3+u \qquad \text{Equation 1}$$
Let $f_1(x) = x_2^3$ and $v=\dot{y}$ then Equation 1 becomes
$$u=v-f_1(x) \qquad \text{Equation 2}$$
Let $v=\dot{y}_d-e$ where $e=y_d-y$ and $y_d$ is a continuous differentiable trajectory. Then isolating input $u$ in Equation 2 yields,
$$u = \dot{y}_d-e-f_1(x)$$
or equivalently,
$$u=\dot{y}_d-e-x_2^3 \qquad \text{Equation 3}$$
Defining the internal state $\eta$ as $$\eta=\dot{x}_2$$ How can I show that the internal dynamics is unstable?
Substituting Equation 3 into Equation 1,
$$\dot{y} = \dot{y}_d-e$$
Noting that $\dot{e} = \dot{y}_d - \dot{y}$, we are left with
$$\dot{e}+e=0\qquad \text{Equation 4}$$
Substituting $-u=\dot{x}_2$ into Equation 3 we get,
$$-\dot{x}_2+x_2^3=\dot{y}_d-e\qquad \text{Equation 5}$$
In the view of facts that e is guaranteed to be bounded by Equation 4 and $\dot{y}_d$ is assured to be bounded, we have the following condition
$$|\dot{y}_d-e|\leq D$$
where D is a positive constant. We can conclude from Equation 5 that
$$|x_2|\leq D^{1/3}$$
Then, when
$$\dot{x}_2<0 \Rightarrow x_2<-D^{1/3}$$
and when
$$\dot{x}_2>0 \Rightarrow x_2>D^{1/3}$$
According to these two Equations, the system has negative velocity at negative position and positive velocity at positive position, respectively. This result proves that the internal state ($\eta$) of $\dot{x}_2$ is indeed unstable.