Can someone please help me to linearize this system, which is given by the differential equation shown in the picture below.
All variables are expressed as deviations from initial values (0, for all variable states (x, x)).
My task is to find the model described as the one shown in the attached picture. Find the matrix A, and display the linearized model in the matrix form just like in the picture.
We define $y:=\dot x$ and plugging this into (1) + some algebra yields
\begin{align} m \dot y+2c(x^2-1) y+kx=0 \\ \Longleftrightarrow \dot y=-\frac{2c}{m}(x^2-1)y-\frac{k}{m}x. \end{align}
To get back into the notation of your picture, we set $\mathbf{X}:=(x,y)^T$ and get the system
$$\dot{\mathbf{X}}=f(\mathbf{X})=\begin{pmatrix}y \\ -\frac{2c}{m}(x^2-1)y-\frac{k}{m}x \end{pmatrix}. $$
Now, as seen in the picture, the matrix $A$ is the Jacobian of $f$ evaluated at some point $\mathbf{X}_0:=(x_0,y_0)^T$, i.e.
$$A=\begin{pmatrix} 0 & 1 \\ -\frac{k}{m}-\frac{4c}{m} xy & -\frac{2c}{m}(x^2-1) \end{pmatrix}\Bigg|_{(x_0,y_0)}=\begin{pmatrix} 0 & 1 \\ -\frac{k}{m}-\frac{4c}{m} x_0y_0 & -\frac{2c}{m}(x_0^2-1) \end{pmatrix}$$
and the linearization of the ODE at some point $\mathbf{X}_0$ is given by
$$\dot{\mathbf{X}}=A\mathbf{X}.$$
For example around the origin we have
$$\dot{\mathbf{X}}=\begin{pmatrix} 0 & 1 \\ -\frac{k}{m} & \frac{2c}{m} \end{pmatrix} \cdot \begin{pmatrix} x \\ y \end{pmatrix}= \begin{pmatrix} y \\ -\frac{k}{m}x+\frac{2c}{m}y \end{pmatrix}$$