My original ODE was
$$m{x}''+\beta {x}'+\kappa x(1+ax^{^{2}})=u$$ however the term $(1+ax^2)$ it is confusing me
What I did was to change my $x$ to $x_1=x$; $x_2=x'=x_1'$ in order to find the state space matrix $A$. However I still have a non-linear term; then I found the partial derivatives with respect to $x_1$ and $x_2$ and $u$, then my $A$ matrix became.
I think I have to linearize the system around an equilibrium point, however the HW does not specify an exact equilibrium point for $x$; it just states that the output is the position of the mass. Should I assume that $x=0$? $$ A=\begin{bmatrix} 0 & 1\\ \frac{-k}{m}-3\frac{kax^{2}}{m} & \frac{-b}{m} \end{bmatrix}x + \begin{bmatrix} 0 \\ 1/m \end{bmatrix}u $$
My final goal is to find the poles of the system by finding the $det(sI-A)=0$ in order to design a PID or PI controller
Your equation is a nonlinear Duffing-type equation
$$m\ddot{x}+\beta \dot{x}+\kappa x(1+ax^{^{2}})=u.$$
introduce $x=x_1,\dot{x}=\dot{x}_1=x_2, \ddot{x}=\dot{x}_2$ to obtain the state space representation
$$\dot{x}_1 = x_2$$ $$\dot{x}_2 = -\dfrac{\kappa}{m} x_1(1+ax_1^2)-\dfrac{\beta}{m}x_2+\dfrac{1}{m}u.$$
Note, that you cannot put this into the form
$$\boldsymbol{\dot{x}}=\boldsymbol{Ax}+\boldsymbol{b}u$$
because your equation is nonlinear.
In order to design a linear controller for this system, you will have to linearize the system at a specific operating point $x_{1,\text{ref}},x_{2,\text{ref}}$ and $u_\text{ref}$. If you are only interested in the output $y=x_1=x$ then you can also use exact feedback linearization, which helps you to guarantee that you do not lose important dynamics, which would happen if you would use standard linearization on the system.
Linearization of an operating point $\boldsymbol{x}_\text{OP}$ and $\boldsymbol{u}_\text{OP}$ of the differential equation
$$\dot{\boldsymbol{x}}=\boldsymbol{f}\left(\boldsymbol{x},\boldsymbol{u} \right)$$
is given as
$$\dfrac{d}{dt}\left[\boldsymbol{x}-\boldsymbol{x}_\text{OP} \right]=\boldsymbol{f}\left(\boldsymbol{x}_\text{OP},\boldsymbol{u}_\text{OP}\right)+\left.\dfrac{\partial \boldsymbol{f}}{\partial \boldsymbol{x}}\right|_{\boldsymbol{x}_\text{OP},\boldsymbol{u}_\text{OP}}\left[\boldsymbol{x}-\boldsymbol{x}_\text{OP} \right]+\left.\dfrac{\partial \boldsymbol{f}}{\partial \boldsymbol{u}}\right|_{\boldsymbol{x}_\text{OP},\boldsymbol{u}_\text{OP}}\left[\boldsymbol{u}-\boldsymbol{u}_\text{OP} \right]$$
As an example, I will show this for your equation at $x_{1,\text{ref}}=x_{2,\text{ref}}=0$ and $u_\text{ref}=0$ as the reference point.
$$\dfrac{d}{dt}\left[x_1-x_{1,\text{ref}}\right]=0+\begin{bmatrix}0&1 \end{bmatrix}\begin{bmatrix}x_1-x_{1,\text{ref}}\\x_2-x_{2,\text{ref}} \end{bmatrix} +\begin{bmatrix}0 \end{bmatrix}\begin{bmatrix}u-u_\text{ref}\end{bmatrix}$$ $$\dfrac{d}{dt}\left[x_2-x_{2,\text{ref}}\right]=0+\begin{bmatrix}-\dfrac{k}{m}-\dfrac{3ka}{m}x_{1,\text{ref}}&-\dfrac{\beta}{m} \end{bmatrix}\begin{bmatrix}x_1-x_{1,\text{ref}}\\x_2-x_{2,\text{ref}} \end{bmatrix} +\begin{bmatrix}\dfrac{1}{m} \end{bmatrix}\begin{bmatrix}u-u_\text{ref}\end{bmatrix}$$
Using your reference point will lead to
$$\dot{x}_1=\begin{bmatrix}0&1 \end{bmatrix}\begin{bmatrix}x_1\\x_2\end{bmatrix} +\begin{bmatrix}0 \end{bmatrix}\begin{bmatrix}u\end{bmatrix}$$ $$\dot{x}_2=\begin{bmatrix}-\dfrac{k}{m}&-\dfrac{\beta}{m} \end{bmatrix}\begin{bmatrix}x_1\\x_2\end{bmatrix} +\begin{bmatrix}\dfrac{1}{m} \end{bmatrix}\begin{bmatrix}u\end{bmatrix}.$$
You should be able to deduce $\boldsymbol{A}$ and $\boldsymbol{b}$ from this.