$Ml\ddot\theta = (M + m)\dot x + mL\ddot\theta = u$
$M\ddot x = u - mg\theta$
Using the variables $x_1 = \theta, x_2 = \dot\theta, x_3 = x, x_4 = \dot x$
Rewrite the system in the form $\dot x = Ax + bu.$ Where $x = (x_1,x_2,x_3,x_4)^T, A$ is a 4 by 4 matrix, b is a 4 by 1 vector and $u$ is scalar.
This is a problem dealing with an inverted pendulum system.
I have searched online but cannot find how to even start this problem. Please any help even if its just pointing me in the right direction would be amazing. Thank you in advance $:)$
Using the indicated change of variables, your system reads:
\begin{align} & m L \, \dot{x_2} = -(M+m) \, {x_4} + u \\ & M \dot{x_4} =- m g \, x_1 + u, \end{align} which has to be integrated together with:
\begin{align} & \dot{x_1} = x_2, \\ & \dot{x_3} = x_4. \end{align} Do you recognize the matrix $A$ and the vector $b$ from here? (It's advisable to divide by $mL$ and by $M$ in the two first equations, respectively)
Cheers!
Edit: let's take a step further on writing this system into matrix form. Note that the equations above are equivalent to:
\begin{align} \dot{x}_1 & = x_2 \\ \dot{x}_2 & = - \frac{M+m}{mL} \, x_4 + \frac{1}{mL}u \\ \dot{x}_3 & = x_3 \\ \dot{x}_4 & = - \frac{mg}{M} x_1 + \frac{1}{M}u \end{align} Define now the vector $x = (x_1,x_2,x_3,x_4)^T$, so the system above can be rearranged up to:
$$ \dot{x} = A x + b u,$$ where the entries of the matrix $A$ can be figured out by noting that $\dot{x}_j = A_{ij} x_j + b_j u.$ The first row of $A$ is, for instance $(0,1,0,0)$ while the first component of $b$ is $0$, because the RHS of the first equation (first row) can be read as $\dot{x}_1 = 0 x_1 + 1 x_2 + 0 x_3 + 0 x_4 + 0 u$. The fourth row of $A$ is then $(-mg/M,0,0,0)$ and the fourth component of $b$ is $1/M$.
I'm sure you can take it from here.