I can numerically solve this system using ode45 (as i understand this is very similar to the runge kutta 4th order method) in matlab no problem:
$x_1'=x_2-x_1^3+3x_1^2-x_3-3.25$
$x_2'=1-5x_1^2-x_2$
$x_3'=\frac{(4(x_1+1.6)-x_3)}{200}$
With initial conditions $x_1=-1.31, x_2=-7.32, x_3=3.35$.
However when adding the fourth equation:
$x_4'=\frac{x_2'x_1-x_1'x_2}{x_1^2+x_2^2}$
ode45 does not give me the correct solution. The fourth equation has derivatives $x_1'$ and $x_2'$ and I am unsure if I need to make a transformation in order to solve, or use a different method entirely.
ode45 is a Runge-Kutta embedded method. Since you have the explicit value of $x_1'$ and $ x_2 '$ you just need to put them in $x_4'$. Once you've found the initial condition for $x_4$, your system becomes:
$$\begin{cases} x_1'=\dots \\ x_2'=\dots \\ x_3'=\dots \\ x_4'=\frac{(1-5x_1^2-x_2)x_1- (x_2-x_1^3+3x_1^2-x_3-3.25)x_2}{x_1^2+x_2^2} \\ \vec{x(0)}= [-1.31,-7.32,3.35,x_4(0)]^T \end{cases}$$
A numerical computation of the system, using $x_4(0)=-0.85$, using ode45 in MatLab gives the following numerical solution