I have an unstable system that doesn't have a useable output when open-loop excitation is applied.
Subsequently I've used a controller to control its output. I want to use the system identification tool on MATLAB, but don't know how to calculate the open-loop input to the plant for a closed-loop system.
All the reading material is advanced and theoretical and has largely gone over my head. Can anyone explain how this open-loop plant input can be derived from known parameters of the controller and the plant output?
Assuming you have a single input and single output system you can use the following method. I assume that the transfer function of the controller is given by $C(s)$ and the transfer function of the plan is given by $G(s)$. Assume a reference signal $r(s)$ and an output $y(s)$. The transfer function from the reference to the output is given by
$$F(s)=\dfrac{G(s)C(s)}{1+G(s)C(s)}.$$
In the Laplace domain, we have
$$y(s)=F(s)r(s)$$
in the time domain we have
$$y(t) = \mathcal{L}^{-1}\left[F(s)r(s)\right].$$
Note that we can measure $y(t)$ given a specific $r(s)$, without knowing the true $F(s)$. Now, come up with different approximations $G_m(s|\theta_m)$ for the plant transfer fucntion $G(s)$, in which $m$ is the index of the approximation and $\theta_m$ is the vector of parameters of the model. And determine $F_m(s|\theta_m)$ to cacluate
$$y_m(t|\theta_m)=\mathcal{L}^{-1}\left[F_m(s|\theta_m)r(s)\right].$$
The last step is to use some reference $r(s)$ and calculate the system response $y_m(t|\theta_m)$. Then you can use the least squares approximation to minimize the sum of squared errors
$$E(\theta_m)=\sum_t [y(t)-y_m(t|\theta_m)]^2.$$
You can use this procedure for different approximations and choose the best model. This method will require you to apply a nonlinear least squares method. In order to derive the analytical inverse Laplace transform you should able to use Maple, Mathematica, Sympy (from Python) or Matlab.
As commented by @SampleTime you can do the same procedure with the $G_m(s|\theta_m)$ looking only at the output $y(t)$ and the input $u(t)$ the problem with this approach is that you need to be able to measure the control input, which might be difficult for some problems. Additionally, you might get problems as a simple reference input might lead to very complicated control inputs. But you can try both methods and see which works best.