I have system : 2 dof system
Speed of the second mass wanted to be controlled by a PID. The transfer function of the system is :
s/( s^4 - 1.89e-17 s^3 + 2 s^2 - 1.408e-16 s - 3.263e-32)
I could not find a viable PID controller for the system. Even the sisotool could not optimized one. Is there a viable PID controller for this plant? If not how can I know this "instabilizability" from the transfer function?
If you correctly calculate the transfer function and reduce it as much as possible you should get
$$ G(s) = \frac{1}{s^3 + 2\,s}. $$
A PID controller can be written as
$$ C(s) = \frac{D\,s^2 + P\,s + I}{s}, $$
so the open loop, while using this controller, can be written as
$$ L(s) = C(s)\,G(s) = \frac{D\,s^2 + P\,s + I}{s^4 + 2\,s^2}. $$
The closed loop transfer function, given the open loop $L(s)$, can be calculated with $L(s)/(1+L(s))$. Applying this to the open loop which uses a PID controller gives
$$ H(s) = \frac{D\,s^2 + P\,s + I}{s^4 + (2 + D)\,s^2 + P\,s + I}. $$
The closed loop $H(s)$ would be stable if the poles all have negative real part. A fourth order polynomial can also be written as $\prod_{n=1}^4 (s - p_n)$ with $p_n$ the $n$th root of the polynomial. Expanding this expression gives that the coefficient in front of $s^3$ term can also be expressed as $\sum_{n=1}^4 p_n$, but the given denominator has that this coefficient is equal to zero. However, if all poles need to have negative real part then that conflicts with $\sum_{n=1}^4 p_n = 0$. So it is impossible to stabilize this system with just a PID controller.
As a rule of thumb you want a minus one slope as assymptote when the open loop crosses the 0 dB line and (to make things easier for yourself) avoid that the open loop crosses the 0 dB line again. Avoiding crossing the 0 dB again would mean that it would be preferred that the 0 dB crossing lies after the resonance peak of the system. However, the slope after the resonance peak has an assymptote of minus three, so in order to get a minus one slope the controller would locally have to have a slope of plus two. A PID controller can at most add a slope increase of one, so from this it can also be concluded that a PID controller would be insufficient to control this system.