While proving an optimal control theorem concerning the controlability criterion they used the following :
consider the system: $$ (S): \begin{cases} \frac{dx(t)}{dt}=A(t) x(t)+B(t) u(t)\\ x(t_0)=0 \end{cases} $$ where $A(t)$, $B(t)$ are two matrices and $u(t)$ is the control. The adjoint system associated to the system $(S)$ is given by:
$$ (S^*): \begin{cases} \frac{dz(t)}{dt}=-A^*(t) z(t)\\ y(t)=B^*(t)z(t)\\ z(t_0)=z_0 \end{cases} $$
I really don't have any clue how they did derive such adjoint system, even on the net I didn't find any method describing how to do it ! does anyone have an idea or maybe the answer ? thank you for your time.
Of course you should first investigate your reference subject to respective definitions. I think they use the ordinary definition of an adjoint linear operator.
You can look at these lecture notes, Section 7.4.