As we know, a LQG has no guaranteed stability margins. And that's very bad if you seeking robustness for a system.
But a LQR has guaranteed stability margins and a LQR has very high strength against disturbances.
Here is a diagram of a LQR with integral action:
And here is the state space model:
$$ \begin{bmatrix} \dot{x}\\ \dot{x_i} \end{bmatrix} = \begin{bmatrix} A & 0\\ -C & 0 \end{bmatrix}\begin{bmatrix} x\\ x_i \end{bmatrix}+\begin{bmatrix} B & 0 \\ 0 & I \end{bmatrix}\begin{bmatrix} u\\ r \end{bmatrix} \\ u = L x + L_i x_i $$
I know that the transfer function of the model is: $$ G_{SYS}(s) = C(s I - A)^{-1} B$$
And the transfer function of the whole LQR-system:
$$G_{LQR}(s) = L(s I - A)^{-1} B$$
But when I have a integral action included into the LQR system. What will the transfer function be then?
You are right about the transfer function of the openloop system, so your first transfer function. However your second transfer function is wrong, because the poles of that closed loop can be unstable if the eigenvalues of $A$ lie in the right half plane. I am not entirely sure where you define the input for LQR, so I am not sure about the matrices you have to pre and post multiply, however in the middle you should have $(s\,I - A - B\,L)^{-1}$. This should make sure that the closedloop poles will be located in the open left half plane.
For the transfer function of the LQR with integral action you can use the fact that the Laplace transform of the derivative of $x$ is equal to $s$ times Laplace transform of $x$. So using that $u = L\,x + L_i\,x_i$, then the differential equation can also be written as
$$ \underbrace{ \begin{bmatrix} \dot{x} \\ \dot{x}_i \end{bmatrix}}_{\dot{z}} = \underbrace{ \begin{bmatrix} A+B\,L & B\,L_i \\ -C & 0 \end{bmatrix}}_{A_z} \underbrace{ \begin{bmatrix} x \\ x_i \end{bmatrix}}_{z} + \underbrace{ \begin{bmatrix} 0 \\ I \end{bmatrix}}_{B_z} r. $$
Taking the Laplace transform from both sides, while using that $X(s)$ is the Laplace transform of $x(t)$, and then solving for $Z(s)$ yields
$$ s\,Z(s) = A_z\,Z(s) + B_z\,R(s), $$
$$ Z(s) = \left(s\,I - A_z\right)^{-1} B_z\,R(s). $$
The output of the system is defined as $y = C\,x + D\,u$, so the transfer function from the input $r(t)$ to the output $y(t)$ can be written as
$$ G_{LQI}(s) = \frac{Y(s)}{R(s)} = \begin{bmatrix} C+D\,L & D\,L_i \end{bmatrix}\left(s\,I - A_z\right)^{-1} B_z. $$
This is the closedloop, also known as the complimentary sensitivity. But in order to calculate the margins of this control law you need the openloop, plant times controller $G_{SYS}(s)\,K(s)$. The complimentary sensitivity and this openloop are related as follows
$$ G_{LQI}(s) = (I + G_{SYS}(s)\,K(s))^{-1} G_{SYS}(s)\,K(s). $$
Solving this equation for the openloop yields
$$ G_{SYS}(s)\,K(s) = G_{LQI}(s) (I - G_{LQI}(s))^{-1}. $$
For this openloop transfer function you can draw the Bode and/or Nyquist diagram and extract the margins you are interested in.