I have the following matrix inequality that I need to express as an LMI
$ (AQ+BY) Q^{-1} (AQ+BY)^T - Q + \sum_i (A_i Q+B_i Y) Q^{-1} (A_i Q + B_i Y)^T < 0$
$Q > 0$
The matrices $A$, $A_i$, $B$ and $B_i$ are given. I need to find a solution for Y and Q.
I tried using the Schur complement (but failed).
So my question is: How do I make this transformation?
As mentioned in the comment from LinAlg, you can turn this into LMI's by introducing intermediate matrix inequalities and apply Schur's complement to each of those. Your first term is of the same form as each term inside summation and for simplicity I will include it into the summation. I will assume that each matrix is real, otherwise this answer would only hold if you would replace each transpose with the Hermitian transpose.
Using from your last matrix inequality that $Q$ and thus also $Q^{-1}$ are positive definite. Now since each term in the summation pre- and post multiplies $Q^{-1}$ with a matrix and the transpose of that matrix respectively, then each term has to be semi-positive definite. And the summation of semi-positive definite matrices is also semi-positive definite. When using that each summation term is upper bounded by some (as of yet unknown) semi-positive definite matrix $M_i=M_i^\top\succeq0$, then using those semi-positive definite matrices instead in your first matrix inequality should yield a more conservative matrix inequality. Now by letting each $M_i$ also be a variable then the solver would be able to let each $M_i$ get arbitrarily close to the actual summation term. However, doing this does allow you to reformulate the original matrix inequality as LMI's. So your matrix inequalities can also be written as
\begin{align} Q = Q^\top &\succ 0 \tag{1} \\ Q - \sum_i M_i &\succ 0 \tag{2} \\ M_i = M_i^\top &\succ (A_i\,Q + B_i\,Y)\,Q^{-1} (A_i\,Q + B_i\,Y)^\top\ \forall\,i \tag{3} \end{align}
where only $(3)$ is not a LMI yet. Applying Schur's complement to $(3)$ yields
$$ \begin{bmatrix} M_i & A_i\,Q + B_i\,Y \\ (A_i\,Q + B_i\,Y)^\top & Q \end{bmatrix} \succ 0\ \forall\,i \tag{4} $$
which makes $(1)$, $(2)$ and $(4)$ a system of LMI's which is equivalent to your two matrix inequalities.
It can be noted that this approach only works because each summation term can be shown to be semi-positive definite. Depending on whether all summation terms are added or subtracted or if each summation term can be shown to be semi-negative definite, then the intermediate matrices might either still be upper a bound or a lower bound, such that they make the initial matrix inequality more conservative. So if a summation term would have been indefinite you would not have been able to use this approach.