I derived the following matrix inequality for finding a stabilizing observer-based feedback gain $K$.
$$ \begin{bmatrix}-\lambda P& \tilde{A}^T & 0\\ \tilde{A} &-P^{-1} & \tilde{B}\\ 0 & \tilde{B}^T & -(1-\lambda)I\end{bmatrix}\preceq0 $$ where $0<\lambda<1$ is a known parameter that I can set at will, $\tilde{B}$ is a known constant matrix, $P\succeq0$ is a semipositive definite matrix variable and $$ \tilde{A}=\begin{bmatrix}A+BK & -BK\\ 0 & A-LC\end{bmatrix} $$ with $A$, $B$, $C$ and $L$ constant known matrices. The matrix $K$ is a variable. It is known that if there exist such $P$ and $K$ satisfying this inequality, $A+BK$ is a stabilizing solution (all eigenvalues of $A+BK$ are inside the unit ball).
This problem is nonlinear, and I am having trouble to linearize it. The standard procedure would be a congruence transformation by multiplying both sides by $Q=P^{-1}$, resulting in
$$ \begin{bmatrix}-\lambda Q& Q\tilde{A}^T & 0\\ \tilde{A}Q &-P^{-1} & \tilde{B}\\ 0 & \tilde{B}^T & -(1-\lambda)I\end{bmatrix}\preceq0 $$
But the product $\tilde{A}Q$ is still nonlinear. To make is simple, let
$$Q=\begin{bmatrix}Q_1 & 0\\ 0 & Q_2\end{bmatrix}$$
Then, $\tilde{A}Q$ results in $$ \tilde{A}=\begin{bmatrix}AQ_1+BKQ_1 & -BKQ_2\\ 0 & AQ_2-LCQ_2\end{bmatrix} $$ A solution would be calling $KQ_1$ say $W$ and $KQ_2$ say $V$, then the relation is linear. But there is another constraint between $W$ and $V$ which is nonlinear.
Does someone has an idea of how to linearize the problem?
First, your original matrix inequality formulation is not the usual one I would have expected, and I'm not sure how you derived it. However, a stabilizing, state-feedback based controller can be found by solving the (decoupled) LMIs: $$ QA^T + AQ + BY + Y^TB^T < 0, \qquad A^TP + PA + WC + C^TW^T < 0, $$ With $Q>0$ and $P>0$. The controller is then $K=YQ^{-1}$ and the observer is $L=P^{-1}W$. The proof that these decoupled LMIs indeed correspond to the coupled stability problem is not trivial. It is given in Boyd's book, Linear Matrix Inequalities in System and Control Theory on page 111, and continued on page 117. The book can be downloaded for free here.