I am trying to find solution $X \in SO(3)$ for this matrix equation.
$$PX - XQ + YXZ = K$$
where matrices $P, Q, Y, Z, K \in\mathbb{R}^{3\times3}$ are known. At the first glance, it does not seem to have a closed form. However, according to this link, Solve matrix equation $AXB+CX=D$ someone has suggested to take vec operation on both side to linearize the equation. I have followed the same instruction and arrived at
$$(I\otimes P + Q^T\otimes I + Z^T\otimes Y)vec(X) = vec(K)$$
But the solution $X$ does not lie in $SO(3)$ anymore. Can anyone here suggest some constrain to ensure this requirement? Thank you in advance for reply.
We assume that $P,Q,Y,Z$ are generic matrix and let $R$ be your matrix function using tensor products. One has $R.vec(X)=vec(K)$ and $R$ is invertible. Then $vec(X)=R^{-1}.vec(K)$ and $X=vec^{-1}(R^{-1}vec(K))$. Then the required condition (for $X\in O(3)$) is $vec^{-1}(R^{-1}vec(K))(vec^{-1}(R^{-1}vec(K)))^T=I_3$, that is a procedure easy to write on a computer.