I am not expert in linear algebra. I couldn't come up with any solution for my problem with my limited knowledge. So the question may be even silly or have no solution, I don't know. But I appreciate your help anyhow. I have the following equality: $$ RS = \beta RT + (1-\beta)R $$ where $\beta$ is a scalar, $R\in \mathcal{R}^{3x3}$ is a rotational part and $S\in \mathcal{R}^{3x3}$ is a symmetrical part of matrix $A\in\mathcal{R}^{3}$. Also $T\in \mathcal{R}^{3x3}$ is a symmetrical part and $R\in \mathcal{R}^{3x3}$ is a rotational part of another matrix called $B\in\mathcal{R}^{3}$.
Is there any way to estimate $\beta$ and $T$ if I know $R$ and $S$? Or to remove unknown matrix $T$ from the equation and find $\beta$?
You should definitely read up on the so-called matrix equations. Manipulation with square matrices is quite similar to school algebra: Multiply by the inverse matrix of R: $$ R^{-1}RS = \beta R^{-1}RT + (1-\beta)R^{-1}R $$ You now obtain: $$ S = \beta T + (1-\beta)I $$ Now you can obtain for T: $$ T = 1/\beta(S - I) + I ,$$ where I is an identity matrix. Now since you have two unknowns and one equation, you need another one that involves T and $ \beta $ as unknowns and other known parameters from your problem.