Let's assume i have a linear system as:
$-\omega^{2}U^{T}(M+\Delta M)U+U^{T}(K+\Delta K)U=0_M$
Where $\Delta M=diag(\begin{bmatrix}dm1&dm2&dm3&dm4&dm5\end{bmatrix})$
$\Delta K=diag(\begin{bmatrix}dk1&dk2&dk3&dk4&dk5\end{bmatrix})$
$\omega=diag(\begin{bmatrix}\omega_1&\omega_2&\omega_3\end{bmatrix})$
$U\in \Re^{5x3}$
$0_M\in \Re^{3x3}$
Note: $diag$ means a matrix with the element on the diagonal.
How can i formulate the problem in an equivalent form:
$Ax=B$
Where
$A\in \Re^{9x10}$
$x=\begin{bmatrix}dm1&dm2&dm3&dm4&dm5&dk1&dk2&dk3&dk4&dk5\end{bmatrix}^{T}$
$B\in \Re^{9x1}$
This is needed in order to find the rank of the system.
What helps to solve your problem is the rule ${\rm vec}\{A \cdot {\rm diag}(b) \cdot C^T\} = (C \diamond A)\cdot b$, where
Let us apply this rule to your problem. First, we rearrange a bit:
$$ \begin{align} -\omega^2U^T(M+\Delta M)U+U^T(K+\Delta K)U&=0_M \\ -\omega^2U^T{\rm diag}\{\Delta m\} U+U^T{\rm diag}\{\Delta k\} U&=\omega^2 U^T M U - U^T K U \end{align} $$ Now, let us vectorize: $$ \begin{align} -(U^T \diamond \omega^2 U^T) \cdot \Delta m +(U^T \diamond U^T) \cdot \Delta k & = {\rm vec} \{ \omega^2 U^T M U - U^T K U\} \\ \left[-U^T \diamond \omega^2 U^T, U^T \diamond U^T\right] \cdot \left[\Delta m^T, \Delta k^T\right]^T & = {\rm vec} \{ \omega^2 U^T M U - U^T K U\}, \end{align} $$ which is in the desired form $A x = b$, where $A = \left[-U^T \diamond \omega^2 U^T, U^T \diamond U^T\right] \in \mathbb{R}^{9 \times 10}$, $x = \left[\Delta m^T, \Delta k^T\right]^T\in \mathbb{R}^{10 \times 1}$, and $b = {\rm vec} \{ \omega^2 U^T M U - U^T K U\} \in \mathbb{R}^{9 \times 1}$.
As a slight simplification, you can rewrite your system matrix $A$ into $$A = \left[-(I_3 \otimes \omega^2) \cdot (U^T \diamond U^T), U^T \diamond U^T\right] = \left[-D \cdot (U^T \diamond U^T), U^T \diamond U^T\right],$$ where $D = {\rm diag}\{\omega_1, \omega_2, \omega_3, \omega_1, \omega_2, \omega_3, \omega_1, \omega_2, \omega_3\} \in \mathbb{R}^{9 \times 9}$, since $\omega$ is diagonal.
Clearly, if your $\omega_i$ are equal, the system has rank at most 5 (not surprisingly). In general, you may be more lucky. I just tried with randomly drawn data and got rank 9. My guess is for randomly drawn $\omega$, $U$ (from a continuous distribution) you get full rank almost surely, but that's a guess only.