I have two square matrices $A= U_a \Sigma_aU_a^\top $ and $B= U_b \Sigma_bU_b^\top$ of same dimension.
I have that $U_a^\top B U_a$ is a diagonal matrix. Can I conclude than that $U_a^\top U_b= I$?
Because in my simulations $U_a^\top U_b \neq I$. How does the rank of the two matrices is going to affect the commutativity and joint diagonalizability property of the two matrices? What can we say in terms of subspaces?
Thank you
Let $V = U_a^\top U_b$. In your situation we have $V \Sigma_b V^\top$ diagonal. As you notice, this does not imply that $V = I$.
We can see $V$ as the transition matrix between the basis given by $U_b$, in which $B$ is diagonal, to the basis given by $U_a$, in which $B$ is also diagonal. So we need to understand to which extend an orthonormal eigenbasis is unique.
There are two sources of non-uniquemess.
First, a symmetric matrix uniquely defines its set of eigenvalues, but not the order of eigenvalues. Any permutation of basis elements will give us a new eigenbasis. In this case $V$ would be a permutation.
Second, the orthonormal eigenvectors are also not uniquely defined by the matrix, only the eigenspaces are. This is easy to see in the completely degenerate case where we have only one eigenvalue — in this case $\Sigma$ is a multiple of identity and $V \Sigma V^\top = \Sigma$ for every orthogonal $V$. But even for a simple eigenvalue we have a freedom to choose one of two normalized eigenvectors $u$ and $-u$.
In general, if we have eigenvalues $\lambda_i$ with multiplicities $m_i$, so we can write the diagonalized matrix as a block diagonal matrix $$\Sigma = \lambda_1 I_{m_1} \oplus \dots \oplus \lambda_k I_{m_k},$$ then for an orthogonal matrix $V$ we have $V \Sigma V^\top$ diagonal if and only if $$V = P (U_1 \oplus \dots \oplus U_k)$$ for some permutation matrix $P$ and orthogonal matrices $U_i \in \mathbb{R}^{m_i \times m_i}$.
In particular, if rank of your matrix $B$ is small, it has a large multiplicity for $\lambda = 0$ and your $V$ will have a large orthogonal block in rows corresponding to the kernel of $B$.