Let $S \in M_3(\mathbb R)$ is some invertible matrices, find a base and dimension of subspace $U$, where $U$ is set of all matrices $A\in U$ for which $S^{-1}AS$ is some diagonal matrix.
I only know one base and that is Identical matrix since $S^{-1}IS=S^{-1}S=I$ and that is diagonal matrix, is there any other base?
Let $S \in M_n(\mathbb{R})$ be a fixed invertible matrix. The definition is $$U = \{A \in M_n(\mathbb{R}) : S^{-1}AS \text{ is a diagonal matrix}\}$$
Let $A \in U$ such that $S^{-1}AS = D = \operatorname{diag}(\lambda_1,\ldots, \lambda_n)$. We have
$$A = S^{-1}DS = S^{-1}(\lambda_1E_{11} + \cdots + \lambda_n E_{nn})S = \lambda_1 S^{-1}E_{11}S + \cdots+ \lambda_n S^{-1}E_{nn}S$$
where $(E_{rs})_{ij} = \delta_{ir}\delta_{js}$. Therefore
$$U = \operatorname{span}\{S^{-1}E_{11}S, \ldots, S^{-1}E_{nn}S\}$$
Furthermore, the set $\{S^{-1}E_{11}S, \ldots, S^{-1}E_{nn}S\}$ is linearly independent as an image of a linearly independent set $\{E_{11}, \ldots, E_{nn}\}$ by the invertible linear map $X \mapsto S^{-1}XS$.
Hence $\{S^{-1}E_{11}S, \ldots, S^{-1}E_{nn}S\}$ is a basis for $U$.