A matrix $A\in M_n(K)$ is diagonalizable iff $\exists\hspace{.03cm}P\in GL_n(K):\ P^{-1}AP$ is diagonal matrix. Then $B=P^{-1}AP$ is called similar to $A$. I realize this is the conjugate element concept in abstract algebra (group theory).
So I wonder if the diagonalization and group theory are related, and if one can study on diagonalization/Jordanization of matrix by using the group theory, but not linear algebra ?
Yes.
You might find a book like Chevalley's The Classical Groups of interest; it studies these groups using tools from all sorts of areas of mathematics, from Abstract Algebra to differential topology (because groups like $GL_n(C)$ or $SO(n)$ or $U(n)$ all end up having natural manifold structures as well, structures in which the group product and the group inverse are both continuous, and the relationship between this continuity and the underlying algebra is the start of things like topological groups or even Lie groups.