I am looking for a good book of topological representation. I have a very good insight of representation theory of finite groups, and I want to explore topological representations.
I saw a book by Kirillov, and it looks quite good, but it involves category theory, which I know nothing...
First of all, you should learn about Haa rmeasure, and measure theory on locally compact groups in general. Here, I enjoyed reading Deitmar-Echterhoff "Principles of Harmonic analysis".
Then, you feel the need to supplement this with structure theory. This is also partly covered by that book, but I really mean considering locally compact groups locally as projective limits of Lie groups. I explain this in the fourth chapter of my thesis with references: http://ediss.uni-goettingen.de/bitstream/handle/11858/00-1735-0000-000D-F074-7/palm.pdf?sequence=1
The Standard reference here is Deane Montgomery and Leo Zippin, Topological transformation groups.
Having understood all this, you might want to specify on which vector spaces and with what kind of representations you might want to work then.
There are many general theorem for unitary representation due to Mackey, which generalize the most important results of the finite group case such as the definition of induction, Frobenius reciprocity, Schurs lemma, Maschke decomposition, induction-restriction formulas and group extensions. Rac For finite groups, all complex representations are unitary.
I like Asim O. Barut and Ryszard Raczka, Theory of group representations and applications for this.
As a starting point, you should start with compact groups and understand the Peter-Weyl theorem pretty well.