I am a physicist but I do really love maths and I would like to learn and have a deep understanding of the maths used in theoretical physics, just for leisure, in my free time.
I know there is a book called "The geometry of Physics", by Frankel, which deals with many results and gives the proofs, but I wonder if you know a more "mathematician" way to learn things such as differential geometry, algebraic topology, the connection group-manifold. The knowledge of QM and of the physical part I think I have it.
Nakahara's book covers a lot of ground and is an overall very nicely written book. In terms of a mathematician's treatment, I recommend looking over Frankel or Nakahara, picking out topics that you're interested in, and look up resource recommendation for each topic. To get you started, some things (in my opinion) any self-respecting physicist should have a grasp of are:
For general mathematical maturity, as well as topological aspects of physical systems (e.g. field theories), you need to get acquainted with topology:
The setting of most physical theories (e.g. GR & QFTs in general backgrounds) is some smooth manifold, you need to get acquinted with basic differential geometry.
For these see this Math SE thread. I personally enjoy the two books by Lee.
Classical and quantum gauge theories are described by bundles associated to principal bundles of the symmetry group, we need
For these, see this thread.
Again I think this is the basics of what every theoretical physicist should familiarize themselves with, and to have a decent grasp of basic theories in modern physics. Again Nakahara is the place to start, and a good reference to come back to for summaries.