I have a math background and am familiar with multivariable calculus but seem to get lost when I read how it may be related to "gauge symmetries" of fundamental particles. The nuts and bolts physics of it so to speak.
Maybe someone has run across a math book or math books that point the way assuming someone is already familiar with linear algebra and multivariable vector calculus...at least familiar still to some degree.
In general the best answer is probably multiple different books. But, one book that may contain all the topics to get you started is "The Geometry of Physics -- An Introduction" by Theodore Frankel.
This book covers a variety of the topics necessary to pursue the study you mentioned. If you look this up on Amazon, you should be able to read over the table of contents and possibly parts of the first chapter.