I was reading Goro Shimura's The Map of My Life. He wrote the following quote in the book. It made me come up with the title question. In particular, is there any application of the theory of modular forms in physics?
A well known math-physicist Eugene Wigner was in our department, and so I occasionally talked with him. He was pompous and took himself very seriously. That is the impression shared by all those who talked with him. At a departmental party, he asked me what kind of mathematics I was doing. He asked that question as if he met me for the first time. At that time I had been a full professor and his colleague at least for six years, perhaps more. I told him vaguely, “Well, mainly things related to modular forms.” Then he said, “Oh, modular forms; we physicists don’t need such” in a very contemptuous tone. I can add that there are some physicists who are interested in modular forms. Once Edward Witten attended my graduate course on Siegel modular forms, and often asked sensible questions in class.
For studying applications of the number theory in physics see NUMBER THEORY IN PHYSICS by Matilde Marcolli.
Also there's a website called "Number Theory and Physics Archive" that is useful.
Finally, you can read Examples of number theory showing up in physics question on TP.SE.