What is the difference between the study of group theory as a mathematical subject and as a physical method in quantum mechanics, for example?
When a person studies group theory, what subjects does he\she focusses on if he will study quantum physics for example?
On
When a physicist says "group theory" she means representation theory, usually unitary representations (as these are the ones relevant to quantum mechanics), and usually of Lie groups. The term "group theory" as mathematicians use it is much more general and includes, for example, geometric group theory.
I think the difference between how a physicist and a mathematician study groups is the same as the difference between them when they use mathematics in general.
Mathematics is a tool to a physicist, a means to an end. To a mathematician, mathematics is the end in itself.
Since I'm criticized about not being informative, let me expand upon my answer. In particular, about the question regarding the use of group theory in quantum mechanics, which I think Qiaochu already answered adequately. But more can be said.
Mathematically, the basic objects of quantum theory are states in Hilbert spaces. These Hilbert spaces will -depending on the application- have additional structure. For instance, if you study wavefunctions for a single particle in space, a natural choice for the Hilbert space is $L^2(\mathbb{R}^3)$. If the particle is free, then the full symmetries of the space $\mathbb{R}^3$ translate to symmetries of the Hilbert space. This has as a consequence that quantum states are superpositions of quantum states that can be labeled by quantum numbers like momentum.
Representation theory is the theory that makes this systematic. It shows how the symmetries of the system work on the Hilbert space level. The actions of the symmetry group (rotations, translations, etc...) on $\mathbb{R}^3$, get a representation on the Hilbert space. Representation theory tells you how these representation decompose into irreducible ones, which are the fundamental building blocks. Interestingly, in particle physics for example, these irreducible representations correspond to the possible particles.
I have illustrated this for the case of symmetries of physical space, but this also works for internal symmetries, those that are related to fundamental forces like electromagnetism. These symmetries form Lie groups, hence the importance of these groups in quantum mechanics and of their representations. But there are also discrete symmetries, think of CPT (charge, parity, time symmetry).