I am trying to learn the dihedral group in group theory and I feel a bit confused about the composition of rotation and reflection, here are my questions:
In what area of mathematics will I learn the theory of function transformations? What books shall I read?
I would recommend having a look at 'groups and symmetry' by M.A. Armstrong. This book discusses extensively the links between group theory and symmetries of geometric shapes (which is what these transformation are really dealing with). It is also just generally a very good book.
As far as what areas of maths do these types of transformations feature the answer is really anything relating to symmetry and possibly geometry, however, I think that you are being taught about these types of transformations in order to try and give you an intuitive idea about groups and how they work, the real point in looking at these transformations in a group theoretic way is because it turns out that these transformations can be entirely explored algebraically, i.e. by showing that they form a group and then studying that group. We can therefore reduce these hard to visualise transformations into algebraic symbols and a few formulas with which we can then know everything. This is the point and the power of using group theory. Therefore, basically I would focus on the group theory rather than the geometric aspects, it does take a while to get your head aroung the reflection/rotation stuff but rather than looking for a book that covers specifically that (I'm not even sure if there really are any) you are better to cut shapes out of paper and physically do the transformation to try and see what's happening. The book I recommended above also does deal very well with explaining the transformations so that might well be exactly what you are looking for.