There is an overwhelming amount of research regarding homogeneous and heterogeneous grouping in education. The former refers to the practice of grouping "like" students together (regarding age, ethnicity, gender, ability, etc.), while the latter refers to the practice of grouping "unlike" students together.
My particular interest is to compare these practices restricted to as many of the following points as possible:
- Undergraduate mathematics education
- Small, intra-classroom groups
- Grouping based on perceived ability (e.g. placement test scores)
Any research (or anecdotes from experienced teachers) approximating this scenario is most welcome.
I am an experienced high school mathematics teacher - an analogy can be made between how senior maths classes are structured and what you are asking.
It is quite common in secondary education systems to group students according to abilities in their senior years, an example is in the International Baccalaureate, where students are grouped in classes that are taught increasingly complex mathematical topics. The same type of differentiation exists in several Australian state systems (as an example). Perhaps this is something you could look into.
Having taught in these systems, I can see some benefits, primarily that students with a similar interest and skill level are brought together. A key factor in these groupings I have found, that is absent in the younger undifferentiated classes (once again, analogous to an undifferentiated undergraduate class), is that the students often share the same motivation for mathematics.
Hope this helps a bit.