A population is a summation of all the organisms of the same group or species, which live in a particular geographical area, and have the capability of interbreeding
so i would like to understand the math behind this. i don't know where to start.
its for example group theory adequate for this, or maybe population and groups are not the same?
because I don't know groups are able of interbreeding?
maybe is not math i need to search but biology?
I would like to understand the dynamics of populations.
what about statistics?
what about Mathematical statistics?
This is Mathematical Biology! Many Mathematics and Statistics department have research groups in this area. Also many university math departments offer a Mathematical Biology course. It usually has the prerequisites: Calculus, Linear Algebra and possible Differential Equations. A good place to start is the text:
A Biologist's Guide to Mathematical Modeling: Otto and Day
And I am sure there are many resources on line. In particular you might be interested in biomath applications to population ecology. Typical models (but not all) are differential equation models which model the change in density of a species.
Here is a very basic model (Logistic Growth Population Model) for the change in population density (over time) for some species. The population density is denoted $P$.
$$\frac{dP}{dt}=rP(1-\frac{P}{K})$$
This population has a growth rate $r$ and a carrying capacity $K$. The carrying capacity is the total number of animals the environment can support. Now that you have an equation for species $P$, you can couple this equation (model) with equations for other species sharing the environment.
If you do not wish to model the total abundance of a certain species in your ecosystem, and you wish to account for individuals, you might consider cellular automaton or even structuring your population on a graph and utilizing some graph theory.
Essentially all of these models work by making a few (sometimes more than a few) assumptions about your population and only accounting for the dynamics that you think really matter. Models become very difficult to analyze if they have too many parameters.