I googled eigenfunction and look it up in wikipedia, but still I do not know where I should start to learn the subject. I have two questions, and allow me to repeat the title of this question.
What subjects should I study to learn about eigenfunctions?
What good textbooks would you recommend for learning the subject?
Thank you for any answers.
This is not a very good answer to your question, but based on the article it looks like you might actually want to study partial differential equations. Eigenfunction expansion is a nice technique to help understand these equations, in exactly the same way eigenvectors let us understand linear transformations (some PDEs actually are just linear transformation, in a sense).
Since you want to understand PDE involved in population dynamics, I would suggest you look at a more applied book on PDEs. I unfortunately don't know any. A quick search for "Applied PDE Book" gave this as the number one on Amazon: http://www.amazon.com/Applied-Partial-Differential-Equations-Edition/dp/0130652431
This book appears to cover the topics you need to understand the paper, based on a very casual glance of both the paper and the contents of the book. I don't know the level of difficulty of the text, so be prepared to review or learn new things in linear algebra or analysis.
Many of the comments suggest learning functional analysis. I initially wanted to recommend this, but I'm afraid that if you go out and pick up a typical book on functional analysis, it will be too abstract and difficult to transfer to your paper. A sufficiently advanced PDE book should include the techniques from functional analysis you need.