Let $K$ be an algebraically closed field and $\Lambda$ a finite-dimensional $K$-algebra. We know that a representation of $\Lambda$ is:
A ring homomorphsim $\theta:\Lambda \to End_{K}V$, where V is a vector space over $K$.
So, a representation is a $\Lambda$-module (not necessarily finitely generated). My question is that: "Why in most books on this subject, $\mathsf{mod}\Lambda$ is investigated?" By $\mathsf{mod}\Lambda$, I mean the category of finitely-generated $\Lambda$-modules. Why don't they study $\mathsf{Mod}\Lambda$, the category of $\Lambda$-modules?
One reason is that many applications, and connections with other areas of mathematics, specifically involve finite dimensional modules. For just one example, there are links to algebraic geometry (e.g., via quiver varieties) that consider spaces of finite dimensional representations.
Another reason is just that there is a lot of theory, from classical results like the Krull-Schmidt theorem to powerful techniques originating with Auslander and Reiten, that only applies directly to finitely generated modules. So the more advanced theory is rather different for finitely generated and infinitely generated modules.
But people do study infinite dimensional modules as well. In fact, there are increasingly frequently cases where, even if you're really only interested in finitely generated modules, studying infinitely generated modules can turn out be a useful tool.