It is well known how to use algebraic geometry (differentials, divisors, and Riemann-Roch) in order to prove the finite dimensionality of the vector space of modular forms of some fixed weight and level.
Is it possible to prove the finite dimensionality of modular forms using representation theory? If so, any references?
On
Depending on what you mean by representation theory the answer is: yes!
Certainly, you can prove it using "analysis". The theorem is: the space of automorphic forms (on a reductive group) of a certain level with a specific $\mathcal Z$-type and $K$-type is finite dimensional.
The proof is based on the fact that a closed subspace of $L^2(X,\mu)$, with $\mu(X)$ finite, made of essentially bounded functions is finite dimensional.
This is Theorem 5.2 in Rudin's "Functional Analysis", where it is attributed to Grothendieck. It is also Lemma 8.3 in Borel's "Automorphic forms", where it is attributed to Godement. The statements are slightly different, so there is not necessarily an attribution error.
Let me sketch the proof for the cusp form case. The general case follows from the cuspidal case and an induction argument (on the rank of the group). For simplicity, $X$ will be $\Gamma\backslash SL_2({\mathbb R})$, where $\Gamma$ is an arithmetic subgroup.
Before you begin, you need:
1) Cusp forms are rapidly decreasing (I should add "in Siegel sets"). This is standard, if somewhat technical.
2) Distributions on $X$ with prescribed $\mathcal Z$- and $K$-types are actually smooth functions. This is essentially an application of elliptic regularity.
3) $L^2_{\rm cusp}(X)$ is closed in $L^2(X)$. This is either obvious or straightforward, depending on what you already know.
(Note that while cusp forms are rapidly decreasing, $L^2_{\rm cusp}(X)$ is not made up of essentially bounded functions, because we have stronger criteria for a function to be a cusp form than just being in $L^2_{\rm cusp}(X)$)
Now that we have these facts at hand, the proof is pretty straightforward.
Cusp forms of specific type are rapidly decreasing (1), so essentially bounded. A convergent (in $L^2$) sequence of cusp forms of a specific type converges weakly to a distribution of the same type (it is necessary to use distributions to talk about the $\mathcal Z$-type of a non-smooth function), hence it was actually a smooth function (2), so it converges to a cusp form (3). So we can deduce that the space of cusp forms of a given type is finite dimensional.
This is not much more than an abbreviated transcription of the argument in Borel's "Automorphic forms on $SL_2({\mathbb R})$". The theorem is originally from Harish-Chandra, in "Automorphic forms on semisimple Lie groups".
The theorem I think Matt E refers to is the much stronger assertion that "irreducible unitary representations of reductive groups over local fields are admissible". This implies that irreducible unitary representations of adele groups are admissible. Which implies that subrepresentations of the space of cusp forms are admissible, which implies automorphic representations are admissible, since they are built up from cuspidal representations in ways that preserve admissibility (parabolic induction and subquotients). This theorem is provable in representation-theoretic terms (though it is surprisingly algebraic: the main tricks are about algebras).
It is important because it lets us factor automorphic representations over primes. It is not hard to show that irreducible admissible representations of adele groups factor over primes, but it wasn't proven that automorphic representations were admissible until it was proven that irreducible unitary representations were.
It is possible to express the finite-dimensionality in representation-theoretic terms: it is amounts to the fact that automorphic forms on $\mathrm{GL}_2(\mathbb A)$ form an admissible representation of $\mathrm{GL}_2(\mathbb A)$. (Actually, this statement is a little stronger, because it includes the Maass form case.)
But I'm not sure that one can prove this purely representation-theoretically. At least, it depends on analytic facts (or algebro-geometric equivalents, if one thinks in terms of Riemann--Roch), which have to be brought into play in one manner or another.