There's the following footnote on the first page of Deligne-Mumford:
The idea of enlarging the category of varieties for the study of moduli spaces is due originally, we believe, to A. Weil.
So, what could supposedly have been the definition of a moduli space according to older-style algebraic geometry? This is in contrast to the modern definition of a fine moduli space being a representable functor, and the similar but slightly more involved definition for coarse moduli space.
On
Deligne and Mumford are referring to Weil's original construction of the Jacobian of a curve (which is the moduli space of degree zero invertible sheaves, or (equivalently) of degree zero divisors up to linear equivalence). Weil's construction proceeded by first building an open subset of the Jacobian (by applying Riemann--Roch to generically chosen divisors), and then using the group law to move this open set around to cover the whole Jacobian. The Jacobian itself was then obtained by gluing together these open subsets.
Because the construction was via gluing, Weil didn't know whether or not the Jacobian was a projective variety (although he knew it was complete), and so he enlarged the category of varieties from (quasi-)projective varieties to varieties that were simply required to be locally affine.
(Of course, it was understood in fairly short time that Riemann's theory of line bundles on Jacobians and projective embeddings could be extended from the complex case to the general case, and hence that Jacobians are indeed projective varieties.)
As for your more general question, a "moduli space" simply means a space in algebraic geometry parameterizing a certain collection of objects up to some appropriately chosen equivalence. The precise categorical definition you are thinking of is due to Grothendieck, but obviously the general idea doesn't need to be formulated categorically. Martin Brandenburg's answer gives one easy example. And it the whole point of the Jacobian of a curve (which goes back to Abel, Jacobi, and Riemann) is that its points are in a natural bijection with degree zero divisors mod linear equivalence.
The idea behind moduli spaces is very, very old, and can be formulated without functors and schemes. The moduli of a geometric object are just some parameters which determine it (as far as I understand it). If the set of these parameters can be endowed with the structure of a variety, we get a (coarse) moduli space.
For example, a conic section is cut out by a polynomial of the form $A_1 x_0^2 + A_2 x_1^2 + A_3 x_3^2 + 2 B_1 x_0 x_1 + 2 B_2 x_0 x_2 + 2 B_3 x_1 x_2$. The conic section only depends on the point $[A_1:A_2:A_3:B_1:B_2:B_3]$ in $\mathbb{P}^5$. Accordingly, the moduli space of conic sections is $\mathbb{P}^5$. (This includes also the degenerate conic sections. If you want to exclude them, consider the open subset of $\mathbb{P}^5$ on which the determinant $\Delta = \det(A_1,B_1,B_2;B_1,A_2,B_3;B_2,B_3,A_3)$ does not vanish).