In the evaluation of mathematical expressions, particularly integrals, I often find a statement that the expression has or does not have a closed form. I looked up the definition, and the important point seems to be that closed forms use a finite number of elementary operations, where "elementary" includes exponentiation, logarithms, and trigonometric functions.
I understand that an infinite sum is somehow "not closed", e.g. in the sense that its actual numerical value can only be computed up to a certain precision; and I also understand that $\exp()$ is somehow elementary. However, the actual numerical value of $\exp(x)$ can also only be computed approximately because the function is defined by an infinite sum. So isn't this just a trick, declaring exponentiation elementary and therefore "closed" while it's actually just as infinite and therefore "open" as any other function defined by an infinite sum?
In particular I'm thinking of the error function, $\operatorname{erf}(x) = \frac{2}{\sqrt{\pi}}\int_{0}^x e^{-t^2}\,\mathrm dt$. How is $$ \exp(x) = \sum_{n = 0}^{\infty} {x^n \over n!} $$ more elementary than $$ \operatorname{erf}(x)= \frac{2}{\sqrt{\pi}}\sum_{n=0}^\infty\frac{(-1)^n ~ x^{2n+1}}{n! ~(2n+1)} $$ ?
As others have pointed out this is mostly a matter of convention, but the convention is not completely arbitrary. Note that it has nothing to do with the numerical approximation or infinite series; by that standard you should be just as unhappy with $\sqrt{x}$ or $\sin(x)$ as you are with $e^x$. (Indeed, rational functions are the only functions whose values can be calculated without infinite series.)
Instead, the convention has to do with differential equations. The reason why $e^x$, $\sin x$, and $\cos x$ are considered elementary is that (together with polynomials) they are the only functions you need to write down solutions to differential equations of the form $Dy = 0$ where $D$ is a linear differential operator (i.e. a polynomial in $\frac{d}{dx}$). Since linearization is is the most basic tool in the theory of differential equations in general, exponential functions and trigonometric functions play a central role. Logarithms (and inverse trigonometric functions) appear as inverses of elementary functions.
This discussion is the starting point for a larger theory. One defines a differential field to be a field $F$ equipped with a derivation $d$; the standard example is the field of rational functions equipped with differentiation. Just as one extends ordinary fields by adjoining a root of a polynomial, one can extend differential fields by adjoining a solution to an equation involving the derivation. It turns out that two specific differential field extensions are most fundamental:
This language can help answer questions about the structure of solutions to linear differential equations (for instance, which functions have elementary antiderivatives).