A topological dynamical system $(X,f)$ is called minimal if whenever there is a closed $Y \subseteq X $ s.t. $f(Y)\subseteq Y$ then either $Y = \emptyset$ or $Y=X$.
Even if an invariant $Y$ isn't closed, $(Y, f\restriction_Y)$ is still a dynamical system. Why do we ask for $Y$ to be closed?
The short answer? Because that definition is the most useful.
The long answer? This definition often goes hand in hand with the property of $X$ (and hence any closed subsystem $Y$) being complete, or more often compact. Compact dynamical systems are just much easier to work with and often of the most importance. This is mostly because convergence properties are key in being able to analyse dynamical systems. We want to know, for instance, recurrence properties and attractors of our system - both of which require a well-defined notion of convergence in the space, which requires the space to be compact (well, complete really).
If we define a metric space $X$ and a homeomorphism $f$ to be a dynamical system only if $X$ is compact, then the above definition of minimality really corresponds to an order of subsystems - here we take inclusion of one subsystem into another to be the ordering, and our elements are the closed non-empty $f$-invariant subspaces of $X$. In this case, minimality corresponds to the existence of a unique such non-empty closed $f$-invariant subspace, $X$ itself.
On a more philosophical level, dynamical systems are normally studied, not in terms of the evolution of single points (orbits), but rather properties of the global system, or open subset of that system. Whenever one wants to consider open subsets, then the evolution of that neighbourhood under a continuous map will determine what also happens on its completion and so we are studying the completion, even if we were not explicitly doing so. Well, if we're doing that anyway, and complete spaces have such nice properties, we might as well suppose a priori that everything we are working with is already complete, and if not complete it. The completion of any non-empty $f$-invariant subsystem of a minimal dynamical system is equal to the whole dynamical system, and hence closed.