I am starting to study infinite-dimensional representations of Lie groups and I am wondering about the following:
Let $G$ be a connected Lie group with Lie algebra $\mathfrak g$ and with a representation $\Phi: G \to \text{GL}(V)$, with $V$ finite-dimensional. Then, $\Phi$ induces a unique representation $\varphi: \mathfrak g \to \text{End}(V)$ of the Lie algebra and a subspace of $V$ is $G$-invariant if and only if it is $\mathfrak g$-invariant.
I am struggling to see why and how exactly is this statement wrong for infinite-dimensional representations? I do not see where the proof does not carry over. Also, what are easy illustrative (counter-)examples and what does it mean for a Lie algebra representation to be integrable?
Edit: On a second thought, could it be that in the infinite-dimensional case, the function $g \mapsto \Phi(g)v$ will not be differentiable for all $v \in V$ and thus we can not that easily differentiate our Lie group representation to a Lie algebra representation? Then my additional question is: where does the proof that a continuous Lie group homomorphism is automatically smooth, does not carry over? The group $\text{GL}(V)$ then no longer is finite-dimensional which makes problems, I guess.
The continuity/smoothness assumptions to require from infinite dimensional representations are a rather subtle issue. You can see that smoothness fails for very simple and natural examples. For example, take a compact Lie group $G$ and the Banach space $V:=C(G,\mathbb C)$ of complex valued continuous functions on $G$. The left regular representation of $G$ on $V$ defined by $(g\cdot f)(h):=f(g^{-1} h)$ for $f\in V$ and $g,h\in G$, is about the most natural representation that you can think of.
But for this representation, not even the map $g\mapsto g\cdot f$ (for fixed $f\in V$) is smooth. Indeed, if it were, you can compose it with the bounded linear (and hence smooth) evaluation map in the point $e\in G$. This gives $g\mapsto (g\cdot f)(e)=f(g^{-1})$, which is not smooth unless $f$ is smooth.
This illustrates the general phenomenon that $g\mapsto g\cdot v$ is smooth (as a map $G\to V$) for the elements of a dense linear subspace of $V$. This is called the space of smooth vectors, and the infinitesimal action of the Lie algebra is only defined on this subspace. (In practice, one uses an even smaller dense subspace, but that's a different issue.)
The correspondence between invariant subspaces for $G$ and $\mathfrak g$ in any case works only for closed subspaces, since derivatives of curves lying in a subspace have only vlaues in the closure of this subspace. You also have to pass to the closure for passing in the opposite direction. (So things are simple in the finite dimensional case only since there all subspaces are closed ...)