In Bluman/Anco's text "Symmetry and Integration Methods for Differential Equations", on p. 46 we find the definition of invariant function F:
$\mathbf{F}(\mathbf{x^*})=\mathbf{F}(\mathbf{x})$, where F must be infinitely differentiable
On the same page this property is required to prove an important theorem (Theorem 2.3.4-1):
$\mathbf{F}(\mathbf{x})$ is an invariant under Lie group of transformations if and only if $X\mathbf{F}(\mathbf{x})=0$, where $X$ is the infinitesimal group operator.
It appears that F must be infinitely differentiable in all its arguments (in jet space), because to prove the theorem he uses the relation
$$\mathbf{F}(\mathbf{x^*})=e^{\varepsilon X}\mathbf{F}(\mathbf{x}) = \mathbf{F}(\mathbf{x})+\varepsilon X \mathbf{F}(\mathbf{x})+\frac{1}{2} \varepsilon^2 X^2 \mathbf{F}(\mathbf{x})+...$$
which involves infinite differentiation of $\mathbf{F}$.
But I was also told by a math professor, that "a function that is invariant need not be infinitely differentiable."
Am I missing something?
Consider the plane $\Bbb R^2$ with the group of rotations $SO(2)$ acting on it. The function $f(\mathbf x) = \|\mathbf x\|$ on the plane is invariant, but definitely not smooth. (This generalizes immediately to dimension $n>2$.)