consider an open set $\tilde{U}\subset\mathbb{R}^n$ and a finite Lie-group $G$, which acts smoothly on $\tilde{U}$, i.e. we have a smooth map $G\times \tilde{U}\rightarrow\tilde{U}$. Suppose further, the action is effective.
Q: What can be said about the set of points with non trivial isotropy group?
I would guess that the this set should be finite or at least countable. But I do not know how to approach this problem. Do you know how to do it or do you have a good reference for this question?
This set is not necessarily finite or countable. For example, let $\mathbb{Z}/2\mathbb{Z}$ act on $\mathbb{R}^2$ by$$(x,y)\mapsto(-x,y).$$The set of points with non-trivial isotropy group is $\{x=0\}$.
There is, however, something else that can be said. Given a compact Lie group action on a manifold, a fundamental theorem says that the set of points of a certain orbit type is a submanifold. Hence, in your case, the set of points with trivial isotropy group is a submanifold, and the complement is the union of submanifolds (a finite number of them, since the group is finite).