Let $G$ be a topological group and $\mu$ be a probability measure on $(G,\mathscr{B})$ (Borel sigma-algebra). Consider the subset
$$S=\{g\in G: \mu(Ag)=\mu(A), \forall A\in \mathscr B\}.$$
$S$ is clearly a subgroup of $G$. But is $S$ necessarily closed?
In the original problem I was studying $G$ is a linear group modulo a lattice. So please feel free to add some mild conditions on $X$ if applicable (I also want to know whether such $G$ without any conditions could fail).
Let $G$, $\mu$, $S$ be as in the question. I will prove the following special case.
Theorem 1: If $G$ is locally compact Hausdorff and $\mu$ is regular then $S$ is closed.
I don't have a reference but it is similar to Exercise 1.6 of Deitmar and Echterhoff, Principles of Harmonic Analysis. This exercise makes a stronger assumption that $S$ is dense and a weaker assumption that $\mu$ is Radon instead of regular and finite. Indeed the proof of Theorem 1 can be modified to work under just assuming $\mu$ is regular and finite on compact sets. But here we assume $\mu(G)=1$ which removes some of the difficulties.
I suppose the counterpoint is that since this isn't the exact result from the book and I don't have another source, then readers may find errors.
So now let's start towards the proof. Here's the idea: integration wrt $\mu$ determines a linear functional on compactly supported continuous functions that is right invariant with respect to $S$. We shall use a uniform continuity argument to deduce that it is right invariant wrt to the closure of $S$. Then Reisz representation will tell us that $\mu$ is right invariant wrt the closure of $S$. So $S$ contains its closure by definition.
Let $C^{0}_{c}(G)$ be the space of compactly supported complex-valued continuous functions on $G$. If $\varphi\in C^{0}_{c}(G)$ and $g\in G$ then let $\varphi_{g}$ denote the map $x\mapsto \varphi(xg^{-1})$ which is in $C^{0}_{c}(G)$. Then define a map $F_{\varphi}:G\to \mathbb{C}$ such that $F_{\varphi}(g)=\int \varphi_{g}d\mu $.
Lemma 2: If $\varphi$ is in $C^{0}_{c}(G)$ then $F_{\varphi}$ is continuous.
Proof: Fix $\varphi$. Fix $x\in G$ and $\epsilon>0$. We need to find an open nbhd $U$ of $x$ such that $|F_{\varphi}(x)-F_{\varphi}(y)|<\epsilon$ for any $y\in U$. Since $\varphi$ is compactly supported it is uniformly continuous and so we can select an open identity nbhd $V$ such that if $g^{-1}h\in V$ then $|\varphi(g)-\varphi(h)|<\epsilon.$ Let $U=Vx$ so $U$ is an open nbhd of $x$. I claim that $U$ is our desired nbhd. Fix $y\in U$. Observe that if $g\in G$ then $$ (gy^{-1})^{-1}(gx^{-1})=yx^{-1}\in V $$ So our choice of $V$ ensures $|\varphi_{x}(g)-\varphi_{y}(g)|<\epsilon$ for all $g\in G$. From this we conclude $$ |F_{\varphi}(x)-F_{\varphi}(y)|\leq \int|\varphi_{x}-\varphi_{y}|d\mu<\epsilon $$ and the lemma is proved.
Now I prove the Theorem 1. Let $T$ be the closure of $S$. Fix $\varphi\in C^{0}_{c}(G)$. By definition of $S$ and approximation by simple functions, we know that $F_{\varphi}(g)=F_{\varphi}(e)$ for any $g\in S$. But $F_{\varphi}$ is continuous by Lemma 2 so $F_{\varphi}(g)=F_{\varphi}(e)$ for any $g\in T$. Therefore if $L$ is the linear functional on $C^{0}_{c}(G)$ that sends $\varphi$ to $\int\varphi d\mu$, then we have shown that $L(\varphi_{g})=L(\varphi)$ for any $\varphi\in C^{0}_{c}(G)$ and $g\in T$. By Reisz representation we deduce that $\mu(Ag)=\mu(A)$ for any $A\in\mathscr{B}$ and $g\in T$. So $T\subseteq S$ by definition of $S$ and we have shown that $S$ is closed.