This is a question I had on an exam from awhile ago and was rather curious about.
"Let $m$ denote the Lebesgue measure on the unit square $I^2 = [0,1]^2$. Define two mappings $S,T: I^2 \to I^2$ as follows: $S(x,y) = (y,x)$ and $T(x,y) = (x,\bar{y})$ where $\bar{y} = y + \frac{1}{\sqrt{2}}$ if $y \leq 1 - \frac{1}{\sqrt{2}}$ and $\bar{y} = y + \frac{1}{\sqrt{2}} - 1$ otherwise. Now suppose that $A \subset I^2$ is a measurable subset such that $S(A) = A$ and $T(A)=A$. Show that either $m(A) = 0$ or $m(A)=1$."
My thoughts/attempt: $S$ is a reflection of $I^2$ along the diagonal and $T$ is a translation of the $y$ coordinate "up" by $\frac{1}{\sqrt{2}}$. But if it were to translate $y$ too far; i.e. leave $I^2$, it loops around so we can think of these isometries acting on a cylinder where we identify the top and bottom of the square.
On the other hand, $STS(x,y) = ST(y,x) = (x + \frac{1}{\sqrt{2}},y)$ (or the other expression, depending on what $x$ is). So $STS$ is a horizonal translation which also loops around so now, it seems we should think of $S,T$ as isometries on the torus.
Now, this set $A$ is invariant under both maps as is $m$ so it makes intuitive sense that it should be either measure-theoretic large or small. Since $\frac{1}{\sqrt{2}}$ is irrational, if we apply our horizontal or vertical translations to a point $p \in A$ and consider its orbit, I believe its orbit should be dense in $I^2$.
So suppose that $0 < m(A)$. It seems like on the one hand, we can "cover" all of $I^2$ by translations of $A$ since $A$ has positive measure; i.e. its orbit should have full measure $1$. On the other hand, $A$ is invariant under $S,T$ so it equals its orbit. I'm just not clear on how to prove this rigorously.
My hunch is to use something like Lebesgue density. The definition of the density of $A$ at a point $p$ is (when the limit exists)
$$D_A(p) = \lim_{r \to 0}\frac{m(A \cap B(r,p))}{m(B(r,p))}.$$
The Lebesgue Density Theorem says that if $E \subset \mathbb{R}^n$ is any Borel set, then for almost every point $p \in E$, $D_E(p) = 1$ and for almost every $p \notin E$, $D_E(p) = 0$. We can assume $E$ to be Borel as there is a theorem which basically allows us to write $E$ as some $G_\delta$ set minus a null set and $G_\delta$ sets are Borel.
So let $p \in A$ be such a point where $D_A(p) = 1$. Let $\mathcal{O}(p)$ be the orbit set of $p$ under $S,T$. It seems that for each $q \in \mathcal{O}(p)$, $D_A(q) =1$ because we may take a small ball around $p$ and intersect that with $A$. Now apply some finite combination of $S,T$ to this intersection set; it is moved about but will still end up in the set $A$ and since these are rigid transformations, the measure of this intersection set should be the same. Moreover, this should hold for every small ball so the density ought to still be 1 around $q \in \mathcal{O}(p)$.
But $\mathcal{O}(p)$ is dense in $I^2$, topologically. So for each point in a dense set in $I^2$, the Lebesgue density at that point is 1.
Question: Can we conclude that since
$$D_A(p) = \lim_{r \to 0}\frac{m(A \cap B(r,p))}{\pi r^2} = 1,$$
for sufficiently small $r$, for every $q \in \mathcal{O}(p)$, $m(A \cap B(r,q)) \approx \pi r^2$, for the same $r$? If so, can we say that the since the measure of $A$ locally is positive in a dense set, then $m(A) = 1$?
Problem: If $q \in \mathcal{O}(p)$ happens to be a corner or edge point in the square, then maybe the density should be $\frac{1}{2}$ or $\frac{1}{4}$.
You pretty much have all the necessary ideas. You are just missing some technical details.
The problem with "corner" or "edge points" can be solved by working directly on the torus, or by lifting your problem periodically to $\mathbb{R}^2$.
For the latter, define $\tilde{A}:= \{(x,y)\in \mathbb{R}^2 : (x - \lfloor x\rfloor, y - \lfloor y \rfloor) \in A\}$, let $\tilde{S}(x,y) = (y,x)$ and $\tilde{T}(x,y) = (x, y + 1/\sqrt{2})$.
Then $S(A) = A = T(A) \implies \tilde{S}(\tilde{A}) = \tilde{T}(\tilde{A}) = \tilde{A}$.
It turns out the argument is slightly easier if we look at the complement of $A$ instead. We will show that if $[0,1]^2 \setminus A =: A^c$ has positive measure, than $A$ has measure zero, which implies the claim you are looking at.
Suppose $A^c$ has positive measure, then there exists $(x_0,y_0)\in A^c \setminus \partial [0,1]^2$ (using that the boundary has zero measure) such that for every $\epsilon > 0$ there exists a radius $r\in (0,1)$ such that
$$ \frac{ | B((x_0,y_0), 3r) \cap \tilde{A} | }{|B((x_0, y_0),3r)|} < \epsilon $$
Using that $\tilde{S}, \tilde{T}, \tilde{S}\tilde{T}\tilde{S}$ are all isometries, and that $\tilde{A}$ is 1 periodic, this implies that for any $p,q,m,n\in \mathbb{Z}$, denote by $z$ the point $z_{p,q,m,n} = (x_0 + \frac{p}{\sqrt{2}} + m ,y_0 + \frac{q}{\sqrt{2}} + n)$
$$ \frac{ | B(z_{p,q,m,n}, 3r) \cap \tilde{A} | }{|B(z_{p,q,m,n},3r)|} < \epsilon $$
The density of the orbits implies you can find a finite subset $\mathcal{Z}\subset \mathbb{Z}^4$ such that $$ \cup_{(p,q,m,n)\in \mathcal{Z}} B(z_{p,q,m,n}, r) \supset [0,1]^2$$ and the corresponding $z_{p,q,m,n}$ are all in $[0,1]^2$.
By the Vitali covering lemma we can find a further subset $\mathcal{Z}'\subset \mathcal{Z}$ such that 1. If $(p,q,m,n), (p',q',m',n')$ are distinct points in $\mathcal{Z}'$ then $B(z_{p,q,m,n},r) \cap B(z_{p',q',m',n'}) = \emptyset$. 2. $$ \cup_{(p,q,m,n)\in \mathcal{Z}'} B(z_{p,q,m,n}, 3r) \supset [0,1]^2$$
So $$ \Big| A \Big| = \Big| A \cap \cup_{(p,q,m,n)\in \mathcal{Z}'} B(z_{p,q,m,n}, 3r) \Big| \leq \sum_{(p,q,m,n) \in \mathcal{Z}'} \Big| \tilde{A} \cap B(z_{p,q,m,n},3r) \Big| $$ using subadditivity (and enlarging $A$ to $\tilde{A}$). The right hand side is bounded by $$ \leq \epsilon \sum_{(p,q,m,n) \in \mathcal{Z}'} \Big| B(z_{p,q,m,n},3r) \Big| \leq 9 \epsilon \sum_{(p,q,m,n) \in \mathcal{Z}'} \Big| B(z_{p,q,m,n}, r) \Big| $$ Now using that the $B(z_{p,q,m,n},r)$ are pairwise disjoint, at that the centers are all in the unit square, and $r < 1$, we have that $$ \cup_{\mathcal{Z}'} B(z_{p,q,m,n},r) \subset [-1,2]^2 \implies \sum_{(p,q,m,n) \in \mathcal{Z}'} \Big| B(z_{p,q,m,n}, r) \Big| \leq 9 $$ So we conclude finally that $$ \Big| A \Big| \leq 81 \epsilon$$ Since $\epsilon$ is arbitrary, this implies that $A$ must have measure zero.