Considering the action of $\mathbb{Z} \diagup 3$ on $\mathbb{R}^3$ given by the cyclic 3-fold rotation $x \mapsto y \mapsto z \mapsto x$, I am trying to find the $\mathbb{R}$-vector subspaces invariant under this action.
By working through some example vectors and what the action does to them, namely the standard basis vectors of $\mathbb{R}^3$, I'm coming to the conclusion that the only such invariant is the zero subspace.
This seems unlikely, and I'm really struggling to see a method as to how to tackle this for bigger subspaces. My intuition is that the action represents a rotation about the origin through a plane\about one of the coordinate axes... but I'm very unsure on this.
Any help would be appreciated!
There is an invariant subspace that you have not considered : the one consisting of vectors with all their coordinates equal, i.e. the one generated by $(1,1,1)$. Clearly all the vectors in this subspace are fixed by the action, so the subspace is invariant. Moreover, the action preserves the scalar product on $\Bbb R^3$; hence the orthogonal subspace is invariant as well. This orthogonal subspace consists of all vectors such that $x+y+z=0$, so you can see directly that it has to be invariant.
By the way your intuition is right : the action correspond indeed to rotations by an angle of $\frac{2\pi}{3}$, but the axis of the rotation is the line generated by $(1,1,1)$. Indeed, you can take a vector in the orthogonal subspace $(a,b,-a-b)$, and check that it makes an angle $\alpha=\frac{2\pi}{3}$ with its image under the action, $(-a-b,a,b)$: \begin{align}\cos \alpha & = \frac{(a,b,-a-b)\cdot (-a-b,a,b)}{\|(a,b,-a-b)\|\cdot\|(-a-b,a,b)\|}\\ & =\frac{-a^2-ab+ba-ba-b^2}{a^2+b^2+(a+b)^2}\\ & =\frac{-a^2-ab-b^2}{2a^2+2ab+2b^2}=\frac{-1}{2}\\ &=\cos \left(\frac{2\pi}{3}\right) .\end{align}
This gives us two non-trivial invariant subspaces. We can show that there cannot be other invariant subspaces. Indeed, if $U$ is an invariant subspace, then :