Given is the following Polyhedron $P$ in the $\mathbb{R}^3$:
I want to define the set $S$ of the rotational symmetries of the Polyhedron in $\mathbb{R}^3$. Hence an element $s \in S$ is a function $s: P \rightarrow P$.
One is obviously the identity function and there are three more. Two like you can find them in equilateral triangles, the rotations $120^\circ$ and $240^\circ$ and the last one $180^\circ$ with which the front becomes the back.
But as you can see my problem is that I am not able to "describe" them adequate enough so it's really clear what I mean. Could you help me with this?
I'm not sure exactly what kind of description you're looking for.
One possible form of description for such a kind of symmetry would be using a Coxeter group. Start not with the rotations but with the reflections. You could say that $a$ is the plane through one symmetry axis of each triangle, $b$ likewise so that $a$ and $b$ enclose an angle of $60°$, and $c$ is the plane through the center parallel to both triangles. Then you'd say that $a$ and $b$ together form a $120°$ rotation, so if you repeat that three times you have identity. Formally: $(ab)^3=\text{Id}$. Since $c$ is orthogonal to $a$ and $b$, any two of them form a $180°$ rotation, so you have $(ac)^2=(bc)^2=\text{Id}$. Taking it all together you have the Coxeter group
$$\langle a,b,c\mid a^2=b^2=c^2=(ab)^3=(ac)^2=(bc)^2=\text{Id}\rangle$$
You could draw a Coxeter graph for this, which is just three vertices, two of them connected by an unlabeled line. You could decompose the group into $A_2\times A_1$, i.e. the symmetry of the triangle combined with reflection in a single orthogonal plane. Since $\lvert A_2\rvert=6$ and $\lvert A_1\rvert=2$ you'd know that the full symmetry group has $6\cdot2=12$ elements. You could then say that the group of rotational symmetries is the subgroup of all combinations of an even number of reflections. So half of the elements would be rotations, and you have $6$ in total: the identity, the two rotations which rotate each endcap around itself, and three rotations which exchange the endcaps and are each perpendicular to one of the rectangles.
You could also go to the level of coordinates, and describe all these $6$ rotations as a linear transformation in some vector space. But with a regular triangle in there, you'll usually end up with some nasty square roots in your formulas.