I am working on a problem that says:
Classify and discuss the isometry $F: \mathbb{E}^3 \to \mathbb{E}^3: (p_1,p_2,p_3)\mapsto (p_2-2,p_3+1,8-p_1)$.
I really don't know where to begin on problems like this. Can someone give me some hints to get me going? I think that I can write $F$ as $$\begin{pmatrix}p_1 \\ p_2 \\ p_3 \end{pmatrix} \mapsto \begin{pmatrix}0 & 1 & 0 \\ 0 & 0 & 1 \\ -1 & 0 & 0 \end{pmatrix} \begin{pmatrix}p_1 \\ p_2 \\ p_3 \end{pmatrix} +\begin{pmatrix}-2 \\ 1 \\ 8 \end{pmatrix} $$
At this point I don't really know what to do. How would I go on to find out what kind of isometry it is and any additional information that can be given about the isometry?
I will expand on the hints I gave in the comments, and use them to work through the solution. If you want to stop reading at any point and try and finish off the question yourself, I recommend doing this. As a warning, writing $F$ as an affine transformation in the way you did in the question is a bit of a red Herring.
Hint 1: Find the fixed points of $F$.
Suppose $F(a,b,c)=(b-2,c+1,8-a)=(a,b,c)$, then $$a=b-2=(c+1)-2=((8-a)+1)-2=7-a$$ which implies $a=7/2$, and consequently $c=8-a=9/2$ and $b=c+1=11/2$, so $F$ has a unique fixed point, $(7/2,11/2,9/2)$.
Hint 2: What types of isometries of $\mathbb{E}^3$ have a unique fixed point? Can you relate these isometries to isometries of $\mathbb{S}^2$?
For simplicity, let's conjugate $F$ by the translation $T$ which sends $(0,0,0)\mapsto (7/2,11/2,9/2)$ to get a new isometry $F'=T^{-1}\circ F\circ T$ which fixes the origin. You can check $$F'(p_1,p_2,p_3)=(p_2,p_3,-p_1).$$ Since $F'$ is an isometry which fixes the origin, it is actually a linear isometry of $\mathbb{E}^3=\mathbb{R}^3$ thought of as a vector space. Because it doesn't change distances, the set of points a distance 1 from the origin is invariant, or put another way, it induces an isometry of the unit sphere $\mathbb{S}^2$.
There is a simple description of the isometries of the sphere, they are either 1) the identity, 2) a reflection in a great circle, 3) a rotation along a great circle, or 4) a glide reflection (rotation along a great circle followed by a reflection in that same circle.
Hint 3: Which type of isometry could $F'$ be, based on the description of fixed points we found?
Thought of as an isometry of the sphere, $F'$ has no fixed points, and hence must be a glide reflection, since the other types all do have fixed points: 1) all points of $\mathbb{S}^2$, 2) all points on the great circle, and 3) the poles of $\mathbb{S}^2$ relative to the great circle.
Hint 4: What is the invariant plane of $F'$?
Since $F'=S\circ R$ where $R$ is a rotation, and $S$ is a reflection we can ask what is the plane of reflection of $S$, which is also the invariant plane under the rotation $R$. Notice that $$F'(1,-1,1)=-(1,-1,1),$$ so the invariant plane is orthogonal to the vector $(1,-1,1)$.
Hint 5: What is the angle of rotation in this invariant plane by $R$?
Extend $(1/\sqrt{3})(1,-1,1)$ to an orthonormal basis of $\mathbb{R}^3$, eg $$\left\{e_1=\frac{1}{\sqrt{3}}(1,-1,1),e_2=\frac{1}{\sqrt{2}}(1,1,0),e_3=\frac{1}{\sqrt{6}}(-1,1,2)\right\}.$$ Then, since the reflection $S$ fixes $e_2$ and $e_3$ by definition, $$R(e_2)=F'(e_2)=\frac{1}{\sqrt{2}}(1,0,-1)=\frac{1}{2}e_2-\frac{\sqrt{3}}{2}e_3,\;\textrm{and}$$ $$R(e_3)=F'(e_3)=\frac{1}{\sqrt{6}}(1,2,1)=\frac{\sqrt{3}}{2}e_2+\frac{1}{2}e_3.$$
So as a rotation in the $(e_2,e_3)$ plane, $R$ is given by the matrix $$\begin{pmatrix}\frac{1}{2} & -\frac{\sqrt{3}}{2}\\\frac{\sqrt{3}}{2} & \frac{1}{2}\end{pmatrix}$$ and hence is a rotation by $\pi/3$.
Hint 6: Put it all together.
$F$ is the isometry which is a glide reflection of a sphere, centred on the point $(7/2,11/2,9/2)$, which rotates the affine plane through this point and orthogonal to $(1,-1,1)$ through an angle of $\pi/3$.
I have no doubt that there are more "direct" ways to compute this, but I hope working through the problem in a detailed geometric way gives you an intuitive understanding of what is going on.