Let $S$ be the plane, that is, $S=\{(x,y)|x,y∈R\}$ and consider $f,h\in A(S)$ defined by $f(x,y)=(−x,y)$. Let $n>2$ and let $h$ be the rotation of the plane about the origin through an angle of $\frac{2\pi}{n}$ in the counterclockwise direction. Define $G=\{f^kh^j|k=0,1;j=0,1,2,\cdots,n-1\}$ and let group operation in $G$ be the usual product of mappings. This group is called the dihedral group of order $2n$. One can verify $fh=h^{-1}f$, find a formula that expresses $(f^ih^j)*(f^sh^t)=f^ah^b$
All I know about $f$ and $g$ is,
$$f^2=h^n=\text{identity mapping}$$
And if I able to verify $fh=h^{-1}f$ then $fh^i=h^{-i}f$. And using
$f^i =$
\begin{cases}
f, & \text{if $i$ is odd} \\
e, & \text{if $i$ is even}
\end{cases}
we can easily list out the cases of $(f^ih^j)*(f^sh^t)$. But what bother me is the $fh=h^{-1}f$.
Rotation of the plane haven't any effect on $(x,y)$ in the plane$(?)$ Before I interpret $h^{-1}f$, I have a doubt on $h^{-1}$. Does it mean the rotaion of the plane about the origin through the same angle in the clockwise direction, because if it is then $fh$ seems not equal to $h^{-1}f$. Then how to verify that?
It is perhaps easier to visualize why $fhf=h^{-1}$ is true, which is of course exactly the same statement.
The first thing you need to believe is that reflection times rotation times reflection is a rotation. Once you believe that, then you can just check one point, which you can do with your finger on the table: the point $(1,0)$ gets mapped to $(-1,0)$, which then gets rotated clockwise (say) up into the top-left quadrant. Then reflection puts it in the top-right quadrant. But that's the opposite rotation---anti-clockwise---to the one you did.
But this applies to all points in the plane, by change of co-ordinate axes. If a symmetry acts like a rotation on all points, it is a rotation.
Another way to visualize this is to see reflection as not a reflection of the square, but moving behind the square. Draw the square on a pane of glass, like in A Beautiful Mind. Reflection is walking around the glass to the other side. Then $fhf$ is walk around, rotate the square (in the opposite direction as you are facing the other way!) and then walk back to the front.
Edit: here are some matrices: $$\begin{pmatrix}-1&0 \\ 0&1\end{pmatrix}\begin{pmatrix}\cos\theta&-\sin\theta\\\sin\theta&\cos\theta\end{pmatrix} \begin{pmatrix}-1&0 \\ 0&1\end{pmatrix}=\begin{pmatrix}\cos\theta&\sin\theta\\-\sin\theta&\cos\theta\end{pmatrix}.$$