Suppose we have a cycle graph on $8$ vertices.
To find the number of ways to choose $3$ vertices up to rotation.
Note that we can choose $3$ vertices from $8$ vertices in $C(8,3) = 56$ ways.
But many of these are rotation similar.
How can I find the number of unique ones?
Please give me some hints.
As @RobPratt hinted at in his comment, this is essentially what's referred to as counting necklaces; in particular you have a 2-color necklace with 3 beads of one color and 5 of the other. You can read about this topic here, but determining the number of these is generally relatively complicated, even with just two colors.
Here, however, the count is considerably simplified because the number of vertices being chosen and the total number of vertices are relatively prime.
My hint would be to consider one particular choice of vertices, and check how many others are rotationally equivalent; e.g. (1,3,7) = (2,4,8) = (3,5,1) = .... If you do this a couple of times it shouldn't be difficult to realize that every combination has exactly the same number of rotationally equivvalent combinations, and therefore all you need to do is divide your count C(8,3) by that number.