I am trying to solve this problem of counting the number of arrangements of 8 people around a square table, as shown in the figure below, To solve this problem you can consider arrangements obtained from rotation to be similar -:
The first part of the question asks how many possible arrangements of 8 people are there on this square table, my reasoning for coming up with an answer is as follows, each of the circular arrangement of 8 people around the square table corresponds to 4 linear arrangements so by this reasoning the answer I came up with $$ \frac{8!}{4} = 10080 $$ square arrangements.
The second part of the question asks me in how many square arrangements do A and B don't sit together, here is how I approached the problem, I first counted the number of linear arrangements in which A and B sit together $ 7! \cdot 2! $ and using this I counted the number of square arrangements in which A and B sit next to each other as $ \dfrac{7! \cdot 2!}{4 \cdot 2!} = 1260 $ square arrangements in which A and B sit next to each other and then I subtract this number from the total number of square arrangements $ 10080 - 1260 = 8820 $ arrangements in which A and B don't sit next to each other, I am not sure if my answer is correct but I think it should be, it would be great if someone could confirm this.
The exclusion of arrangements that can be obtained from rotation comes to the same as the extra condition that $A$ is seated e.g. at the upper side. This because in any case there is exactly one rotation that brings him there. Then there are $2$ possibilities for $A$. The first part then gives $2\times7!=10080$ possibilities, confirming your own answer. The second part gives $2\times6\times6!=8640$ possibilities (if I understand well that $A$ and $B$ are not sitting next to eachother here). The factor $6$ corresponds with the possibilities for $B$.