In how many ways the $8$ people $A,B,C,D,E,F,G,H$ can be arranged around a square table assuming $A$ should not be seated in front of $B$.
Also
$\;\;\;\;\;\;\;\;\;\;\;\;\;\;$ 
The rightmost one is the same as the middle one, however they two are different from the leftmost one.
The number of arrangements that $8$ people can sit around such a table is $2\cdot7!$ On the other hand for each one of the sides one of the two cases happens:
The number of such arrangements is $2\cdot6!$, so the desired answer is $2\cdot7!- 2\cdot6!=12\cdot6!=8640$
But the answer is $5760$

Your answer is correct. Look at it this way:
Accounting for the four rotations of the table, we find:
$$\frac{8 \cdot 6 \cdot 6!}{4} = 8640$$