I read multiple approaches on how to solve this problem. However, I'm still not sure on how to do it as I am only acquainted with basic combinatorics. (clockwise and counterclockwise are considered the same)
My first approach was to calculate the total number of calculations and then get rid of the permutations that include two of equal color next to each other.
There are 11 Balls; Therefore the number of permutations should be: 11! = 39916800
Getting rid of the perms where two adjacent balls are equal in color is the part i can't figure out. (If this is even the right approach to this)
Hint:
Suppose we assume that all the same color balls are identical.
$B,B,B,B,B$--> Blue balls $Y,Y,Y$--> Yellow balls $O,O,O$-->Orange balls
$_B_B_B_B_B_$
There are $6$ blank spaces.
If the first space filled by $Y$ then last space should be filled by $O$. If the first space filled by $O$ then last space should be filled by $Y$.
First space can be filled in $2$ ways. (i.e) It is filled be either $Y$ or $O$.
So totally, first and last spaces can be filled in $2$ ways.
The remaining four spaces can be filled by {$Y$,$Y$,$O$,$O$}. Thus the remaining four spaces filled in $\frac{4!}{2!×2!}=6$ ways.
Thus overall the 6 blank spaces can be filled in $2×6=12$ ways.
Thus we have 12 linear permutation with one end start with $Y$ and ends with $O$ or one end start with $O$ and ends with $Y$. If we bend this line and merge the two ends we will get a cyclic arrangements which satisfying our requirements. Thus there are $12$ such an arrangement.
If CW and ACW are considered as same
Then required number $=12/2=6$.
Similarly you can check other possibilities. Like $B_ _ B_B_B_B_$ $B_ B_ _B_B_B_$ $B_ B_B_ _B_B_$ $B_ B_B_B_ _B_$ $B_ B_B_B_B_ _$ $_ _B_ B_B_B_B$ Etc.