I am trying to use the inclusion-exclusion principle here but having a problem calculating the cases.
Assume $D_1$ - the possible ways to choose a number not equal to $5$ members for $A$
and $D_2$ - the possible ways to divide the elements where $|B| \leq |A|$
and $U$ - all possible divisions of $n$ elements to $A,B,C$
And I want to calculate $\bar{D_1} \cap \bar{D_2}=|U| - |D_1|-|D_2|+|D_1 \cap D_2|$
Now, $|U|$ is the number of functions from a set of the cardinality of $n$ to a set of the cardinality of $3$ which given $|U|=3^n$
But the other calculations seem too confusing. Should we solve it this way? how to calculate the rest of the expression if so? or how to solve it in a better way?
You can use a greatly simplified inclusion-exclusion idea.
Once you have chosen the 5 elements for set A you must split the remaining $n-5$ elements into two sets. When you do this you either have or do not have two sets of equal size.
If $n-5$ is odd
The sets have to be unequal in size and so half the possibilities have B larger than C. The number of possibilities is $2^{n-6}\binom{n}{5}$.
If $n-5=2k$ is even
We have to exclude the cases when the sets are equal. The number of possibilities is $\frac{1}{2}(2^{n-5}-\binom{n-5}{k})\binom{n}{5}$.