I know that the following formula will calculate the number of combinations.
(32 * 31 * 30 * 29 * 28) / (5!)
However, I want to remove all the duplications where the cards' suit and rank distribution is the same.
For example, J♣ 9♣ Q♥ K♥ A♥ would be equal to J♥ 9♥ Q♦ K♦ A♦ but would not be equal to J♦ 9♦ Q♥ K♥ A♣
Is it possible to calculate this?
If all the cards are of the same suit, then the hand is counted $4$ times: once for each suit. If there are exactly $2$ suits then the hand is counted $12$ times, as there are $4$ ways to choose the suit with more cards and $3$ ways to choose the other suit.
If there are more than $2$ suits, the situation is a bit more complicated. For example, suppose there are $3$ cards of one suit, and $1$ of each of $2$ others. Then there are $4$ ways to choose the suit with $3$ cards and $3$ ways to choose the suits with no cards. Now if the other two cards are of different ranks, we have $2$ ways to choose the suit of the higher-ranking card, so that there $24$ ways in all, but if the two cards are of the same rank, then both choices are the same, so there are only $12$ ways.
The same kind of analysis for the $2-2-1-0$ distribution shows that if the two $2-card$ suits have the same rank, there are $12$ distributions, but if not, there are $24$.
You need to extend this analysis to the case of $4$ different suits. Then for each possible distribution, count how many hands there are, divide by the appropriate factor, and add up the answers.