The answer according to edx course HarvardX: FC1x Fat Chance: Probability from the Ground Up
is $$\frac{6*5*5* \left(^5_2\right)}{6^5} =\frac{1500}{6^5}$$
The remaining two dice can be same say this is valid favourable outcome 4,4,4,5,5.
But according to my reasoning the probability is much more than that,
My reasoning:
6 options for 3 of a kind
5 options for 4th dice
4 options for 5th dice(let us consider only the cases where 4th and 5th dice are different, for sake of showing that even with excluding certain favourable outcomes, namely in which 4th and 5th dice are same, my probability is higher than the course answer.)
therefore total number of ways
$$ \frac{ 6*5*4*5!}{3!} = 2400$$
5! ways of arranging 5 items, divided by 3!, since 3 are of a kind.
$$ p=\frac{2400}{6^5} $$
Which is greater than the probability calculated by the edx course, and I have not even considered the case when we allow 4th and 5th dice to have same number.
what is wrong with my reasoning?
As pointed out in a comment, the main problem with your reasoning is that you're overcounting the possibilities for the two leftover dice: for instance, you're counting "three $1$s and a $2$ and a $3$" and "three $1$s and a $3$ and a $2$" among the sets of dice that can each be arranged in $\frac{5!}{3!}$ ways... hence double-counting arranged cases like $[1,3,2,1,1]$. For the cases where those dice are distinct (only), you need to divide by $2$. So the probability of three-of-a-kind with the two leftover dice distinct is half what you stated: $\frac{6\cdot 5\cdot 4 \cdot 5!}{2\cdot 3!}=1200$ cases out of $6^5$. Three-of-a-kind plus a pair leads to $\frac{6\cdot 5\cdot 5!}{2!\cdot 3!}=300$ additional cases out of $6^5$, for a total probability of $\frac{1500}{6^5}$.
It is simpler to solve it without distinguishing between the two possibilities for the leftovers: there are ${{5}\choose{3}}=10$ ways to select the dice participating in the three-of-a-kind, $6$ possibilities for their value, and $5^2$ possibilities for the values of the leftovers, for a net count of $10\cdot6\cdot 5^2=1500$.