Five mice are chosen (without replacement) from a litter, three of which are tagged $A$, $B$ and $C$. The probability that all three tagged mice are chosen is twice the probability that $A$ is the only tagged mouse chosen. How many mice are there in the litter?
So far I have that the total number of ways to pick five mice from a litter of $n$ mice is $$ ^nC_5 = \left(\frac{n}{5}\right) = \frac{n!}{\left( n - 5 \right)!}. $$
The ways to pick so that $A$ is the only tagged one are:
AUUUU UAUUU UUAUU UUUAU UUUUA
where $U$ is an untagged mouse, which totals 5 ways. So the probability of this is $$ \frac{5n!}{\left( n - 5 \right)!} $$
But how do I determine the number of ways to pick all three tagged mice when I draw five?
The experiment is to pick a combination of 5 mice from a litter of $n$. There are ${n\choose 5}$ ways to do this, as you've noted.
(1) How many combinations contain $A$ but not $B$ or $C$? This requirement forces one of your mice to be $A$. The other four mice in your combination have to be chosen from the remaining $n-3$ available (remember, you've ruled out $A$, $B$, $C$ as candidates for the other four mice). This gives ${n-3\choose 4}$ combinations.
(2) How many combinations contain all three of $A$, $B$, $C$? We have accounted for three of the mice, so the other two mice must be chosen from $n-3$ remaining. This gives ${n-3\choose 2}$ possible combinations.
As stated, the probability of event (2) is twice the prob of event (1). Clearing the denominator ${n\choose 5}$, this means $${n-3\choose 2}=2{n-3\choose 4}.$$
You can solve this by trial and error. Or simplify: $$ {(n-3)!\over 2!(n-5)!}=2{(n-3)!\over4!(n-7)!}.$$ Use $(n-5)!=(n-5)(n-6)(n-7)!$ to obtain the quadratic equation $$6=(n-5)(n-6),$$ whose only legal solution is $n=8$.