I am using $$\frac{\left(\frac{13!}{5!(13-5)!}\right)\left(\frac{13!}{4!(13-4)!}\right)\left(\frac{26!}{3!(13-3)!}\right)}{\left(\frac{52!}{13!(52-13)!}\right)}$$
but it seems I am doing something wrong here , the answer in the text book key is 0.02166
but my calculation gives 0.0037
This is a textbook example of a multivariate hypergeometric distribution.
Pick the five spades that occur in your hand. $\binom{13}{5}$ options.
Pick the four diamonds that occur in your hand. $\binom{13}{4}$ options.
Pick the four remaining non-spade non-diamond cards. $\binom{26}{4}$ options.
Divide by the total number of hands possible. $\binom{52}{13}$ options.
$$\frac{\binom{13}{5}\binom{13}{4}\binom{26}{4}}{\binom{52}{13}}\approxeq 0.21664206\dots$$