This problem is from the book, "Introduction to Probability" by Hoel, Port and Stone. It is problem 24 on page 48.
Problem:
Find the probability that a poker hand of $5$ cards will contain no cards smaller than $7$ given that it contains at least $1$ card over $10$,
where aces are treated as high cards.
Answer:
Let $p$ be the probability we seek. Let $p_1$ be the probability that a poker hand of $5$ cards will contain no cards smaller than $7$. Let $p_2$ be the probability that a poker hand of $5$ cards will contain at least $1$ card over $10$, where aces are treated as high cards. Let $p_3$ be the probability that a poker hand of $5$ cards will contain no card over $10$, where aces are treated as high cards.
\begin{align*}
p &= \frac{p_1}{p_2} \\
p_1 &= \frac{ { {32} \choose {5} } } { {{52} \choose {5}} } \\
p_2 &= 1 - p_3 \\
p_3 &= \frac{ { {36} \choose {5} } } { {{52} \choose {5}} } \\
p_2 &= 1 - \frac{ { {36} \choose {5} } } { {{52} \choose {5}} } \\
p_2 &= \frac{ {52 \choose 5 } - {36 \choose 5 } } { {52 \choose 5 } } \\
p &= \frac{\frac{ { {32} \choose {5} } } { {{52} \choose {5}} } }{\frac{ {52 \choose 5 } - {36 \choose 5 } } { {52 \choose 5 } } } \\
p &= \frac{ {32 \choose 5 } }{ {52 \choose 5 } - {36 \choose 5 } }
\end{align*}
However, the books answer is:
$$ \frac{ {32 \choose 5 } - {16 \choose 5 } } { {52 \choose 5} - {36 \choose 5 } } $$
The sample space is the set of all five-card poker hands with at least one card with rank greater than 10. As you found, the number of such hands is $$\binom{52}{5} - \binom{36}{5}$$ since we must subtract those hands in which no card has rank greater than 10 from the total number of five-card poker hands.
The favorable cases are those five-card poker hands containing no card with rank smaller than 7 that contain at least one card with rank greater than 10. There are $32$ cards with rank at least 7 in the deck, of which just $16$ cards have rank not greater than 10. Since the favorable cases must include at least one card with rank greater than 10, we must subtract the number of five-card poker hands in which all the cards have rank at least 7 but at most 10 from the number of five-card poker hands in which each card has at least 7. Thus, the number of favorable cases is $$\binom{32}{5} - \binom{16}{5}$$
Hence, the probability that a five-card poker containing at least one card with rank greater than 10 has no cards with rank less than 7 is $$\frac{\dbinom{32}{5} - \dbinom{16}{5}}{\dbinom{52}{5} - \dbinom{36}{5}}$$