A bag contains 28 balls, 11 red balls, 10 black balls, 7 yellow balls. In how many ways can 3 balls be selected from the bag without replacement, if no more than two red balls can be included in the draw?
I started by laying out all of the combinations:
$$ \begin{array}{lll|lll|lll} Y & Y & Y & Y & B & Y & Y & R & Y \\ B & Y & Y & B & B & Y & B & R & Y \\ R & Y & Y & R & B & Y & R & R & Y \\ Y & Y & B & Y & B & B & Y & R & B \\ B & Y & B & B & B & B & B & R & B \\ R & Y & B & R & B & B & R & R & B \\ Y & Y & R & Y & B & R & Y & R & R \\ B & Y & R & B & B & R & B & R & R \\ R & Y & R & R & B & R & R & R & R \end{array} \\ \frac{7}{27} $$
then I counted the number of times more than two reds are selected which is 1. so the answer should be 26. Am I wrong here?
See that when you listed out all the possible combinations, you have counted some of them multiple times ($RBB$ and $BBR$ are the same combination, for example)
In the context of this problem, we have three colors to choose from: red, black, and yellow. Since there are at least three of each type in the bag, we can treat this as picking $3$ balls from three colors where repetition is allowed, but order is not important. This is also referred to as the Stars and Bars technique.
The total number of ways this can be done is $10$. One of these ways includes the combination $RRR$, which does not meet the requirement of at most $2$ reds. Therefore the total number of ways this can happen is $9$.
This answer can also be found using your method if you remove duplicate combinations.
If the order of how you choose the balls matters, then what you have found is correct.