We have a basket and there are some balls in it (It is not stated how many balls, I suppose this to be equal to $N$ but probably $N\rightarrow \infty$). The balls have $m$ colors and we pick $n$ balls from the basket. We know that the probability of picking each color is equal to others. (i.e. number of balls from each color is equal to $N/m$). We want to find expected value of number of distinct colors we have between those $n$ balls we picked from basket. (i.e. if we pick six balls RRGBYY, We have 4 distinct colors).
My approach to this problem:
Let $X_i$ be the random variable that we picked a ball from color $i$. It is 1 if we pick a ball from color $i$ and zero if we don't pick any. Then what the problem wants is actually Expected Value of $Y$ where: $$Y = X_1 + X_2 + ... + X_m$$
and because the probability of each color is same to others and by linearity of expectation, we found that $E(Y) = m \times E(X_1)$
for finding $E(X1)$ we actually find the probability of picking a ball with color $1$ and for finding these we find the probability that we pick no balls of color $1$, we have: $$P(no~ ball ~with~ color ~1) = \frac{\binom{N -N/m}{n}} {\binom{N}{n}}$$
so $E(1) = 1- \frac{\binom{N -N/m}{n}} {\binom{N}{n}} $ and therefore $E(Y) = m \times(1- \frac{\binom{N -N/m}{n}} {\binom{N}{n}}) $
I think I saw a problem like this between in math.se but they all had $N$ as the total number of balls. My problem is that actually, the question didn't say anything about total number of balls. So it seem we must find $\lim_{N \rightarrow \infty} E(Y)$ but I don't know any idea how to find this limit.
Your approach is excellent.
For simplicity let us assume that $N=km$ where $k$ is an integer, so that $$\frac{\binom{N-N/m}{n}}{\binom{N}{n}}=\frac{\binom{km-m}{n}}{\binom{km}{n}}=\left(1-\frac{k}{km}\right)\left(1-\frac{k}{km-1}\right)\cdots\left(1-\frac{k}{km-n+1}\right)$$So we are dealing with $n$ factors that all converge to $1-\frac1m$ by increasing $k$.
This evidently leads to: $$\lim_{N\to\infty} EY_N=m\left(1-\left(1-\frac1m\right)^n\right)$$Also observe that this is the outcome you will get if we work just with $m$ ball and replace the drawn balls.