we say x,y is equivalent to language L, if for any $z \in L$ we have:
$xz \in L \Longleftrightarrow yz \in L$.
for $ L= (ab \cup aab)^* $, what is the equivalence class for L?
my professor calculate the [b], [aa], [a], $[\epsilon]$.
I think this is false. anyone could help me? some detail for how he calculate it?
This equivalence is called the Nerode equivalence of $L$. The set of equivalence classes is in bijection with the set of states of the minimal DFA of $L$. To be more precise, for each word $u$, the equivalence class of $u$ is $[u] = \{ v \in A^* \mid uv \in L\}$. Now the minimal DFA of $L$ is $\mathcal{A} = (Q, \cdot, i, F)$ with $Q = \{ [u] \mid u \in A^* \}$, $i = [\varepsilon]$, $F = \{ [u] \mid u \in L \}$ and the transitions are given by $[u] \cdot a = [ua]$. One can use this fact to cheat a little bit by first computing the DFA and then finding the equivalence classes.
Here $L = (ab \cup aab)^*$ and its minimal DFA has 4 states, $\{1, 2, 3, 0\}$. The transitions are $$ 1 \xrightarrow{a} 2 \xrightarrow{a} 3 \xrightarrow{a} 0 \xrightarrow{a} 0 \text{ and } 1 \xrightarrow{b} 0,\quad 2 \xrightarrow{b} 1,\quad 3 \xrightarrow{b} 1,\quad 0 \xrightarrow{b} 0 $$ The initial state and unique final state is $1$. It now suffices to exhibit words sending $1$ to each of the states, for instance $1 \cdot \varepsilon = 1$, $1 \cdot a = 2$, $1 \cdot aa = 3$ and $1 \cdot b = 0$. Thus your professor is right, there are four different equivalence classes $[\varepsilon]$, $[a]$, $[aa]$ and $[b]$.
Computing these equivalence classes is another thing. One way would be to use the fact that $[u] = \{ v \in A^* \mid uv \in L\}$. Thus $[\varepsilon] = \{v \in A^* \mid 1 \cdot v = 1\}$, $[a] = \{v \in A^* \mid 2 \cdot v = 1\}$, $[aa] = \{v \in A^* \mid 3 \cdot v = 1\}$ and $[b] = \{v \in A^* \mid 3 \cdot v = 1\}$. There are other direct ways that I suppose your professor taught you.