Could someone give a simple explanation why it is special at $N=2$ that fundamental and anti-fundamental representation of $SU(2)$ is the same?
Then how is the case for Fundamental v.s. Anti-Fundamental Representation of $SU(N)$ for generic $N$? How are Fundamental v.s. Anti-Fundamental Representation of $SU(N)$ differed from each other?
p.s. You may like to explain in terms of Young tableaux, but it will be nice to explain without going to something more abstract that it has to be.
Recall that for compact groups we can test whether two representations are isomorphic by testing whether their characters are equal, and that taking duals takes the complex conjugate of the character. The character of the fundamental representation is just the trace $\text{tr} : SU(n) \to \mathbb{C}$ and the character of the antifundamental representation is the complex conjugate of this, so the question is why the trace of an element of $SU(n)$ is always real iff $n = 2$.
This is because the condition that an element of $SU(n)$ has determinant $1$, in addition to the condition that it's unitary, forces its eigenvalues to be unit complex numbers with product $1$. When $n = 2$ there are two eigenvalues $z, w$ which must satisfy both $|z| = |w| = 1$ and $zw = 1$, which forces $w = \bar{z}$, and hence the trace $z + w = z + \bar{z}$ is real.
When $n \ge 3$ these conditions no longer force the trace to be real, and it's an interesting question to characterize exactly what values the trace takes in general.