For example, I draw samples until I have I have the random variables $X_1,...,X_n$. I'm told that if I select any of these random variables, say $X_4$, its expected value will equal the population mean, i.e. $E[X_4]=\mu$. This is what I do not understand.
I understand that if my sample included every single possible observation from the population, i.e. $n=N$, then $E[X_4]=\mu$ makes sense to me. But if $n<N$, why do we still expect it to be $\mu$?
If my population is $\{1,2,3,4,5\}$, each value equally likely. Clearly, $\mu=3$. If I take a sample, say, $X_4 = \{1,2,3\}$, then $E[X_4]=(1/3)+(2/3)+(3/3)=2\ne \mu$. So I seem to have calculated a sample mean if anything. So something is wrong with this calculation. Therefore, I do not understand why $E[X_4]=\mu$.
A sample takes the form $\{X_1,\dots,X_n\}$ and the $X_i$ are random variables.
Then it is true that: $$\mathsf EX_i=\mu\text{ for every }i$$ but stating that is something different as stating that: $$\frac1n(X_1+\cdots+X_n)=\mu$$ which is in general not true.
By writing $X_4=\{1,2,3\}$ as a possible sample you suggest that $X_4$ is a set with $3$ random elements (that all have taken some value). In that context $\mathsf EX_4$ has no meaning at all.