Are iid samples and sampling with replacement analogous to each other?
iid - Independent and Identically Distributed RVs, means that each sample is independent of all others and all samples are drawn from the same distribution.
sampling with replacement - You sample a value from a distribution ( eg. A ball from an urn) and once you note it down, you replace it back. Does replacement ensure that the draws are independent and from identical distributions?
I have not seen this analogy spoken about in statistics books, maybe it's trivial but I wanted to make sure my intuition was correct.
Yes, in sampling with replacement with all outcomes equally likely, the samples are indeed independent and identically distributed.
Suppose you have a bin with $n$ balls, numbered $\{1, ..., n\}$, and you sample 3 times with replacement. So all outcomes have the form $(x_1, x_2, x_3)$, where $x_i$ represents the ball number of sample $i \in \{1, 2, 3\}$, and $x_1, x_2, x_3 \in \{1, ..., n\}$. So there are a total of $n^3$ possible outcomes. Suppose all $n^3$ outcomes are equally likely. Then each outcome has probability $1/n^3$. Since an event is just a set of outcomes, the probability of any event $B$ is equal to: $$ P[B] = \frac{|B|}{n^3} $$ where $|B|$ denotes the number of outcomes in the event $B$.
Define the random variable $X_i$ to be the ball number of sample $i$. For given integers $k_1, k_2, k_3 \in \{1, ..., n\}$, define the events: \begin{align} A_1 &= \{X_1 \leq k_1\}\\ A_2 &=\{X_2 \leq k_2 \}\\ A_3 &= \{X_3 \leq k_3\} \end{align} Then $A_1 \cap A_2 \cap A_3$ is the event that all three events happen. There are $k_1k_2k_3$ ways for this to occur, that is, $|A_1 \cap A_2 \cap A_3|=k_1k_2k_3$. So: $$P[A_1 \cap A_2 \cap A_3] = \frac{k_1k_2k_3}{n^3}=\frac{k_1}{n}\frac{k_2}{n}\frac{k_3}{n}$$ On the other hand, you can show that $|A_1|=k_1n^2$, $|A_2|=k_2n^2$, $|A_3|=k_3n^2$, and so
$$P[A_1]=\frac{k_1}{n}, P[A_2]=\frac{k_2}{n}, P[A_3]=\frac{k_3}{n} $$ Thus, $$P[A_1 \cap A_2 \cap A_3] \overset{(a)}{=} P[A_1]P[A_2]P[A_3]$$ Similarly, you can show \begin{align} P[A_1\cap A_2] &\overset{(b)}{=}P[A_1]P[A_2]\\ P[A_1\cap A_3] &\overset{(c)}{=}P[A_1]P[A_3]\\ P[A_2\cap A_3]&\overset{(d)}{=}P[A_2]P[A_3] \end{align} We deduced all this directly from the fact that all samples are equally likely, without using the concept of independence. However, this motivates a good definition for independence. Indeed, three events $A_1, A_2, A_3$ are defined to be mutually independent if equalities (a)-(d) hold.
In terms of the random variables $X_i$, the equality (a) implies: $$P[X_1\leq k_1, X_2\leq k_2, X_3 \leq k_3]\overset{(e)}{=}P[X_1\leq k_1]P[X_2\leq k_2]P[X_3\leq k_3]$$ This holds for all integers $k_1, k_2, k_3 \in \{1, ..., n\}$. With some thought, you can convince yourself that it also holds for all real numbers $k_1, k_2, k_3 \in \mathbb{R}$. In general, we define three random variables $X_1, X_2, X_3$ to be mutually independent if equality (e) holds for all real numbers $k_1, k_2, k_3 \in \mathbb{R}$. Hence, the three random variables $X_1, X_2,X_3$ of interest here are mutually independent. In fact, it is easy to see that they are also identically distributed.