Page 68 in pattern recognition and machine learning (free) says
there are infinitely many probability distributions that could have given rise to the observed finite data set. Indeed, any distribution p(x) that is nonzero at each of the data points x1, . . . , xN is a potential candidate.
Consider flipping a coin, with param $\mu$.
There are infinite number of probability distributions, such as $\mu$ = 0.1, 0.001, 0.2, ....
could give rise to the observed finite data set {H,T}
what does the second part ("any distribution p(x) that is nonzero at each of the data points x1, . . . , xN is a potential candidate") mean?
are "each of the data points" referring to the set {H,T}, or a single element of the set, such as "H"?
I assume "each of the data points" refer to a single element of the set, "any distribution at each of the data points" implies H and T may have different distributions, each element in set has its own distribution, is my understanding right? if it is, it is NOT i.i.d, right?
In statistics and machine learning your primary input is a sample $x_1, ..., x_n$.
Where the sample comes from changes from problem to problem, but in the usual case we assume that each element of the sample comes from the same distribution, sampled independently.
That is, each $x_i$ is sampled from a random variable $X_i$ such that all the $X_i$ are iid, and distributed according to a probability mass function (assuming that the outcomes are discrete) $p$.
The central problem of statistics is, given a sample, figuring out what $p$ is.
The sentence above highlights that by mere deduction we cannot deduce much about the probability distribution the sample originates from.
Indeed, let our sample be $S = {1,2,1,4,5,3}$ for example.
Then we know that $p(3)>0$ (since at least one instance of the sample $3$ was observed). But any $p$ such that $p(n)>0$ for all $n\in S$ will be a possible candidate for $p$.