Maximum Likelihood Estimation: difference between probability and likelihood

Question

In Maximum Likelihood Estimation (MLE), we want to maximize the probability $P(x_1, x_2, x_3,.. x_n | \theta)$, where $x_1, x_2, x_3,.. x_n$ are the datapoints and $\theta$ is the parameter vector. Conceptually, since our goal is to find $\theta$, I find more intuitive to think of MLE as $P(\theta | x_1, x_2, x_3,.. x_n)$. Indeed, the datapoints are given and we are trying to find a specific value for $\theta$. I know that this is not right since all the books and courses I have been reading on the subject formalize it as $P(x_1, x_2, x_3,.. x_n | \theta)$, but I don't understand why. I guess it concerns the conceptual difference between likelihood and probability, but I would be grateful if someone could explain it to me.

Answer 1

Your intuition is correct. Actually, making decision based on $P(\theta |x_1,\cdots ,x_n)$ is called Maximum Aposteriori Probability (MAP) Decision Making which is the start point of a variety of decision making rules in detection and communication theory. Both MAP and ML translate to the same decision making rule when all the possibilities of $\theta$ occur equally likely, since $$P(\theta |x_1,\cdots ,x_n)={P(\theta)\cdot P(x_1,\cdots ,x_n|\theta)\over P(x_1,\cdots ,x_n)}$$which is equivalent to $P(x_1,\cdots ,x_n|\theta)$ since $P(x_1,\cdots ,x_n)$ is independent of $\theta$ and $P(\theta)$ is equal based on our assumption. However, ML does have an advantage. You can interpret decision-making as choosing the region in which the observation occurs. For example, if $\theta $ were to choose from $-1,1$ equally likely over whole the $\Bbb R$, an observation $y>0$ would be translated to $\theta=1$ (since it has fallen in the positive numbers region) and an observation $y<0$ would be translated to $\theta=-1$ (since it has fallen in the negative numbers region).