I would like some help in understanding the nature of the following:
Let's assume that someone has a way to estimate the distribution of a random variable (e.g. heights). Let's also assume that someone else has another way (equally accurate) to estimate the same phenomenon (e.g. these heights).
Imagine that these distributions (for this common variable), are respectively $P_1$ and $P_2$ (and are not the same exactly). If my objective is to take advantage of both beliefs to get a better estimation of the real distribution, I assume I could get the product of these univariate distributions, and make a new density function (which is not normalized).
I am having difficulties in understanding..
- Does this make sense to do?
- I guess that these two ways should be independent of each other?
- Also, this is not a Joint or Conditional probability, right?
- Finally, would the distribution mixture $P_1(x)+P_2(x)$ be a wrong way for achieving the same objective?
In general, how should I interpret the density product $P_1(x)P_2(x)$ (or from N density functions of the variable x) in probabilistic terms (Bayes theory, Joint/Marginal/Conditional probs, etc)?
Here is how I would interpret and approach modeling your question (note there may be other interpretations). First, a little notation, let $x$ be a sample from random variable $X$ with unknown distribution. What you want to determine is the probability $P(X=x)$. There are two possibilities, either $X \sim P_1$ or $X \sim P_2$. As a Bayesian, you should prescribe a prior distribution that describes your degree of belief in these 2 possibilities. Let $P(X \sim P_1) = A$ and $P(X \sim P_2) = B$, the values of $A$ and $B$ depend on your prior knowledge about the two possibilities, if you don't have good prior knowledge than an uninformative flat prior where $A=B=1/2$ is a safe choice (note that $A+B$ must equal $1$). Now let's use the laws of probability to find $P(X=x)$. $$P(X=x) = P(X=x \text{ and} X \sim P_1) + P(X=x \text{ and} X \sim P_2)=$$ $$= P(X=x|X \sim P_1)P(X \sim P_1) + P(X=x|X \sim P_2)P(X \sim P_2) =$$ $$=A P_1(x) + B P_2(x). $$ So essentially your resulting probability should be a weighted average of the two possibilities where the weights are determined by your degree of belief in the given estimates.