Quite often, when I read some text where density parameters are used, some special rules apply. Example:
The $x$ is a random vector drawn from the same distribution as samples in $X$. I understand, that they use law of total probability, but I arrive at something slightly different:
$$ p(x|X) =\int p(x,\theta|X) d\theta =\int \frac{p(x,\theta,X)}{p(X)} d\theta =\int \frac{p(x|\theta,X)p(\theta,X)}{p(X)} d\theta =\int p(x|X,\theta) p(\theta|X)d\theta $$
For me, $\theta$ is just a random variable. I don't understand these special rules. I've never read a proper introduction about such parametrised densities. Would you please explain to me why am I free to omit $X$ in the condition as they did in the picture above? Would you please recommend me some articles, books, anything, that'll help me understand it?
Thank you.
The rule is Conditional Independence. The article on Wikipedia links to great answers on this site. My case is probably similar to biased coin tossing example.
Bob tosses the coin and gets heads. Since I had no knowledge about the coin, I can now assume that the coin is slightly biased towards heads. If Alice tosses the same coin after Bob, then $P(A=H|B=H) > P(A=H)$. But if I had the information (event $C$) that the coin is biased towards heads, then $$P(A=H|B=H,C) = P(A=H|C) > P(A=H).$$ Given $C$, event $B=H$ doesn't change the likelihood of event $A=H$ (I already know the coin is biased from $C$. But can results of Bob affect my belief in $C$?).
Same goes for my vectors $x$ and $X$. Given previous samples $X$, I can assume the likelihood of vector $x$. But once I have the parameter of the distribution $\theta$, previous samples $X$ give no additional information over $\theta$ on the likelihood of $x$. So $$ p(x|\theta,X) = p(x|\theta). $$
I'm not familiar with this concept and I've never encountered it in my probability courses or in other texts that use probability (not even in appendices where probability basics are introduced). Everyone seems to tacitly assume that everyone understands it. It's obvious when explained like in articles above.