Bayes' Theorem clarification

Question

The Bayes' Theorem formula is:

$$P(x|y)=\frac{P(x,y)}{P(y)}=\frac{P(x)P(y|x)}{\sum_{x^{'}}P(x^{'})P(y|x^{'})}$$

In this part, $P(x|y)=\frac{P(x,y)}{P(y)}$, we don't change the $x$ and $y$ order ($P(x|y)$ has the same order as $P(x,y)$), but in the last part $P(x)P(y|x)$ we change it.

Could you please explain me why?

Answer 1

There is no "order", $p(x,y)$ and $p(y,x)$ are the same thing (in the sense that event $A \cap B$ is the same as event $B \cap A$).

Now, from the defition of conditional probability the joint probability can be expressed in two ways

$$p(x , y ) = p( x | y) p(y) = p (y | x ) p(x)$$

Takin the second equality, you can express one conditional probability in terms of the other (which is the point) :

$$ p( x | y) = \frac{p (y | x ) p(x)}{p(y)}$$

Answer 2

That's a main point of the theorem :) Note that $P(x|y)$ may be quite different from $P(y|x)$, and moreover, it may be that one of these probabilities is available whereas the other is of interest to calculate. Note that the Bayes' theorem allows working in both directions as $P(x,y)=P(x)P(y|x)=P(y)P(x|y).$

Answer 3

Actually Bayes Theorem is some union of the first and the second equality: $P(x | y) = \frac{P(y | x) P(x)}{P(y)}$. Your last equality is consequence of Bayes theorem and Law of total probability.

This rearrangements are useful as we often know probabilities of hypothesis ($x$) and observations given hypothesis $(y | x)$, but not probabilities of observations $(y)$ or intersections of observations and hypothesis $(x, y)$, but we are interested in probability of hypothesis given observation $(x | y)$ - then left part contains what we are interested in, and right part contains what we know.