Reversing conditional distribution

Question

I want to find $p(x \vert y)$ given $p(y \vert x)$. I am aware of Bayesian formula but I can not understand why the following logic is wrong.

Let $x$ some random variable with pdf $p(x)$ and $\varepsilon \sim N(0,1)$.

Define $y(x) \equiv x + \varepsilon$, then $(y\vert x) \sim N(x,1)$.

But it also should be true that $x=y-\varepsilon$ and because standard normal distribution is symmetric around zero, then it should follow that $(x \vert y) \sim N(y,1)$. Thus we didn't use the information about unconditional distributions, needed for Bayesian formula.

Please, point me where I am wrong.

Update 1: by Bayesian formula I mean $p(x \vert y) = \frac{p(y \vert x)p(x)}{\int_{-\infty}^{+\infty}p(y \vert x)p(x)dx}$.

Update 2: The following analytical solution should prove that two distributions are indeed not equal.

Let $x \sim N(\mu,\sigma)$, then $p(x)=\frac{e^{-\frac{(x-\mu )^2}{2 \sigma ^2}}}{\sqrt{2 \pi } \sigma }$.

Let $y(x)=x+\varepsilon$ and $\varepsilon \sim N(0,1)$, then $p(y\vert x)=\frac{e^{-\frac{1}{2} (y-x)^2}}{\sqrt{2 \pi }}$. Hence, $\int_{-\infty}^{+\infty}p(y \vert x)p(x)dx=\frac{e^{-\frac{(y-\mu )^2}{2 \left(\sigma ^2+1\right)}}}{\sqrt{2 \pi } \sqrt{\frac{1}{\sigma ^2}+1} \sigma }$ and therefore by Bayesian formula $p(x\vert y)=\frac{\sqrt{\frac{1}{\sigma ^2}+1} \exp \left(-\frac{(x-\mu )^2}{2 \sigma ^2}-\frac{1}{2} (y-x)^2+\frac{(y-\mu )^2}{2 \left(\sigma ^2+1\right)}\right)}{\sqrt{2 \pi }}$. Finally, this is not equal to pdf of $N(y,1)$, which is $\frac{e^{-\frac{1}{2} (x-y)^2}}{\sqrt{2 \pi }}$.

Answer 1

Take $f(x,y)=p(y|x)$ as given conditional PDF and $p_X{x}$ as PDF of $X$. therefore:

$$p(x,y)=p(y|x)p_X(x)\to p_Y(y)=\Sigma_x p(y|x)p_X(x)\to p(x|y)={{p(y|x)p_X(x)}\over{\Sigma_x p(y|x)p_X(x)}}$$

So this question requires extra information. Now using your method the question is: "What if $\epsilon\nsim N(\mu,\sigma^2)$?"

Answer 2

The best is to distinguish between random variables and values taken by them. So let me reformulate the question:

We have three random variables, $X$ and $$Y=X+\varepsilon,$$ and $\varepsilon$ is of $N(0,1)$. If we may, we assume that $X$ and $\varepsilon$ are independent. The conditional distribution of $Y$ given that $X=x$ is then, indeed $N(x,1)$.

On the other hand $$X=Y-\varepsilon$$

and let's calculate $X$'s conditional distribution given that $Y=y$.

$$F_{X\mid Y=y}(x)=P(X<x\mid X+\varepsilon=y)=P(X<x\mid X=y-\varepsilon)=$$ $$=P(y-\varepsilon<x\mid X=y-\varepsilon)=P(y-\varepsilon<x)$$ because of the independence of $X$ and $\varepsilon$. From here,

$$F_{X\mid Y=y}(x)=P(y-\varepsilon<x)=P(\varepsilon>y-x)=P(\varepsilon<x-y)$$ because of symmetry. And this is the distribution function of $N(y,1)$. So, there is no mistake.

The only thing is that independence had to be assumed. Also, the symmetric nature of $N(0,\sigma)$ was exploited.

---

EDIT

On the necessity of independence.

Let $\epsilon^-$ and $\epsilon^+$ be two random variables of "half normal $\pm$" distribution. The distributions are defined as follow

$$F_{\epsilon^+}(x)=\begin{cases}2N(0,1)(x)&\text{ if } x\geq 0\\0&\text{otherwise}\end{cases}.$$

($\epsilon^{-}$'s pdf is similar but it is $0$ for $x$s less than zero...) Also, assume that $\epsilon^{\pm}$ are independent of $X$. Now, assume that there is an $x_0>0$ so that $P(X<x_0)=P(X\geq x_0)=\frac 12$ and define $$\varepsilon=\begin{cases}\varepsilon^-&\text{ if }&X<x_0\\\epsilon^+&\text{ if }&X\geq x_0.\end{cases}.$$

The distribution of $\varepsilon$ is $N(0,1)$ but it is not independent from $X$.

Let't calculate $Y$'s conditional distribution given that $X=x$:

$$P(X+\epsilon<y\mid X=x)=P(\varepsilon<y-x\mid X=x)=\begin{cases}F_{\epsilon^-}(y-x)&\text{ if }& x<x_0\\ F_{\epsilon^+}(y-x)&\text{ if }& x\geq x_0.\end{cases}$$

For a given $x>x_0$ the conditional pdf of $Y$ looks like this

Answer 3

Let's clean up notation first; I will use $Z$ instead of $\varepsilon$. We have 2 independent random variables $X$ and $Z$. To simplify the following I'll assume that $X$ and $Z$ are discrete. Now we define $Y=X+Z$

The conditional PMF of $Y$ given $X$ is $$p_{Y|X}(y|x)=P(Y=y|X=x)=P(X+Z=y|X=x)=P(x+Z=y)=P(Z=y-x)=p_Z(y-x)$$ so the conditional PMF $p_{Y|X}(y|x)$ if the same as $p_Z(y)$ shifted right by $x$.

The conditional PMF of $X$ given $Y$ is $$p_{X|Y}(x|y)=P(X=x|Y=y)=P(Y-Z=x|Y=y)$$

We can't proceed from here to

$$P(Y-Z=x|Y=y) \overset{?}{=} P(y-Z=x)=P(Z=y-x)=p_Z(y-x)$$

because $Y=X+Z$ and $Z$ are dependent.

Answer 4

The problem comes from the second part of the argument where you write $x=y-\varepsilon$. Because $y$ and $\varepsilon$ are not independent variables, $P_{X|Y}(x|y)$ is not just a shift of $P_\varepsilon$.

To convince yourself that is the source of the problem, think of the following argument:

If $y = x+\varepsilon$, and $x$ and $\varepsilon$ are independent variables, then $y$ will have a distribution with a variance larger than that of $x$.

If $x = y - \varepsilon$, and $y$ and $\varepsilon$ are independent variables, then $x$ will have a distribution with the larger variance.